Conclusion First: AI is Burning Money by “Overthinking”

Small and medium-sized enterprises that have integrated ChatGPT and Claude Code into their operations are voicing the same complaint.

“The monthly costs are higher than expected”

You may have experienced the shock of opening an API usage bill, where an estimated monthly cost of 50,000 yen turned into 150,000 yen. Upon investigation, the culprit was found to be the AI “overthinking”. Even for simple questions, it generates lengthy chains of thought, consuming tokens at an alarming rate.

This is not just a matter of perception; it’s a structural issue.

Latest inference models like OpenAI’s o1 and o3, and DeepSeek-R1, generate a massive amount of internal “Chain-of-Thought” to improve answer accuracy. The problem is that most of this thinking contributes little to no improvement in answer accuracy. In other words, you are paying to listen to unnecessary ramblings.

So, what can be done? Recent research has revealed three concrete approaches.

—

1. 99% Compression of Inference Tokens: Keep Only “Meaningful Thoughts”

A shocking method was presented by a research team from Microsoft and Carnegie Mellon University (Sui et al., 2025 “Stop Satisficing”).

They filtered the inference tokens (text of the thought process) generated by LLMs based on entropy (uncertainty of information). The approach is simple: they assess whether “this token is really necessary for the next judgment” using the entropy value and eliminate the unnecessary ones.

The results are as follows:

• Even with a 99% reduction in inference tokens from DeepSeek-R1, accuracy on the math benchmark (MATH-500) barely dropped.
• This means that 99% of the “thought process” generated by the AI did not contribute to answer accuracy.

Consider the implications of this figure. The API fees for inference models are higher for output tokens (including inference tokens) than for input tokens. For OpenAI’s o1, the cost of output tokens is about four times that of input tokens. If 99% of that output is “equally accurate without it”, then you are essentially paying every month for nearly meaningless text generation.

What Small and Medium-Sized Enterprises Can Do Right Now

Currently, directly applying this entropy filtering at the API level is challenging. However, there are practical steps that can be taken:

• Limit the use of inference models (o1, o3). In scenarios like simple text formatting, standard summarization, and FAQ responses, inference models are unnecessary. By sorting tasks that can be handled by GPT-4o mini or Claude 3.5 Haiku, monthly costs can be reduced to less than half.
• Narrow down the `max_tokens` and `reasoning_effort` parameters. Many companies use OpenAI’s o1 models with default settings for the maximum inference tokens. By adjusting these parameters according to the task, costs can significantly decrease.
• Instruct the AI to “think concisely” in the prompt. Instead of saying “Think step by step”, you can say “Answer directly without lengthy reasoning”, which has been reported to halve the number of inference tokens in some cases.

In terms of costs, a monthly API fee of 150,000 yen could drop to 30,000 to 50,000 yen just by differentiating model usage and adjusting parameters. This is not a theoretical figure; it has been confirmed with our clients.

—

2. Diagnosing Wasteful Inference: A Technique to Make AI Say “I Don’t Know”

Simply cutting down inference tokens is not enough. There is another structural issue.

AI generates plausible reasoning even for “unknown problems”.

The research by Zhang et al. (2025) titled “Diagnosing Reasoning Degradation in LLMs” quantitatively clarified this issue. They analyzed LLM reasoning and categorized “Unfaithful Reasoning” into three categories:

1. Fabricated Reasoning: Constructing false logic to make it appear as if a correct answer is being provided.
2. Inaccurate Reasoning: Reaching a confident conclusion despite errors in intermediate calculations or logic.
3. Incomplete Reasoning: Jumping to conclusions while skipping important steps.

The problem is that full tokens are consumed for these “wasteful inferences”. You are paying to receive incorrect answers.

The proposed solution to this is CaRL (Capability-aligned Reinforcement Learning). This trains the model to recognize its own limitations and to stop reasoning when it determines, “I cannot solve this problem”. In essence, it trains the AI to say, “I don’t know”.

What Small and Medium-Sized Enterprises Can Do Right Now

While the training of CaRL is still in the research phase, the underlying concept can be implemented immediately:

• Filter outputs using confidence scores. There is an API option to return LLM outputs with logprobs. Responses with low confidence can be flagged for human review. This creates a system that prevents erroneous answers based on wasteful reasoning from being used in operations.
• Explicitly instruct in the prompt to “answer with ‘unknown’ if unsure”. This simple addition can have a significant effect. The AI will no longer force itself to extend reasoning, reducing token consumption. In our experiments, adding this single line resulted in a 30% reduction in inaccurate answers and a 20% reduction in average output tokens.
• Systematize monitoring of answer quality. Sample AI outputs weekly and check for any instances of “fabricated reasoning”. Instead of relying on subjective reviews, create a checklist that anyone can use.

The cost-saving effect here is not just from token reduction. The cost of rework due to incorrect answers decreases. The highest cost for small and medium-sized enterprises is labor. If an employee spends one hour correcting an AI’s incorrect answer, that results in a loss of 3,000 to 5,000 yen. If this occurs 20 times a month, that amounts to 60,000 to 100,000 yen. Reducing this by half dramatically improves the ROI of AI implementation.

—

3. Zero-Token Memory: A System That Costs Nothing to “Remember”

The third point is about memory management.

When using LLMs as chatbots or internal assistants, it is necessary to “remember” past conversations. Traditionally, this required additional LLM calls for reading and writing memory. In other words, every time the AI “remembers”, it consumes tokens and incurs charges.

The zero-token memory operation proposed by Huang et al. (2025) fundamentally changes this structure.

The mechanism works as follows: it extracts entities (such as names, company names, project names) from conversations and organizes their relationships in a graph structure. This allows memory reading and writing to be completed without calling LLMs, relying solely on graph manipulation.

Results:

• Token consumption for memory operations is zero.
• Time costs are reduced by 57.6%.
• Memory accuracy (F1 score) is equal to or better than traditional methods.

What Small and Medium-Sized Enterprises Can Do Right Now

• Reassess how conversation history is managed. Are you cramming all past history into the prompt for every conversation? The longer the history, the more input tokens increase, leading to higher costs. By only passing the last N conversations or providing summaries, input tokens can be reduced by 50 to 70%.
• Implement RAG (Retrieval-Augmented Generation) for external memory. Store internal documents in a vector DB and only search for the necessary information to inject into the prompt. This is significantly cheaper than reading the entire text each time.
• Adopt the concept of entity graphs. Maintain customer information and project data as structured data, providing the LLM with only the minimum necessary information, such as “the last three inquiries from this customer”.

When viewed monthly, use cases involving chatbots tend to have input tokens dominating costs. Just optimizing history management can lead to a projected monthly savings of 20,000 to 50,000 yen.

—

Combining the Three Approaches Can Reduce Monthly Costs by 90%

Let’s summarize the three approaches discussed:

Assuming a company is paying 150,000 yen monthly for API costs, if they halve output tokens through model differentiation (equivalent to -75,000 yen), halve input tokens through optimized history management (equivalent to -20,000 yen), and save on labor costs due to reduced rework from incorrect answers (equivalent to -30,000 yen), they would be left with 20,000 to 30,000 yen.

From 150,000 yen to 20,000-30,000 yen. About 1/5 to 1/7.

Furthermore, if the use of inference models is optimized by task, achieving 1/10 becomes a realistic figure.

—

So, What Should Be Done?

Let’s narrow it down to three actionable steps that can be taken starting today:

1. Break down the API billing statement into “input tokens” and “output tokens”. You need to understand where the money is being burned to take appropriate measures.

2. Inventory the areas where inference models (o1/o3/R1) are used. The tasks that truly require inference models are likely less than 20% of the total. Switch the remaining 80% to lightweight models.

3. Include “answer concisely” and “if unsure, answer unknown” in the prompts. This can yield results starting today with zero technical implementation.

AI performance is improving daily. However, as performance increases, the costs of “overthinking” also rise. Using AI wisely means knowing how to silence it effectively.

For small and medium-sized enterprises, a difference of 100,000 yen per month is significant. That 100,000 yen could cover the cost of hiring one person. Reducing AI costs creates room for investment in human resources.

There’s no need to read technical papers. Start by opening the bill and looking at the numbers.