Conclusion First: Half of the Monthly Cost of AI Agents is ‘Wasted Memory’

There has been an increase in inquiries from small businesses that have implemented AI agents.

“At first, it was about 20,000 yen a month, but before I knew it, it had jumped to 80,000 yen.”

This is not a bug. It is the result of operating without understanding the ‘memory mechanism’ of AI.

AI agents retain ‘context’ with each conversation. They appear intelligent because they remember past interactions, but this ‘remembering’ incurs costs measured in tokens. The troublesome part is that this cost does not increase linearly with the length of the conversation; it grows quadratically.

If the conversation doubles, the cost quadruples. If it triples, the cost increases ninefold. This is the true nature of the ‘invisible invoice’.

For small businesses, a difference of several tens of thousands of yen a month is significant. Therefore, I want to convey the fact that simply reassessing the ‘memory design’ can reduce costs to less than half, along with concrete numbers.

—

Why Does ‘Memory’ Consume Costs?—Understanding the Structure

First, let’s clarify where the memory costs of AI agents arise.

Taking OpenAI’s GPT-4o as an example, the cost per input token is about $2.5 (approximately 375 yen) per million tokens, while the output costs about $10 (approximately 1,500 yen). At first glance, this seems inexpensive.

However, the problem lies in the mechanism where the agent resends all past conversation history as input tokens with each response.

Let’s calculate this specifically.

• Assume an average of 500 tokens are exchanged per inquiry.
• In a conversation with 10 exchanges, the last input alone would consume about 5,000 tokens.
• However, in reality, each input includes the entire past history, resulting in a cumulative input token count of 500 + 1,000 + 1,500 + … + 5,000 = approximately 27,500 tokens.

If this operates with 50 inquiries a day, the monthly input tokens alone would amount to approximately 41.25 million tokens. For GPT-4o, this translates to about 15,000 yen just for the ‘input memory cost’. When combined with output, it easily reaches 30,000 to 50,000 yen a month.

And this is the figure for the scenario where ‘no adjustments were made’. In customer support applications where conversations tend to be lengthy or in sales support agents referencing past negotiation histories, this number can easily multiply several times.

In one regional real estate company, the monthly API cost for their property guidance AI agent ballooned to 3.2 times its initial amount within three months of implementation. The cause was that they were sending all conversation histories for each customer every time.

—

What Changes with ‘Memory Design’?—Three Levers

So, what can be done? There are three main levers to reduce memory costs.

1. ‘Summary Compression’ of Conversation History—You Don’t Need to Remember Everything

This is the most impactful.

Instead of retaining past conversations as they are, after a certain number of turns, you can summarize and compress them. For example, instead of sending 5,000 tokens for 10 exchanges, you can compress it to a summary of 200 tokens for the past plus 1,500 tokens for the last three exchanges, totaling 1,700 tokens.

This alone can reduce the number of input tokens by approximately 66%.

Implementation is not difficult. LangChain and LlamaIndex have a mechanism called ‘ConversationSummaryMemory’ that automatically generates summaries every few turns. By simply changing the settings, the next month’s bill will visibly change.

In the aforementioned real estate company, after implementing this summary compression, their monthly API cost dropped from 87,000 yen to 31,000 yen, a reduction of about 64%.

2. Utilizing KV Cache—Don’t Resend the Same Context Multiple Times

OpenAI and Anthropic APIs have a ‘prompt cache’ feature. By caching parts that send the same content each time, such as system prompts or common instructions, you can reduce input token costs by up to 50%.

In the case of Anthropic’s Claude API, the reading cost of cached tokens is typically one-tenth. If the system prompt is 2,000 tokens, simply enabling caching can change the cost for that part by several thousand yen monthly.

Small business AI agents typically send the same system prompt (“You are the customer support representative of XX company…”) every time. Just caching this will definitely make it cheaper than doing nothing. Changing the settings requires only a few lines of code.

3. Setting ‘Limits’ on the Context Window—Don’t Let It Remember Limitlessly

The third point is to set a limit on memory in the first place.

While GPT-4o supports a context window of up to 128K tokens, there is no need to use all 128K. Rather, it is better to restrict it to 4K to 16K according to the application, making it easier to control both cost and quality.

You might find it surprising, but research has shown that overly long contexts can reduce response accuracy. This is known as the ‘Lost in the Middle’ phenomenon, where AI tends to overlook information in the middle of lengthy contexts. In other words, narrowing memory can lead to cheaper and more accurate responses.

For small business inquiry responses, a recent five exchanges plus a summary—around 4,000 tokens—is usually sufficient.

—

How Much Will It Actually Change?—Cost Estimation

Let’s look at specific numbers.

Conditions: 50 inquiries a day, average of 10 exchanges per inquiry, using GPT-4o

An annual savings of 210,000 yen. For small businesses, this amount is close to the monthly salary of one part-time employee. Moreover, this is an estimate for just one agent. If multiple AI agents are running within the company, the effects will multiply even further.

—

Why Small Businesses Can Win with ‘Memory Design’

Now we come to the main point.

Large corporations do not worry about token costs. They fully utilize their ample budgets to use 128K tokens, prioritizing accuracy above all else. That is how large companies compete.

But small businesses are different. The winning pattern is to keep costs down while achieving sufficiently practical quality. And this ‘memory design’ becomes precisely that lever.

Consider this:

• Is a context of 128K tokens necessary for responding to inquiries at a local construction company?
• Is it necessary for a local tax accountant’s FAQ bot to retain all conversation histories?

The answer is No. Recent context and a summary of past interactions are sufficient. Rather, small businesses that can design memory with a specific purpose have a cost efficiency advantage over large companies.

Large corporations, in seeking versatility, inevitably have to be rough in their memory design. Small businesses, with clear applications, can optimize precisely. This is a structural advantage.

—

So, What Should You Do?—Three Steps You Can Start Tomorrow

Putting aside the complex discussions, here are three things you can do starting tomorrow.

Step 1: Check Your Current API Invoice

First, check this month’s token usage and billing amount. If you are using OpenAI, you can view it on the Usage dashboard; for Anthropic, check the Console. If you feel “it’s higher than expected,” it is likely a memory design issue.

Step 2: Implement Conversation Summaries

If you are using LangChain, switch to `ConversationSummaryBufferMemory`. By simply setting the max_token_limit to 2,000 to 4,000, previous conversations will be automatically summarized. This alone will reduce input tokens to less than half.

Step 3: Enable Prompt Caching

For Anthropic’s Claude API, set the `cache_control` parameter. In OpenAI’s API, using the Assistants API’s thread function will also enable internal caching. The longer the system prompt, the greater the effect.

Executing these three steps will take about half a day. There is no need to outsource. If you have one engineer, it can be completed within this week.

—

Designing Memory Means Deciding ‘What to Forget’

Finally, one last point.

Designing AI memory is about deciding what to forget, not what to remember.

Like human memory, it is not necessary to remember everything. What is important is the ability to retrieve necessary information at the right time. And for that, the ability to discard unnecessary information is crucial.

AI agents with well-designed ‘forgetting’ can maintain low costs, respond quickly, and achieve high accuracy—all three factors.

Conversely, agents that try to remember everything become slow, costly, and inaccurate.

If you look at your AI agent’s monthly invoice and think, “This is expensive,” I urge you to first question the memory design. The answer is usually found there.