Conclusion First: Teaching AI to ‘Remember’ is More Expensive Than You Imagine

Having AI agents remember past interactions. Keeping the context of customer support. Making proposals based on the content of previous business discussions—this sounds incredibly convenient, but behind the scenes, tokens are being consumed at an alarming rate.

According to a recently published benchmark study on memory systems, when memory functions are added to AI agents, the API usage cost incurred during 400 turns (approximately 200 exchanges) can, in some systems, be higher than simply resending the entire conversation history each time.

For agents handling several dozen customer interactions a month, it is not uncommon for memory-related costs to exceed 100,000 yen per month. That amounts to 1.2 million yen annually. For local SMEs, this is equivalent to “the labor cost of one part-time employee.”

The problem is that this cost is largely invisible. Furthermore, there are cases where the naive expectation that “the more it remembers, the smarter it becomes” is structurally betrayed.

Comparing Three Memory Systems Revealed Cost Differences of Up to Six Times

The benchmark study compared three memory systems: Mem0, Hindsight, and Mastra Observational Memory. All of these systems are designed to allow AI agents to “remember past conversations,” but their internal structures are completely different.

The results are as follows:

• Cheapest system: Approximately $3–5 (about 450–750 yen) for 400 turns
• Most expensive system: Approximately $18–30 (about 2,700–4,500 yen) for 400 turns
• Full transcript resending (no memory, resending the entire text each time): Approximately $8–12 (about 1,200–1,800 yen)

What’s noteworthy is the reverse phenomenon where it is cheaper to resend past conversations each time than to use the most expensive memory system.

Why does this happen? Memory systems call the LLM internally every time they “write,” “search,” or “update” memory. During a single conversation, there are 3 to 5 API calls happening behind the scenes. Although invisible to the user, tokens are definitely being consumed.

Let’s convert this into monthly operational volume:

• 20 customer interactions per day × average 50 turns per interaction
• 20 business days per month × 20 interactions = 400 interactions/month
• If the memory cost per interaction is $5, then the monthly cost is $2,000 (about 300,000 yen)
• Even the cheapest system would cost at least $1,200 (about 180,000 yen) per month.

While claiming to reduce labor costs with AI, memory costs alone can consume the equivalent of two part-time employees. This is the reality of the cost structure.

‘Summarizing to Reduce Costs’ is Not as Universal as You Might Think

A common method to reduce costs is “context compression.” This involves summarizing (gistifying) past conversations and compressing them into shorter texts before sending. Since the number of tokens decreases, costs go down. The reasoning is sound.

However, there is a pitfall here.

Research has shown that when gist compression is performed, the accuracy of answers to questions regarding time and dates significantly decreases. Questions like “What was the deadline decided in last week’s meeting?” or “When was the last estimate provided?” become unanswerable.

Summarization is the act of “discarding unimportant information.” However, what is considered important is not known until a question is asked. Information that the AI judged as “low importance” and discarded during summarization may later become critically necessary.

Consider this in the practical context of SMEs:

• When a customer says, “Under the same conditions as last time,” the previous conditions are not remembered.
• The “deadline communicated last month” is lost in compression.
• The “price previously proposed” is rounded off.

In this case, there is no point in having memory. As a result of cutting costs, trust is lost. This is the worst trade-off.

So, What Should SMEs Do?

“It’s expensive to have memory. Compressing it reduces accuracy. So isn’t it a dead end?”

Not at all. The issue lies in the design that attempts to “remember everything.”

Here’s the “right way to cut costs” for SMEs.

1. Narrow Down What to Remember

It is not necessary to remember all conversations. What should be remembered are only the “confirmed facts for each customer.”

• Confirmed information such as contract terms, prices, and deadlines
• Customer preferences and dislikes
• Past complaints and their resolutions

These should be stored as structured data in a database and retrieved only when necessary. The flow of conversation and small talk should not be remembered. By doing this, memory costs can be reduced to 1/5 to 1/10.

2. Solve with ‘Search’ Instead of Memory

Using the RAG (Retrieval-Augmented Generation) mechanism, past information can be retrieved through “search” rather than “memory.” The monthly cost of a vector database is just a few thousand yen. Compared to the 100,000 yen for memory systems, this is a significant difference.

Save the conversation history as is, and only search for and insert the relevant parts into the prompt. This way, there is no information loss due to compression.

3. Incorporate a ‘Forget’ Design

Humans do not remember everything either. AI should also have a “forgetting policy” designed.

• Automatically delete conversation histories older than 30 days
• Only transcribe confirmed facts into structured data before deletion
• Include only the most recent 5 conversations in the context

With just these rules, token consumption can dramatically decrease. It is entirely possible for monthly costs to drop from 100,000 yen to 10,000 to 20,000 yen.

4. First, Operate Without Memory

Is memory really necessary? There are many tasks where starting conversations from scratch is not an issue. Internal FAQ responses, standard estimate creation, manual searches—these do not require memory.

Identify the “tasks that can operate without memory” first. This is the most cost-effective action.

Costs That Are Not Visible Cannot Be Controlled

The greatest danger in introducing AI agents is the design of “just adding everything because it seems convenient.” Memory functions are a prime example, where costs can skyrocket the moment they are added, but no one realizes it until the bill arrives.

Here are three things you should do immediately:

1. Measure the token consumption of your current agent. You can find out in five minutes by opening the API dashboard.
2. Separate and understand memory-related costs. Determine what percentage of the total is consumed by memory.
3. Reassess on a business unit basis: “Is this memory really necessary?”

Large companies may tolerate memory costs of 1 million yen per month. However, SMEs are different. To achieve maximum effect with a limited budget, it is essential to have the ability to design “what not to remember.”

With the evolution of AI, the cost of memory will decrease in the future. However, waiting for it to drop is too slow. Understanding the current structure and achieving results with current costs is the way SMEs can compete.