KV Cache Reduced to 1/8. So, What Changes?

To get straight to the point.

With the introduction of DeepSeek-V4.1-Flash, the API cost for handling long-context texts has dropped to about 1/4 to 1/8 of what it used to be. This is not just a minor price reduction. The API usage fee that used to cost 200,000 yen per month is now in the range of 20,000 to 50,000 yen per month.

Whether small and medium-sized enterprises can utilize AI ultimately boils down to “how much can we operate for per month?” That barrier has been lowered once again.

This time, we will bundle three technological news items together to specifically consider what can be assembled with a “50,000 Yen AI.”

• DeepSeek-V4.1-Flash KV cache compression (significant reduction in API costs)
• Selection criteria for open-source agent harnesses for local LLMs (in-house operational foundation)
• Threat hunting using small language models (SLM) (examples of task automation)

Let’s take a look at each one.

—

1. DeepSeek-V4.1-Flash — The Shock of KV Cache at 890 Bytes/Token

DeepSeek-V4.1-Flash is a Mixture-of-Experts (MoE) model with 552 billion parameters, capable of handling a context of up to 1 million tokens. Up to this point, it’s just “impressive.”

What’s really noteworthy is that the footprint of the KV cache has been compressed to 890 bytes/token. This is about 1/4 of the previous generation DeepSeek-V4-Flash. Furthermore, when combined with an inference optimization called “SWA Bounded Replay,” the persistent KV cache can shrink to about 1/8.

What does this mean?

The KV cache is the biggest factor consuming GPU memory during LLM inference. As the cache size decreases, the number of requests that can be handled by the same GPU increases. The costs for API providers go down, and so do the usage fees. In other words, “with the same budget, you can process longer contexts and more requests.”

In terms of specific costs, if you handle inquiries using a 1 million token context several thousand times a month, the API costs would typically be in the range of 150,000 to 200,000 yen per month. With the compression efficiency of V4.1-Flash, the equivalent processing could potentially fit within a range of 30,000 to 50,000 yen per month.

There was a time when a 3 million yen business system became 300,000 yen. Now it’s time for the monthly 200,000 yen AI operational cost to drop to 50,000 yen.

This change in cost structure is critically important for small and medium-sized enterprises in rural areas. The assumption that “AI is for large corporations” is being challenged once again.

—

2. Local LLM × Agent Harness — The Realistic Solution of “Running It In-House”

Even though API costs have decreased, there are still risks associated with continuing to rely on cloud APIs. Price revisions, rate limits, and external data transmission are all concerns. For small and medium-sized enterprises dealing with customer data, the desire to “not send data outside” is a pressing issue.

This is where the configuration of local LLM + agent harness comes into play.

An agent harness is a framework that provides LLMs with “limbs” for tool invocation, task decomposition, memory management, and more. As of 2026, there are 11 types of open-source options ranked.

Four Criteria to Consider in Selection

1. Is it OSI-approved? — Essential to avoid pitfalls in commercial use.
2. Quality of documentation — In small enterprises with only one engineer, frameworks without documentation cannot be used.
3. Maintenance frequency — Avoid those with the last commit more than three months ago.
4. Safety management — Control over tool invocation permissions, presence of sandbox execution.

Especially when using local models, the context window is small, and the accuracy of tool invocation is low. Therefore, the design quality of the harness is crucial for success or failure. Prompts that might work with large cloud API models may not work with local 7B to 14B models. If the harness does not appropriately divide tasks and manage context, it will not function properly.

Here’s a practical tip. When running local models with Ollama, the default context length is often insufficient. You can extend it to 64,000 tokens with the following line:

“`
/set parameter num_ctx 64000
“`

Just this simple adjustment can multiply the amount of information the agent can retain. Half of the consultations about “I tried running it, but the accuracy is lacking” stem from insufficient context length settings.

An Example of a Local Configuration That Can Be Assembled for 50,000 Yen

With this setup, tasks like “automated internal FAQ responses,” “automated daily report summaries,” and “draft generation for estimates” can be adequately handled. You can achieve about 70% of the functionality of systems that large corporations build for several million yen for just 30,000 to 40,000 yen a month.

Because it’s a small and medium-sized enterprise, not all employees will use it. If 10 people are using it, this scale is sufficient.

—

3. Automating Specialized Tasks with Small Language Models — A Case Study in Threat Hunting

The third piece of news involves an automation case in the security field using small language models (SLM).

In Security Operations Centers (SOC), the Kusto Query Language (KQL) is used for threat hunting. The problem is that there are overwhelmingly few personnel who can write KQL accurately. This has become a bottleneck for scaling security teams.

This is where SLM comes into play. By inputting natural language like “List users who had suspicious login attempts in the past 24 hours,” the SLM converts it into an accurate KQL query.

There are two key points to this approach:

1. Error feedback prompts — Execute the generated KQL, and if an error occurs, it automatically corrects it repeatedly.
2. LoRA fine-tuning — A lightweight adapter specialized in KQL is placed on the small model, achieving higher accuracy than general-purpose LLMs.

Validation using Microsoft’s NL2KQL Defender Evaluation dataset showed that this two-step approach achieves accuracy approaching that of large models like GPT-4, at less than 1/10 the cost.

Why is this case important for small and medium-sized enterprises?

Queries and code that “cannot be written without specialized knowledge” exist in every industry, not just security.

• SQL queries for quality control databases in manufacturing
• API integration scripts for accounting software
• Inventory management logic for e-commerce sites

A system that converts these from “natural language to specialized code” can be sufficiently constructed using small models and LoRA fine-tuning. Moreover, the training data can simply be your company’s past query logs. A few hundred entries can bring accuracy to a practical level.

Specialized tasks that were previously reliant on individuals can now be managed through systems. This is structurally stronger than simply “increasing personnel.”

—

So, What Should We Do?

When we bundle these three pieces of news together, the emerging configuration pattern looks like this:

“Run everyday tasks with local small models and only throw difficult decisions to the DeepSeek-V4.1-Flash API. The agent harness will automatically handle that distribution.”

This configuration can operate within a budget of 50,000 yen per month. Moreover, data will primarily remain local.

Here are three things you should do right away:

1. Inventory your company’s “repetitive tasks” — Daily report summaries, inquiry responses, data aggregation. Start by selecting one.
2. Try running a local LLM + agent harness on one machine — Run a 14B model with Ollama and set the context length to 64K. Get it running first.
3. Measure API costs monthly — Determine whether you are “using less than expected” or “higher than expected” based on the numbers.

There’s no need to wait for technological evolution. The components are already in place.

“What will you automate for 50,000 yen?” — What is being questioned is not technology, but management judgment.