Conclusion First

The technology that reduces AI operational costs to one-hundredth has arrived.

A monthly cloud API usage fee of 30,000 yen is not an amount that small and medium-sized enterprises (SMEs) can simply brush off as “let’s give it a try.” For a company with five employees, this is higher than the communication costs for one person. As a result, many SMEs have postponed the adoption of AI.

However, this week, technologies have emerged that could overturn this structure.

• “KVMem,” which can process one million tokens using consumer-grade GPUs (graphics cards costing tens of thousands of yen)
• Edge AI devices the size of a postage stamp that can run LLMs locally
• “Reversible computing” technology that recycles chip waste heat into energy

What happens when these three technologies are combined? The monthly AI usage fee of 30,000 yen can drop to 300 yen, including electricity costs. This is a reduction to one-hundredth. This isn’t just a story of lower costs; it signifies the disappearance of the very criteria for deciding whether to use AI or not.

KVMem: A Mechanism for Processing “One Million Tokens” with a 50,000 Yen GPU

First, let’s discuss KVMem, which has the most significant impact.

When using large language models (LLMs) as agents, the biggest hurdle has been “context length.” As past interactions and reference data increase, they can exceed the GPU’s memory (VRAM). Thus, reliance on large cloud GPUs was unavoidable, which is the true nature of the 30,000 yen fee.

KVMem fundamentally changes this. Here’s how it works:

1. Past conversation history and contextual data are stored in SSDs or main memory in “blocks.”
2. When a new query arrives, a lightweight index (attention space index) is used to select only the “blocks relevant to the current question.”
3. Only the selected blocks are loaded into the GPU’s VRAM for processing.

In short, instead of loading the entire history of one million tokens onto the GPU, it uses a “virtual memory” approach that pulls only the necessary parts as needed. The concept is similar to a PC’s swap space but is optimized for the attention mechanism of LLMs.

The effectiveness is clear when viewed in numbers. In the software engineering benchmark (DeepSWE), the task success rate was 43.8% with conventional compression methods, but it increased to 48.4% with KVMem. This is an improvement of about five percentage points. Moreover, this was achieved not with expensive cloud GPUs but with a consumer-grade GPU like the RTX 4070 (retail price of 80,000 to 100,000 yen).

Consider this: processing one million tokens monthly with a GPT-4 class model via a cloud API costs about 20,000 to 30,000 yen. If the same processing is done locally with an open-source model plus KVMem, the only cost incurred is electricity. The RTX 4070 has a power consumption of 200W. Operating for 8 hours a day results in a monthly electricity cost of about 1,500 yen. If the processing volume is lower, a monthly cost in the 300 yen range becomes realistic.

The initial investment of 80,000 yen for the GPU can be recovered in three months of cloud API usage.

Postage-Size Edge AI: Another Route to “De-Clouding”

If KVMem is about “running AI on existing gaming PCs,” another trend is about “running AI on dedicated ultra-compact devices.”

Recently introduced edge AI chips are equipped with LLM inference engines on boards the size of a postage stamp (2-3 cm square). Their power consumption is just a few watts. LLMs can operate on power comparable to that of a smartphone charger.

What does this mean for SMEs? Three things:

1. No need to send data to the cloud

Many business owners are reluctant to send customer data or internal documents to servers operated by OpenAI or Google. With edge AI, all data can be processed within the company’s devices. While this doesn’t eliminate the risk of information leaks, having the option to “keep it in-house” is significant.

2. Independence from internet connections

In rural factories and warehouses, many sites lack stable high-speed connections. Edge AI can operate offline, allowing inspection AI or inventory management AI to be placed in a corner of a warehouse without connectivity.

3. Elimination of monthly subscription fees

If a device is purchased outright for 10,000 to 30,000 yen, the only ongoing cost is electricity. With a power consumption of 5W, running it 24/7 results in a monthly electricity cost of about 100 yen. You can obtain an AI assistant for zero monthly fees.

However, there are caveats. Currently, the models that run on edge AI devices are primarily small models with 7B to 13B parameters. They won’t match the performance of massive models like GPT-4. However, they are more than sufficient for tasks involving repetitive, standardized inquiries, summarizing meeting notes, and reading invoices. About 80% of SME operations fall into this category of “standardized patterns.”

Chip Energy Recycling: A “Subtle but Essential” Technology with Long-Term Impact

The third point is a bit further down the line but is structurally important to mention.

Hannah Earley of Vaire Computing is developing “Reversible Computing” technology. Conventional chips discard energy as heat every time they erase intermediate data during calculations. This is dictated by a physical law known as Landauer’s principle, which states that “erasing information requires energy.”

Reversible computing avoids this “information erasure” design, allowing for the recovery of some of the energy that would have been discarded. In theory, this could drastically reduce the power consumption of current chips.

To be candid, this technology is likely still 5 to 10 years away from commercialization. However, what should be noted is that the power consumption of AI will “only decrease.” If it costs 300 yen now, it wouldn’t be surprising if it drops to 30 yen in five years.

So, What Should SMEs Do Now?

Having discussed the technology, let’s talk about practical steps.

There are two things you can start doing today.

1. Set up a local LLM for testing

Prepare a gaming PC lying around in the office or a used GPU-equipped PC costing around 50,000 yen. Using tools like Ollama, you can have a local LLM running in 30 minutes. Start by testing it on non-critical tasks like internal inquiry responses or summarizing daily reports, where failures won’t incur significant damage.

If your company is using a cloud API, you can compare the same processing done locally to see “how much accuracy drops.” In many cases, the conclusion will likely be, “Oh, it hardly changes.”

2. Inventory your “AI monthly costs”

ChatGPT Plus (3,000 yen/month), API usage fees (10,000 to 30,000 yen/month), and other SaaS AI feature charges… It’s not uncommon for companies to exceed 50,000 yen in total. Sort out which processes can be moved locally and which ones require the cloud.

Based on experience, about 70% of AI usage in SMEs can be transitioned to local processing. Even if the remaining 30% remains cloud-dependent, the monthly costs will still be reduced to less than half.

The Real Change is Not in “Costs” but in “Decision-Making Speed”

Finally, I want to emphasize the most important point.

A monthly cost of 30,000 yen becoming 300 yen is impactful in itself. However, the fundamental change is not cost reduction.

It will eliminate the need for management to make decisions on whether to use AI or not.

At 30,000 yen, approval is required. You must explain the cost-effectiveness. You become cautious, saying, “Let’s try it for three months first…” At 300 yen, the personnel on the ground can start it based on their judgment. They can try it out and stop if it doesn’t work. The removal of this “trial barrier” will exponentially increase the number of experiments with AI utilization.

Large corporations take six months to devise their AI strategies. SMEs can start testing next week. When costs drop to one-hundredth, organizations that can make decisions quickly will win. That will be SMEs, not large corporations.

The technology is now in place. The only question is whether to engage with it or not. That’s all there is to it.