Routing Technology Reduces LLM Costs by 90% — Saving 3 Million Yen Annually Just by Stopping ‘All GPT-4o’ Usage

Monthly cost: 300,000 yen, annual cost: 3.6 million yen.

Many small and medium-sized enterprises (SMEs) currently pay this range for LLM usage fees, as they are “just throwing everything at GPT-4o.” Internal inquiry responses, meeting minutes summaries, email drafts, data formatting — all are processed using the same model.

When you think about it calmly, this is absurd.

Is it really necessary to process “simple summaries” and “complex contract reviews” using the same model? It’s like sending out a 10-ton truck every time you go to the supermarket in a light truck.

Now, three technologies have emerged simultaneously to structurally eliminate this waste: Smart Model Routing, DeepSeek Flash v4.1, and 1-bit Quantization with Squeeze10-LLM. All share the same philosophy: “Apply the necessary model for the required processing.”

To conclude, by combining these technologies, the monthly cost of LLM can drop from 300,000 yen to between 30,000 and 50,000 yen. This results in an annual savings of over 3 million yen, with quality remaining virtually unchanged.

—

What’s Wrong with ‘All GPT-4o’?

First, let’s clarify the current situation.

The input token price for GPT-4o is $2.50 per 1M tokens, and the output is $10.00 per 1M tokens (as of OpenAI’s official pricing in 2024). In contrast, GPT-4o mini has an input cost of $0.15 per 1M tokens and an output cost of $0.60 per 1M tokens. There is a 15 to 17 times difference in cost.

Despite this, many SMEs use GPT-4o for all tasks simply because “GPT-4o is smarter.”

In fact, when analyzing corporate LLM usage, it turns out that about 70 to 80% of tasks are “simple processes”. These include generating standard responses, answering FAQs, format conversions, and simple summaries. These tasks can be adequately handled by GPT-4o mini or even lighter models. Only the remaining 20 to 30% require the reasoning capabilities of GPT-4o class.

In other words, paying over 15 times the cost for 80% of tasks is the essence of the “All GPT-4o problem.”

—

Smart Model Routing — Achieving 90% Cost Reduction Just by Distributing Tasks

Smart Model Routing is a technology that automatically determines the difficulty and type of input tasks and assigns them to the optimal model.

The mechanism is simple: a lightweight classification model (router) first receives the request and assigns it accordingly, saying, “This is a simple task, so send it to the mini model,” or “This is complex, so send it to GPT-4o.” The cost of the router itself is almost negligible.

Let’s look at some specific numbers.

• Assuming a company with 1 million requests per month
• All requests using GPT-4o: Monthly cost of approximately 300,000 yen
• After introducing routing (80% to mini, 20% to GPT-4o): Monthly cost of approximately 50,000 yen

Cost reduction to one-sixth. Annual difference of 3 million yen.

Moreover, the accuracy of routing has rapidly improved in recent research. Well-designed routers can correctly assign over 95% of tasks. Mistakes like using GPT-4o for simple tasks are almost eliminated, and the risk of assigning difficult tasks to the mini model is minimized.

The key point is that costs drop without sacrificing quality. From the user’s perspective, the quality of the responses remains unchanged; only the models are switched behind the scenes.

—

DeepSeek Flash v4.1 — The Shock of 1/437 KV Cache

Another significant development is the introduction of DeepSeek Flash v4.1.

The most notable feature of this model is its 1M token context window while having a global KV cache footprint of just 890 bytes, which is about 1/437 of traditional models.

What does this mean?

A large portion of the inference cost for LLMs is taken up by the KV cache (memory area for storing past inputs). The longer the context, the larger the KV cache grows, which puts pressure on GPU memory, slows down processing speed, and increases costs.

DeepSeek Flash v4.1 structurally resolves this bottleneck. Despite being a large model with 552 billion backbone parameters, it consumes an order of magnitude less memory.

The practical benefits for SMEs are as follows:

• Processing long documents becomes cheaper: Whether it’s contracts, manuals, or meeting minutes, the cost doesn’t skyrocket even with inputs of tens of thousands of tokens.
• Increased concurrent processing: Handle many requests with less GPU memory.
• Increased feasibility of self-hosting: Significantly lower operational costs on in-house servers or cloud GPUs.

As a rough estimate for monthly costs, if processing equivalent to GPT-4o is done with DeepSeek Flash v4.1, a cost reduction of 50 to 70% is expected. The monthly cost of 300,000 yen could drop to between 100,000 and 150,000 yen.

However, the effect is greater when combined with the aforementioned routing.

—

Squeeze10-LLM — Compressing Models by 90% with 1-Bit Quantization

The third technology is Squeeze10-LLM, which is a quantization technique that compresses model weights by 10 times.

Typically, LLM weights are stored in 16-bit (FP16). Squeeze10-LLM quantizes 80% of the weights to 1 bit and the remaining 20% to 4 bits. The average bit width is 1.6 bits. This is a 10-fold reduction from 16 bits to 1.6 bits.

It’s natural to wonder, “Won’t such heavy compression ruin the accuracy?” However, research results indicate that while traditional 1-bit quantization methods achieved 43% accuracy, Squeeze10-LLM has improved this to 56%. While it still doesn’t reach FP16 accuracy, it is sufficient for simple tasks.

The significance of this technology for SMEs is that it dramatically lowers the barrier for operating LLMs in-house.

• A 70B parameter model that previously required 140GB of GPU memory can now run with about 14GB.
• This means that a large model can run on a GPU costing just a few tens of thousands of yen.
• No need for cloud APIs. No need to expose data externally. No monthly subscription fees.

If operated through self-hosting, excluding initial investment (GPU purchase), the monthly inference cost, including electricity, can realistically be under 30,000 yen.

—

Monthly Cost Comparison — The Reality in Numbers

The annual cost of 3.6 million yen for using all GPT-4o can be reduced to between 360,000 and 600,000 yen with routing and appropriate model selection. The difference amounts to 3 million yen annually. Over five years, this totals 15 million yen. For SMEs, this difference could mean the ability to hire one additional employee.

—

So, What Should Be Done?

I’m not suggesting that you should implement all three technologies.

The first thing to do is simple: Review your company’s LLM usage logs to understand the distribution of task difficulties.

• What percentage of all requests truly require GPT-4o class capabilities?
• What percentage is made up of routine processes (summaries, classifications, formatting)?
• What is the average token count per request?

Once you understand this, the potential financial benefits of introducing routing will become clear. In many cases, the conclusion will be that “80% can be handled adequately by lighter models.”

As the next steps, it’s practical to consider the following in order:

1. Implement Routing (the easiest and most impactful option; simply add a thin layer to the existing API structure).
2. Evaluate DeepSeek Flash v4.1 (especially beneficial for companies that handle a lot of long documents).
3. Consider Self-hosting with Squeeze10-LLM (for companies that want to avoid external data exposure and eliminate monthly fees).

—

Let’s Talk About Reality

The costs of LLMs are going to drop even further. The competition for model efficiency has just begun, and it’s possible that the figures will change again in six months.

That’s why locking into “all GPT-4o” at this point is the worst strategy. Instead of relying on a specific model, it’s crucial to establish a structure that allows for switching models according to tasks.

With a routing mechanism in place, you can simply swap in new models as they are released. Models are consumables; routing is infrastructure. This shift in mindset can fundamentally change AI costs for SMEs.

What will you do with the 3 million yen saved? Hire someone, invest in a new business, or allocate it to the next AI experiment?

The ability to make that decision is a strength of SMEs. Unlike large corporations, there’s no need to circulate approvals for three months. Changes can be made starting next week.

I hope you start by opening your company’s LLM usage logs.