Conclusion

Let’s get straight to the point. The concept of AI ‘usage fees’ is beginning to crumble.

Many small and medium-sized enterprises are paying around $500 (approximately ¥75,000) per month for API usage of models equivalent to GPT-4. Customer inquiries, meeting minutes summarization, drafting sales emails—this usage-based pricing model means that costs increase the more you use it.

However, by combining three technologies announced in recent weeks, we are beginning to see the reality that a large model with 120 billion parameters can run on a gaming PC (with 16GB of VRAM) that costs around ¥150,000, generating tokens at a rate of 9 to 19 tokens per second.

The monthly running cost is just the electricity bill, roughly ¥1,500.

What was once ‘¥75,000 per month’ is now ‘¥1,500 per month.’ Small and medium-sized business owners should seriously consider the implications of this drastic change.

—

What Happened—Three Technologies in 30 Seconds

① RotaryQuant: Tri-Axis Compression to Fit 120B Models into 16GB

The latest research, “RotaryQuant,” proposes a method to quantize the weights of Mixture-of-Experts (MoE) models with different bit widths according to the role of each layer.

• Dense layers (always in use) → 4 bits
• Routed experts (activated only when needed) → 2 bits
• Shared experts (reused across the model) → 8 bits

Instead of compressing everything uniformly to 4 bits, it preserves bits where precision is needed and reduces them to the extreme in less frequently used areas. As a result, a 120B parameter model fits into 16GB of VRAM, achieving a token generation speed of 9 to 19 tokens per second, which feels almost equivalent to ChatGPT’s response speed.

The key point is the number “120B.” While it does not reach the 175B of GPT-3.5, it is sufficiently large for practical tasks—summarization, classification, Q&A, code generation—within the class of current open-source MoE models (like Mixtral 8x22B).

② SPECTRA: Compressing KV Cache to 2 Bits

Even if the model itself can fit on the GPU, there was a problem with the KV cache expanding during inference, consuming memory. When processing long documents, the cache alone could reach several gigabytes.

“SPECTRA” statistically analyzes the importance of each channel in the KV cache and allocates more bits to important channels and fewer bits to less important ones. Traditional uniform quantization would lead to a collapse in accuracy when reduced to 2 bits, but SPECTRA maintains practical accuracy even with 2-bit compression by rotating the coordinate system.

This allows for significantly reduced memory consumption even when handling long contexts (over 32K tokens). Applications like full contract reviews or summarizing past meeting minutes become feasible even on consumer GPUs.

③ WebGPU Optimization: Addressing the Reasons Why LLMs in Browsers Are ‘Slow’

The third technology is the optimization research for WebGPU inference running LLMs in browsers. The bottleneck was not the computational load but the overhead of dispatching instructions to the GPU. Each dispatch took several hundred microseconds, and when repeated hundreds of times, it resulted in delays measured in seconds.

This research significantly improves the inference speed in browser-based environments by reducing the number of dispatches through the fusion of multiple operations.

In other words, we are beginning to see a future where AI inference can run locally on employees’ browsers without even needing dedicated servers. For small and medium-sized enterprises lacking IT infrastructure, this could be the most realistic path to implementation.

—

Calculating the Breakeven Point for Small and Medium-Sized Enterprises

Abstract discussions aside, let’s verify the numbers.

Scenario: A company with 10 employees using AI for summarization, inquiry response, and email drafting

[Pattern A: API Usage (Current Situation)]

The more you use it, the more it increases. If inquiries surge during busy periods, the bills will skyrocket.

[Pattern B: Local Inference (Utilizing New Technology)]

A difference of ¥730,000 in the first year. A difference of ¥880,000 per year from the second year onwards.

The breakeven point is in the third month. The initial investment of ¥150,000 is recovered through the savings of ¥75,000 on API costs per month. By the third month, the investment will be recouped, and from the fourth month onward, it will be pure cost savings.

Of course, there will be setup efforts and trial and error in model selection. However, this is not about buying a ¥3 million server and hiring a dedicated engineer. It’s about a ¥150,000 PC and a setup time of 2 to 3 days. At this scale, it falls within the range of “let’s try it out first.”

—

What Happens Beyond ‘Lower Costs’

This is where the main topic begins. When costs decrease to one-fiftieth, the way we use it fundamentally changes.

When using an API with a pay-per-use model, the mindset was to “only throw it at AI when absolutely necessary.” Cost awareness limited usage.

When local inference costs ¥1,500 per month, it becomes normal for AI to be running continuously.

• Automatically classify all incoming emails, notifying urgency and response suggestions
• Read daily reports in full and automatically extract business risk signals
• Summarize phone conversations with customers in real-time and automatically input them into CRM
• Automatically generate drafts of estimates from past similar cases

Usage that previously felt wasteful under a pay-per-use model becomes virtually “free.”

This structure favors small and medium-sized enterprises over large corporations. Large companies are bound by existing IT infrastructure and contracts, which slow down the transition. Small and medium-sized enterprises can change “starting next week.” The speed of decision-making directly translates into a cost structure advantage.

—

Caution: The Gap Between ‘Working’ and ‘Usable’

There are also points to consider calmly.

1. Accuracy Issues
2-bit quantization comes with a degradation in accuracy. The RotaryQuant paper states that the accuracy drop on benchmarks is limited, but verification is needed in specific domains (such as medical and legal). While it may be sufficient for “in-house inquiry responses,” caution is warranted for tasks like “legal risk assessment of contracts.”

2. Model Selection
Currently, RotaryQuant is compatible only with MoE architecture models. It cannot be applied to all open-source models. However, MoE models like Qwen, DeepSeek, and Mistral are rapidly becoming more robust, providing ample practical options.

3. Setup Hurdles
Setting up the Python environment, downloading models, executing quantization—this is the biggest barrier for small and medium-sized enterprises that are not IT-savvy. However, this is also an area that can be resolved through “systematization.” Once set up, it can continue to operate.

—

So, What Should We Do?

I propose three actions.

① Right Now: Accurately Assess Your Company’s API Usage Fees
How much are you paying per month, and how many tokens are you using? Without this knowledge, you cannot calculate the breakeven point. If you are paying over ¥30,000 per month, it is worth considering a transition to local inference.

② Within One Month: Try Running a Small Model on a 16GB VRAM GPU
There is no need to aim for 120B right away. Running models in the 7B to 14B range (like Qwen2.5-7B, Llama 3.1-8B) using Ollama or llama.cpp can be done in half a day. The experience of “AI running on your company’s PC” should first be experienced by the management.

③ Within Three Months: Switch One Business Operation from API to Local
Whether it’s summarizing meeting minutes or drafting emails, switch one operation to local inference and measure the cost and quality differences. Once you can speak in numbers, you will be able to make the next investment decision.

—

Summary: The Era Where ‘AI’s Electricity Costs’ Become a Variable in Management Decisions

A 120B model runs on a PC costing ¥150,000. KV caches can be compressed down to 2 bits. Inference runs from local GPUs accessed via browsers.

Individually, these are “advancements in technology,” but when combined, they signify a structural change where the cost of using AI becomes just the electricity bill.

What was once ¥75,000 per month is now ¥1,500. This difference directly impacts the annual profits of small and medium-sized enterprises.

The important thing is not to let it end with “amazing technology has emerged.” Apply it to your company’s numbers, calculate the breakeven point, and try out one unit first. There’s no need to wait for technological evolution. The operational environment is already here.