Conclusion

Let’s get straight to the point. The true nature of “high GPU costs” is memory bandwidth.

The inference cost of AI is high. Monthly GPU expenses can reach several hundred thousand yen. This is out of reach for small and medium enterprises.

This is a common story. However, few can break down and explain “why it is high.”

The answer is simple. The majority of the inference cost for LLMs is determined not by “computation” but by “memory reads.”

What happens when you run a 13B parameter model? Each time you generate a token, you need to read all the model parameters (about 26GB) from GPU memory. When stored in FP16 (16-bit floating point), it amounts to 13 billion parameters × 2 bytes = approximately 26GB. This occurs for every token.

If you return an answer of 100 tokens, that results in 2.6TB of memory reads. All from just one response.

The computational performance of GPUs has been increasing year by year. However, the growth in memory bandwidth has not kept pace. The memory bandwidth of the A100 is about 2TB/s. This means it takes about 13 milliseconds to generate one token with a 13B model. This is the bottleneck and the real reason for high GPU costs.

The idea of “just renting a faster GPU” is a perspective for large corporations. What small and medium enterprises should consider is, “Is there a way to avoid reading 26GB?”

Strategy 1: Quantization—Compressing 26GB to 6.5GB

The most direct method is quantization.

If you quantize a 26GB model stored in FP16 (16-bit) to INT4 (4-bit), it shrinks to about 6.5GB. This reduces the amount of memory read to a quarter, resulting in approximately four times the inference speed on the same GPU. Alternatively, if the same speed is acceptable, it can run on a cheaper GPU.

Recently, a method called “BitDecoding-FP4” has gained attention. Traditional 4-bit quantization faced challenges with accuracy degradation, but this technique uses FP4 format and selectively retains high-precision values for important parameters. According to reports in academic papers, it can reduce memory usage by about 75% while keeping accuracy degradation within 1-2% compared to FP16.

In terms of specific costs, a case using an A100 (80GB) cloud GPU costing about 300,000 yen per month could potentially achieve equivalent inference performance with an A10G (24GB) for about 70,000 to 80,000 yen per month due to quantization. From 300,000 yen to 80,000 yen. That alone is about 1/4.

However, there is a caveat. Quantization sacrifices some model accuracy. While it is not an issue for generating responses for internal FAQs or standard phrases, quality checks are necessary for advanced inference or specialized document generation. You must test whether it is acceptable for your specific use case.

Strategy 2: Reusing Inference Cache—Avoiding Double Calculations for the Same Questions

Next, we have “reusing inference cache.”

Consider this: what percentage of questions posed to your internal chatbot are questions that have been asked before? Based on experience, 70-80% are similar questions repeated. Reading 26GB and recalculating each time is like making the same dish from scratch every time.

The method called “CPU-ARC (Adaptive Retrieval Cache)” eliminates this waste. It caches past inference results in CPU memory and returns results from the cache when similar inputs are received, avoiding the use of the GPU.

The numbers reported in papers are shocking. The inference cost during cache hits is 16 to 291 times cheaper than regular GPU inference. For applications with high hit rates—such as FAQ responses, standard inquiries, and internal knowledge searches—GPU operating time can be dramatically reduced.

What’s important here is the structure that “the operations of small and medium enterprises are more likely to benefit from caching.” General-purpose chatbots in large corporations have a vast variety of questions, resulting in lower cache hit rates. In contrast, internal tools and bots for specific tasks in small and medium enterprises have limited question patterns. This means they are more likely to achieve high cache hit rates.

If 70% of the inference can be handled by cache from a GPU cost of 300,000 yen, then GPU operation can be reduced to 30%. From 300,000 yen to 90,000 yen. When combined with quantization, costs can decrease even further.

Strategy 3: Deploying Edge Devices—Eliminating Cloud GPUs Altogether

The third option is to not use cloud GPUs at all.

Recent research on deploying LLMs on edge devices has rapidly advanced. Models in the 7B to 13B class, when quantized, can now run at practical speeds on devices such as Macs with Apple Silicon, NVIDIA Jetson, or even the latest smartphones.

For example, running a 4-bit quantized 13B model on an M2 chip-equipped Mac mini (around 100,000 yen) can generate about 10-15 tokens per second. It feels “a bit slow but bearable.” This is quite practical for internal use.

What would the costs look like in this case? One Mac mini costs 100,000 yen. Electricity costs are just a few hundred yen per month. An initial investment of 100,000 yen with almost zero running costs. Compared to a cloud GPU costing 300,000 yen per month, the break-even point is reached in just four months.

The benefits of edge deployment go beyond just cost.

• Data does not leave the premises. For small and medium enterprises handling customer information or internal secrets, this is significant. There is no need to send data to a cloud API.
• Latency becomes stable. There is no network delay, so response times remain consistent.
• No API limitations. There are no rate limits, sudden price increases, or risks of service discontinuation.

On the other hand, there are also disadvantages. Model updates become manual, and if used across multiple locations, you need a device for each location. It is not suitable for applications with many simultaneous connections. A scale of “5-10 users within the company” is the sweet spot.

Combining the Three Strategies

What’s crucial is that these are not mutually exclusive options.

Examples of realistic combinations:

• Internal FAQ → Run a quantized model on an edge device (Mac mini). Monthly cost is almost zero.
• Customer support chatbot → Use a cloud GPU with a quantized model + cache reuse. Monthly cost drops from 300,000 yen to 60,000 yen.
• Advanced document generation (used occasionally) → Use a cloud API on a pay-as-you-go basis. Monthly cost in the thousands of yen.

With this combination, the AI inference costs that were over 300,000 yen per month can drop to a total of 70,000 to 80,000 yen. Less than 1/4.

So, what should we do?

Before lamenting about “high GPU costs,” first analyze your company’s inference patterns.

1. Measure the repetition rate of questions. If over 70% are similar questions, caching will be dramatically effective.
2. Count the number of simultaneous connections. If it’s fewer than 10 people, edge devices are likely sufficient.
3. Assess accuracy requirements. For internal use, the accuracy degradation from quantization is often acceptable.

Just understanding these three points will reveal the optimal configuration.

Technological evolution is rapid. Six months ago, saying “I can run a 13B model locally” would have drawn laughter. Now, it runs on a 100,000 yen Mac mini. It might even run on smartphones a year from now.

If you understand the bottleneck of memory bandwidth, you will be able to recognize any upcoming technology as “this is a technology that reduces read amounts” or “this is a technology that makes caching smarter.”

For small and medium enterprises to succeed with AI, it is not about renting the same GPUs as large corporations. It is about creating a system that “avoids reading 26GB” tailored to your business patterns.

Start by reviewing one week’s worth of logs from your company’s chatbot. How many times have the same questions been asked? That number represents your company’s potential for AI cost reduction.