GPUが買えないなら、使わなければいい

When it comes to introducing AI, the question of “what to do about GPUs” always arises.

NVIDIA’s H100 costs over 5 million yen per unit. Renting cloud GPUs can cost tens of thousands to hundreds of thousands of yen monthly. The moment a small or medium enterprise president says, “We want to implement AI too,” they hit this wall.

However, the situation is fundamentally changing. Technologies that allow AI to operate without GPUs are beginning to reach practical levels.

AI applications are now running on a 5,000 yen Raspberry Pi Zero. New caching technologies have reduced inference costs by up to 291 times using only CPUs. Large Language Models (LLMs) can run on edge devices.

This is not just a story about costs coming down; it is about “the elimination of entry barriers.”

—

5,000円の基板でAIが動くという事実

The Raspberry Pi Zero is a single-board computer that fits in the palm of your hand, priced at about 5,000 yen.

AI applications are actually starting to run on this board. For example, the project “Skrongle Mini” has realized a self-hosted AI course generator on the Raspberry Pi Zero, without any dependence on the cloud. Data never leaves the device, and there are no monthly fees.

To be honest, it would be unrealistic to expect GPT-4 level general responses from this board due to its limited processing speed and memory.

However, what we should consider is not a “one-size-fits-all AI” but rather a “small AI specialized for specific tasks within the company.”

• Automated responses to standard questions at the reception
• Searching and summarizing internal manuals
• Initial assessments of inspection images
• Extracting keywords from daily reports

For these types of “repetitive patterns,” edge devices are more than sufficient. The AI that small and medium enterprises truly need is precisely this “small and reliable automation.”

—

推論コスト291分の1——ARCという発明

Another technology that cannot be overlooked is ARC (Agentic Reasoning Cache).

In simple terms, the mechanism works like this: the inference processes solved by an LLM are cached (saved), and when a similar problem arises, that cache is reused. There is no need to run a GPU from scratch every time.

The reported numbers are astounding.

• Cost: 16 to 291 times cheaper than before
• Performance: Maintains levels comparable to traditional LLMs
• Operating environment: Can run on CPUs only

Let’s put the figure of 291 into concrete terms.

For example, if inference processing costs 50,000 yen per month using a cloud API, using ARC would reduce that to between 172 yen and 3,125 yen for the same processing. No GPU is needed; only a CPU is sufficient.

This is not just about cost reduction; it is about the disappearance of the cost structure itself.

Of course, ARC won’t work for every task. Caching is effective for cases where “similar patterns of inquiries are repeated.” However, consider this: most AI inquiries that arise in small and medium enterprises are precisely those “repetitive patterns.”

Customer inquiries, standard internal questions, daily report analysis—these all come in with nearly the same structure every day. The area where ARC will be most powerful is precisely here.

—

コスト比較：クラウドAPI vs エッジAI

Let’s compare the numbers in practice.

【クラウドAPI利用の場合】

• Monthly API costs: 30,000 to 100,000 yen (depending on usage)
• Annual costs: 360,000 to 1,200,000 yen
• Data: Sent to external servers
• Dependency: Risks of vendor price changes and service discontinuation

【Raspberry Pi Zero＋エッジAIの場合】

• Equipment cost: About 5,000 yen (initial investment only)
• Electricity cost: About 300 yen per month (3,600 yen annually)
• Annual costs: 8,600 yen in the first year, 3,600 yen from the second year onward
• Data: Completely processed locally
• Dependency: Self-managed, no vendor lock-in

【ARC併用の場合】

• Initial LLM inference costs + cache setup: Several thousand to tens of thousands of yen
• Operational costs: A fraction of the traditional costs
• Required hardware: Only a CPU machine

An annual API cost of 1 million yen can be reduced to under 10,000 yen. What could be done with that difference? You could hire an employee or invest in a new business.

—

「GPU不要」が壊すもの、生まれるもの

Now we come to the main point.

The fact that GPUs are no longer necessary means that the entry barrier for AI has shifted from “financial strength” to “design capability.”

Until now, large companies have monopolized the benefits of AI by building GPU clusters worth tens of millions of yen and processing vast amounts of data. Small and medium enterprises could only watch from the sidelines or sign expensive SaaS contracts.

However, the moment GPUs are no longer needed, the axis of competition changes.

It will no longer be about “how much can you invest?” but rather about “how deeply do you understand your own business?”

Large companies excel at creating general-purpose AI. However, small and medium enterprises that know their operations well will be overwhelmingly stronger at creating highly specialized AI, such as “identifying this defect pattern in this product on this production line in our factory.”

Moreover, with edge AI, there is no need to send data outside. There is no need to share data filled with the company’s know-how with cloud vendors. This also protects the source of competitive advantage for small and medium enterprises.

—

「で、結局どうすればいいの？」

Let’s move on from abstract discussions. Here are three things small and medium enterprises can start doing today.

1. Identify your company’s “repetitive tasks”

ARC is effective for “repetitive similar patterns.” Start by listing tasks within the company that involve making similar judgments or responses daily or weekly. Inquiry responses, inspections, daily report aggregations, quote preparations—there are bound to be some.

2. Buy one Raspberry Pi and try it out

It costs 5,000 yen. Even if it fails, that’s just the cost of three lunches. By running open-source projects like Skrongle Mini, you can get a feel for “what edge AI can do and what it cannot.”

3. Take stock of your cloud API expenses

For companies already using AI services, check how much they are paying monthly. How many of those processes can be standardized? If you can identify areas that can be replaced with ARC or edge processing, that alone could significantly reduce costs.

—

注視すべき3つの動き

Finally, here are three points to watch in the future.

① The competition for lightweight edge LLMs

The development race for compact, high-performance LLMs like Microsoft’s Phi and Google’s Gemma Nano is accelerating. The day when these can run practically on devices the size of Raspberry Pi is not far off.

② Standardization of inference caching technologies

If technologies like ARC become widespread, the cloud AI business model of “charging for every inference” will be shaken. This is a welcome change for small and medium enterprises, but existing vendors’ movements should be monitored closely.

③ The emergence of industry-specific edge AI packages

We can expect to see edge AI packages optimized for specific industries, such as inspection AI for manufacturing, inventory prediction AI for the food industry, and monitoring AI for nursing facilities. This presents business opportunities for small and medium enterprises.

—

GPUの壁が消えた先にある世界

Looking back at the history of AI utilization, it has always been a game where “those who have computational resources win.”

However, with the practical application of GPU-free AI technologies, the rules of this game are being rewritten. With a 5,000 yen board and a deep understanding of your own business, we are entering an era where even small and medium enterprises can benefit from AI.

The question is whether you will notice this change and take action.

The technology is starting to come together. All that remains is to “try it out first.”