“The Era of Expensive AI is Over”

Let’s get straight to the point. This week, three significant news items emerged simultaneously.

1. With Raspberry Pi 5 + AI HAT+, you can set up an LLM execution environment for 50,000 yen
2. With WebLLM, you can perform LLM inference just by opening a browser
3. With RoundPipe, you can bundle several home GPUs for distributed learning

The implications of these three developments are profound. In the three essential processes for running AI—”inference,” “deployment,” and “learning”—the path has opened up to eliminate the need for expensive servers or cloud billing.

The choice is now between paying tens of thousands of yen monthly for API usage or buying a 50,000 yen box to run it yourself. This option is now available even to small companies in rural areas with around ten employees. This is not just a matter of “AI becoming cheaper.” It signifies the beginning of a breakdown in the cost structure of AI infrastructure itself.

—

What Can Run for 50,000 Yen — The Power of Raspberry Pi 5 + AI HAT+

First, let’s clarify the facts. When you equip a Raspberry Pi 5 with the Hailo-made AI accelerator known as “AI HAT+”, it can run LLMs in an edge environment. Here’s the breakdown of the configuration and pricing:

• Raspberry Pi 5 unit: Approximately 10,000 yen
• AI HAT+ (26 TOPS): Approximately 30,000 yen
• microSD, power supply, case, and other peripherals: Approximately 10,000 yen
• Total: Approximately 50,000 yen

A board the size of a business card can perform 26 trillion calculations per second. This was previously the kind of processing that required GPU servers costing hundreds of thousands of yen just a few years ago.

Of course, there are limitations. There are constraints on the model sizes that can be run, and it is not realistic to deploy massive 70B-class models directly. However, for quantized models in the 7B to 13B range, inference can be performed at practical speeds. Tasks like responding to internal FAQs, summarizing meeting minutes, and drafting standard documents—most of the applications that small and medium-sized enterprises would “like to use first” can be adequately covered within this range.

What we should consider here is not what can be done, but rather what costs can be eliminated.

—

A Three-Year Cost Comparison with Cloud AI — A Numerical Reversal

For small and medium-sized enterprises using ChatGPT API or Azure OpenAI for business, a monthly running cost of around 20,000 to 50,000 yen is not uncommon. Let’s conservatively calculate it at 20,000 yen per month.

A difference of 650,000 yen over three years. This is not an amount that a company with ten employees can ignore. Moreover, while cloud AI operates on a pay-per-use model that increases costs the more it is used, your own environment allows for unlimited use. Whether running batch processes at midnight or experimenting on holidays, the additional cost is zero.

Another often overlooked issue is data privacy. Every time internal documents are sent to a cloud API, data is exposed externally. Personal information, client data, know-how—small and medium-sized enterprises, especially in rural areas, operate in a “face-to-face relationship” manner, making them highly sensitive to the risks of data leakage. If it all stays within your own Raspberry Pi, no data leaves your premises. While this may not be quantifiable, it provides a significant sense of security on the ground.

—

AI with Just a Browser — How WebLLM Changes Deployment Norms

Next, let’s talk about WebLLM. This is an engine that executes LLM inference in a browser. By utilizing the WebGPU API, it runs models on the user’s local machine GPU.

What is revolutionary here is that the concept of “deploying AI” disappears.

Traditionally, when trying to introduce AI tools within a company, it required setting up servers, designing APIs, creating front-end interfaces, implementing authentication, and configuring operational monitoring—this process could cost hundreds of thousands of yen if outsourced, or take several months if done in-house.

With WebLLM, employees only need to open Chrome and access a URL. No installation is required. No server is needed. IT staff do not have to “install software on everyone’s PCs”.

Of course, the speed of browser-based inference is slower compared to native execution. However, the trade-off of “slightly slower speed for zero deployment cost” is sufficient for many workplaces.

For small and medium-sized enterprises, the biggest bottleneck is not the performance of AI, but the hassle of deployment. WebLLM is set to eliminate that entirely.

—

Learning by Bundling Personal PC GPUs — The Impact of RoundPipe

The third development may seem the most understated, yet it has the potential to change the structure significantly.

RoundPipe is a pipeline scheduler that connects multiple consumer GPUs (like GeForce RTX 4090) over a network to efficiently distribute the training of large models.

Let’s pull some numbers from research. When using eight RTX 4090s to fine-tune models ranging from 1.7B to 32B parameters, a throughput improvement of 1.48 to 2.16 times has been confirmed compared to existing methods.

What’s remarkable about this is that the most costly process in AI—training—is becoming viable at a level where you don’t need to rent cloud A100/H100 GPUs to compete.

An RTX 4090 costs about 250,000 yen. Eight of them would total 2 million yen. You might think that’s expensive. However, renting eight NVIDIA H100s in the cloud can cost several thousand to ten thousand yen per hour, amounting to several million yen monthly. With an initial investment of 2 million yen, you can free yourself from monthly charges of several million yen. The payback period is just a few months.

Of course, it’s not realistic to pre-train cutting-edge models with hundreds of billions of parameters from scratch. However, what small and medium-sized enterprises want to do is fine-tune 7B to 13B models using their own data. Teaching the model to understand industry-specific terminology, generating text in the tone of past proposals—this level of work is well within reach with just a few RTX 4090s.

—

What Happens When All Three Come Together — The Option for “AI Self-Sufficiency”

Combining these three developments paints a clear picture:

1. Fine-tune models using your own data with several RTX 4090s (training)
2. Run the resulting model on a Raspberry Pi 5 + AI HAT+ as an internal server (inference)
3. Employees access it through WebLLM via their browsers or use it directly on their local machines (deployment)

No server room is needed. No cloud contracts are required. No IT department is necessary. No monthly fees.

It’s “AI self-sufficiency.”

This may not resonate with large corporations. They have ample IT budgets and dedicated ML engineers. However, for local manufacturing firms, construction companies, legal offices, and retail chains—those operating in a world with annual IT budgets of less than 1 million yen—this structural change holds decisive significance.

Until now, AI was considered the domain of large corporations. Now, it is becoming a world where it can run on a 50,000 yen box, a browser, and a gaming PC.

—

So, What Should We Do?

“I get it, it’s interesting. But where should we start?”

If asked this, the answer is simple.

Step 1: Buy one Raspberry Pi 5 + AI HAT+ and have one person in-house experiment with it. An investment of 50,000 yen.

Load a quantized 7B model (like Llama 3.1 8B) and automate just one internal routine task—like drafting emails, summarizing meeting minutes, or responding to FAQs.

Step 2: If it proves effective, consider fine-tuning it with your own data.

Prepare a PC equipped with an RTX 4090 (about 400,000 yen) and tune it lightly with LoRA. What would cost hundreds of thousands of yen to outsource for a “custom AI” can be created in-house.

Step 3: Deploy it internally with WebLLM.

Being browser-based means even employees with low IT literacy can use it. You can achieve a state where “everyone can use AI” with almost zero additional costs.

The key is to not aim for perfection from the start. Start with 50,000 yen, and if you see results, move on to the next step. If you don’t see results, you can withdraw with just a 50,000 yen loss. This is a vastly different risk compared to signing a yearly contract for cloud AI and then saying, “It wasn’t used.”

—

This is the Beginning of “Decentralized AI”

Every week, news comes out about large corporations in Tokyo stacking GPU clusters and announcing AI investments worth hundreds of millions of yen. That is a valid approach. However, there’s no need to fight on that battlefield.

With a 50,000 yen box, you can run AI that is specialized for your own business using only your own data. No data is sent outside. There are no monthly fees. If it breaks, you can simply replace it.

While large corporations boast about investing hundreds of millions of yen in AI, local factories are running AI for just 50,000 yen. All the components for that future came together this week.

Now, it’s just a matter of assembling it. I encourage you to buy one unit and give it a try.