Conclusion First: The Era Without GPUs Has Arrived

“To implement AI, you must first buy GPU servers”—this common belief is quietly crumbling.

A full-size model with 120 billion parameters. Normally, it would consume a monstrous 63GB of memory. Yet, it runs on a 16GB M1 Pro MacBook. No GPU investment required. No cloud API contracts needed. Just the laptop at hand is sufficient.

What does this mean? It means that small and medium-sized enterprises (SMEs) that have been paying hundreds of thousands to millions of yen annually for API usage can now run equivalent AI inference with just the equipment they already have.

“Not being able to buy GPUs” is no longer a handicap. In fact, the agility of not having unnecessary fixed costs becomes a weapon. This structural change can be understood through three technological trends.

—

ExpertCache—The “Magic” That Runs a 63GB Model on 16GB

The project “ExpertCache” has been garnering attention on Hacker News. Simply put, it operates on the idea of loading only the “necessary parts” of a massive model into memory.

In models using the Mixture of Experts (MoE) architecture, not all parameters are required simultaneously during inference. ExpertCache predicts which expert (sub-model) will be needed next and preloads it from the SSD into memory. In other words, instead of loading the entire 63GB model into RAM, it continuously swaps only the necessary few GB at high speed, maintaining full-size inference quality even with 16GB of RAM.

The key here is “speed.” If it merely read from the SSD in a piecemeal fashion, it would be too slow to be practical. ExpertCache maintains a usable perceived speed through predictive prefetching. According to the paper, speed improvements of several times compared to naive offloading methods have been reported.

Cost Structure Turned Upside Down

Let’s consider some specific numbers.

• Using cloud APIs (GPT-4 class) for business: Monthly fees range from 30,000 to 100,000 yen. Annual costs amount to 360,000 to 1,200,000 yen.
• Building a GPU server in-house: A machine equipped with an NVIDIA A100 costs at least 2 to 3 million yen, not including electricity and maintenance costs.
• Using ExpertCache with your existing MacBook: Additional cost, zero.

“From 3 million to 0 yen”—this disparity changes the decision-making landscape for SMEs.

Of course, at this point, throughput (the amount of processing per unit time) does not match that of cloud GPUs. It is not suited for applications that need to handle a large number of requests simultaneously. But consider this: how often do small businesses in rural areas need “100 requests per second”? For daily tasks like summarizing meeting minutes, drafting estimates, or classifying inquiry emails, a MacBook is more than sufficient.

Companies realizing that “cloud GPUs were overkill for our scale” will soon become commonplace.

—

JEDEC’s New Memory Standard—The Beginning of Hardware Price Disruption

The second change is occurring in the very cost structure of hardware.

JEDEC, which sets industry standards for semiconductors, has announced a new standard called “SPHBM4.” There are two technical points to note.

1. It adopts a 512-bit wide interface, increasing data transfer speeds.
2. It eliminates the expensive silicon interposer, significantly reducing manufacturing costs.

An interposer is a substrate used to connect chips at high density. It is a necessary component in current HBM (High Bandwidth Memory) but has accounted for a large portion of manufacturing costs. By making this unnecessary, SPHBM4 has the potential to reduce AI memory prices by 20-30%.

This directly impacts the “price of PCs that can run AI.”

Currently, machines equipped with memory suitable for AI inference are expensive. However, if memory prices drop by 20-30%, we could see the arrival of a time when “32GB laptops capable of running AI inference comfortably” can be purchased for just over 100,000 yen within a few years. If technologies like ExpertCache allow for operation with just 16GB, then machines with cheaper 32GB or 64GB of memory will run even larger models, even faster.

The simultaneous progress of hardware cost reduction and software efficiency is the essence of the current situation. If only one of these were happening, it would end up being just “an interesting technology.” Because both are occurring simultaneously, we can say that “the structure is changing.”

—

Go 1.27’s SIMD Support—AI Inference Becomes “Run Anywhere”

The third change is at the programming language level.

The upcoming version 1.27 of the Go language is expected to introduce a SIMD package. SIMD, or Single Instruction, Multiple Data, is a CPU feature that processes multiple data points simultaneously with a single instruction. This will be key to processing matrix operations, which are central to AI inference, quickly using only the CPU without a GPU.

Until now, speeding up AI inference on CPUs required writing optimized code for each CPU in C/C++. Separate code for Intel and ARM. This high development cost was not practical for the development teams of SMEs.

The Go SIMD package addresses this issue by ensuring that “the code written runs fast on any CPU.” It provides a cross-platform, portable SIMD implementation.

Implications for SMEs

What does this change?

• The need for specific servers to run AI will disappear. Any Windows PC, Mac, or Linux server in-house can run the same AI inference binary.
• It becomes less specialized. Go’s simplicity means that maintenance can be done without specialized knowledge like that required for C++. The risk of a system becoming inoperable if one engineer leaves is reduced.
• Deployment becomes easy. Go compiles into a single binary. There’s no hell of dependency libraries. You can carry the AI inference environment on a single USB drive.

We are moving from an era where “dedicated infrastructure engineers are needed to run AI” to one where “you just need to place the built binary.” This represents a significant reduction in the barrier to entry for SMEs.

—

What Happens When These Three Changes Overlap

Let’s summarize the three changes so far.

These three are separate news items. However, when viewed together, they point in one direction.

“The costs of money, people, and things necessary to run AI are all decreasing simultaneously.”

This is not just a story about “AI becoming cheaper.” It is a structural story about “no longer needing to be a large corporation to use AI.”

—

So, What Should SMEs Do Now?

I don’t want this to end with “Interesting technology has emerged.” Here are three specific actions that can be taken starting today.

1. Try running a local LLM on your MacBook.

Even without ExpertCache, tools like llama.cpp or Ollama can run models in the 7B to 13B class right now. First, I want you to experience “AI running on your company’s PC.” You cannot make judgments without experience.

2. Inventory how much you are paying for AI APIs monthly.

Review your cloud API invoices. If you are paying more than 30,000 yen a month, it’s worth considering whether there are applications that can switch to local inference and reduce costs to zero.

3. Discard the internal assumption that “AI is impossible without GPUs.”

This is the most important point. Technological changes are rapid. The common sense from six months ago is no longer common sense. “We don’t have a GPU server” or “We don’t have the budget”—the premise behind these statements has already crumbled, and it is essential for business leaders to recognize this.

—

The Turnaround Has Already Begun

Large corporations spend tens of millions of yen to build GPU clusters, establish dedicated teams, and take six months to implement AI through internal coordination.

SMEs can install ExpertCache on their MacBooks tomorrow and start using it next week.

Which is faster? Which is more agile?

The era where the “have-nots” gain an advantage is quietly but surely arriving.

What matters is whether you notice this change and take action. Technology does not wait. But technology is open to everyone.

First, try running AI on your PC. That is the first step.