Conclusion First

The era of running a 35B model on a personal computer has arrived.

“I want to use AI in-house, but I can’t afford 300,000 yen a month for a GPU server.”

This is the most common sentiment expressed by small and medium-sized enterprises (SMEs) in rural areas. It’s understandable; for a company with sales in the tens of millions, a fixed cost of 300,000 yen a month can be crippling.

However, the combination of three technologies announced in recent weeks has the potential to turn this situation upside down:

• The engine “Edge0” that runs a 35B model at 20 tokens/second directly from an SSD
• “BITCOS,” which breaks the compression limits of ternary LLMs
• The quantization method “REQAP” that reduces edge device memory usage by up to 62%

All three technologies come from different research teams, but they share a common goal: “Running practical-level AI without a GPU.” What does this mean? The monthly cost could drop to zero. Is there any better news for SMEs?

—

Edge0: A Paradigm Shift Using SSDs Instead of GPUs

Let’s start with the most impactful technology.

To run a large language model (LLM) of the 35B class, you typically need a GPU with around 80GB of memory. An NVIDIA A100 costs over 2 million yen, and renting it in the cloud can cost between 200,000 and 300,000 yen per month.

Edge0 allows this model to be run directly from a standard PC’s SSD. The required active memory is only 3GB, and the entire machine only needs 24GB of RAM.

Why has this become possible? The key lies in the characteristics of the MoE (Mixture-of-Experts) architecture. MoE models do not use all parameters simultaneously; only a subset of “experts” is activated based on the input. Edge0 capitalizes on this.

Specifically, it predicts which experts will be needed for the next layer while processing the current layer and pre-reads from the SSD. Traditional offloading methods read data “when needed,” which can cause delays, but Edge0 avoids this issue by enhancing prediction accuracy.

As a result, the 35B MoE model runs at 20 tokens/second, which feels almost equivalent to the standard response speed of ChatGPT. And this is achievable on a personal computer.

However, there are some caveats. The 20 tokens/second figure applies to the MoE model, and the same speed may not be achievable with a Dense model where all parameters are always active. Additionally, the SSD’s read speed can become a bottleneck, so an NVMe SSD (with sequential reads of over 3,000MB/s) is practically a must. Using a cheaper SATA SSD could significantly degrade performance, so this should be verified before implementation.

—

BITCOS: Breaking the “Physical Wall” of Model Size

Next, let’s discuss ternary LLMs.

A ternary LLM is an ultra-lightweight model that restricts its weights to three values: “-1, 0, +1.” BitNet is a representative example, and it can be less than one-tenth the size of a standard 16-bit model.

Until now, it was believed that storing ternary weights required a theoretical lower limit of 1.585 bits/weight due to information theory—this is the mathematical wall of log₂(3) ≈ 1.585.

BITCOS has broken through this wall.

How did it do this? It leverages the statistical bias that the occurrence of “0” is high in ternary models. By compressing blocks with a high frequency of zeros, it has demonstrated that the sparsest models can be compressed to 1.485 bits/weight.

At first glance, a difference of “0.1 bits” might not seem significant. But let’s calculate it for a 35B model:

• 1.585 bits × 35 billion parameters = approximately 6.93GB
• 1.485 bits × 35 billion parameters = approximately 6.50GB
• Difference: approximately 430MB

The 430MB difference directly impacts the read time from the SSD. In an SSD-serving environment like Edge0, a smaller model size reduces I/O bottlenecks and increases inference speed. The combination of BITCOS and Edge0 has the potential to create a multiplicative effect rather than just an additive one.

However, it’s important to note that BITCOS’s compression effects are maximized only for models with a high density of zeros. Not all ternary models can achieve 1.485 bits. It’s crucial to verify the sparsity for each actual model.

—

REQAP: AI That Works Even When Hardware Fails at the Edge

The third technology is somewhat different but extremely important for SMEs.

REQAP is a quantization method that gives AI models on edge devices resilience against hardware failures. It sets different bit widths for each layer, reducing memory usage by up to 62% and computational cycles by up to 56%.

Why is this important?

When SMEs implement AI in the field, they cannot expect ideal conditions like those in a data center. They must operate in environments with high temperatures in factories, vibrations on-site, and 24-hour operational store terminals—conditions that make hardware prone to degradation.

Traditional quantization methods assumed “normal hardware.” There were cases where just one bit flip could cause the model’s output to collapse. REQAP identifies bit positions that are prone to failure and adds redundancy there, allowing the model to maintain accuracy even if the hardware is somewhat damaged.

This is the final piece needed to “place AI in the field.” Local AI that does not rely on the cloud is also resilient to communication failures. If REQAP enhances resilience against hardware failures, it will enable “non-stop AI” in the workplaces of SMEs.

—

So, How Much Will It Cost?

This is the most crucial part. Let’s do some specific calculations.

For Cloud GPU (Monthly)

For Edge0 + In-house PC Operation (Initial Investment Only)

The monthly cost of 300,000 yen can be reduced to an initial investment of 150,000 yen plus 3,000 yen per month. This means you can break even in the first month.

Of course, this calculation is based on the specific configuration of “running a 35B MoE model using SSD serving” and may not apply to all use cases. It may not be suitable for tasks requiring GPT-4 class accuracy. However, many routine tasks in SMEs, such as internal FAQs, meeting minutes summarization, generating standard documents, and initial inquiry sorting, can be adequately covered by a 35B class model.

—

What This Trend Means

“You need a GPU to use AI” and “You need the cloud to use AI”—these common beliefs are beginning to crumble.

What’s important is that this is not just a cost-cutting discussion. The structure is changing.

In a world where cloud GPUs are the norm, financially strong large companies had a significant advantage. A company that could set up a 1 million yen GPU cluster had no competition against a small factory with only a 50,000 yen IT budget.

However, when the execution cost of AI drops to the level of “one PC + SSD,” this gap disappears. A 150,000 yen PC is affordable even for a company with ten employees. Moreover, since it’s open-source, there is no vendor lock-in.

The area where large companies have been winning through “economies of scale” is being leveled by technological advancements. This presents the biggest opportunity for SMEs in recent years.

—

What You Should Do Right Now

Don’t just end with “That’s an interesting technology.” Take concrete action.

1. Check the specs of your current PC. If it has 24GB of RAM and an NVMe SSD, you may be able to try it out starting today.
2. Watch the developments of Edge0. While it is currently in the research stage, there are signs it will be released as OSS. You should validate it immediately once it is released.
3. Identify “tasks that can be sufficiently handled by a 35B model” in your business. Not everything needs to be thrown at GPT-4. 80% of tasks can run on a local 35B model.
4. Start verifying the accuracy of ternary models (BitNet series). If BITCOS’s compression becomes practical, it will run on even smaller machines.

There is no need to wait for all the technologies to come together. Start by testing one task. That’s all you need to do.

The world where the monthly cost of 300,000 yen becomes 3,000 yen is no longer just a story in a paper; it’s right on your desk.