Conclusion

To put it simply, we are entering an era where AI can operate for under 5,000 yen per month.

I want to ask small and medium-sized enterprises that are paying 100,000 yen per month for cloud APIs: “Do you really need that API?”

As we enter 2025, three technologies are maturing simultaneously: a sixfold increase in inference speed, streaming inference on edge devices, and model compression through LoRA. When these three elements converge, the cost structure for the option of “running AI on in-house PCs” fundamentally changes.

In this article, we will estimate the cost break-even point between “cloud API vs. in-house PC operation” using specific numbers. To give you the answer upfront, if you use about 100,000 tokens per month, you can recoup your investment in in-house PCs within 3 to 6 months.

—

Sixfold Inference Speed — What FlashMap Has Changed

The “FlashMap” developed by Polymathic AI and Princeton University has increased the decoding speed of LLMs by up to 6.14 times.

Traditional autoregressive decoding generates one token at a time sequentially. FlashMap uses continuous flow maps to draft 16-token blocks in a single forward pass. In other words, the task that was previously done “one by one” has now changed to “throwing 16 at once.”

What does this mean for small and medium-sized enterprises?

A sixfold increase in speed means that the same GPU can handle six times the processing. In other words, the required GPU specifications can be reduced to one-sixth.

Let’s look at it specifically. When running a model like Qwen3-8B with FlashMap, throughput improves dramatically while maintaining accuracy comparable to traditional methods on the GSM8K benchmark. It opens up the possibility of achieving practical speeds with consumer GPUs in the RTX 4060-4070 class, rather than needing high-end GPUs like the H200.

The retail price of the RTX 4070 is about 90,000 yen. This serves as a guideline for the initial investment required to “run AI on in-house PCs.”

—

Inference on Edge SoCs — A Future Where PCs May Not Be Needed

The discussion goes even further. Research on streaming VLM inference on edge SoCs (System on Chip) is nearing practical levels.

The key point of this technology is to operate three computational units—CPU, GPU, and NPU—concurrently. Traditionally, inference has been solely reliant on the GPU, but this design utilizes all idle resources within the chip.

In practical measurements, throughput improvements of up to 19.6% have been confirmed on devices like the Apple M4 Max and Qualcomm QCS6490. While 19.6% may sound modest, this performance gain is achieved with “zero additional cost.” By simply tapping into the capabilities of existing hardware, processing speeds can increase by about 20%.

Considering this in the context of small and medium-sized enterprises, it looks like this:

• Automatic summarization of meeting minutes runs on in-house Macs (with M4 chips).
• Image inspection AI can perform real-time processing on edge devices in factories.
• Customer service support AI responds on tablets in stores.

There’s no need to send data to the cloud. There’s no network latency. There are no communication costs. And data doesn’t leave the company. For local manufacturing and retail businesses, the fact that data stays in-house is actually a significant advantage that outweighs costs.

—

LoRA Compression — Moving Away from “One Model Does It All”

LoRA (Low-Rank Adaptation) is a technology that allows for rewriting only parts of a large model to specialize it for specific tasks. This eliminates the need to maintain multiple full models.

For example, if you have one base model with 8 billion parameters, you can achieve the following by simply swapping out LoRA adapters:

• An adapter for creating estimates (tens of MB)
• An adapter for replying to customer emails (tens of MB)
• An adapter for summarizing technical documents (tens of MB)

The VRAM usage of the base model is about 16GB (when using 8B and 4-bit quantization). LoRA adapters range from a few tens of MB to a few hundred MB each. Even with the 12GB VRAM of the RTX 4070, it can run sufficiently when combined with 4-bit quantization.

In validation data involving 78 LoRA adapters, the accuracy drop compared to full fine-tuning is limited to about 1-3%. This means that at a practical level, AI can operate with “almost the same quality” using overwhelmingly fewer resources.

This is critically important for small and medium-sized enterprises. Large companies can afford to set up dedicated models for each application. Small and medium-sized enterprises cannot. That’s why the configuration of “one base model + multiple LoRA adapters” is effective. It allows for the systematization of business know-how that tends to become personalized.

—

Calculating the Cost Break-Even Point

Now we get to the main topic. Which is cheaper: cloud APIs or in-house PC operation? Let’s calculate it specifically.

In the Case of Using Cloud APIs

• GPT-4o class API: Input $2.5/1 million tokens, Output $10/1 million tokens
• Assuming monthly usage of 5 million tokens input and 2 million tokens output (the level used by 10 employees in their daily work)
• Monthly cost: Input $12.5 + Output $20 = approximately $32.5 (about 5,000 yen)
• However, costs increase linearly as usage increases.
• For monthly input of 50 million and output of 20 million tokens (full-scale operation), monthly cost is approximately 50,000 yen.

In-House PC Operation

• GPU: RTX 4070 (approximately 90,000 yen)
• PC body (desktop capable of housing a GPU): approximately 150,000 yen
• Total initial investment: approximately 240,000 yen
• Monthly electricity cost: GPU running at 200W × 8 hours/day × 22 days × 30 yen/kWh ≈ approximately 1,060 yen
• Maintenance and software costs: approximately 2,000 yen/month (assuming open-source software)
• Total monthly running cost: approximately 3,000 yen

Break-Even Point

Even for light usage, you can recoup your investment in a year. For medium usage or higher, you can recover it in two months.

Moreover, with the implementation of FlashMap’s sixfold speedup, the processing capacity that can be handled by the same GPU will further increase. Performance improvements without additional hardware will also be realized through edge SoC optimization. LoRA compression allows for multiple applications to be covered by a single device.

In other words, AI can operate in-house for around 3,000 yen per month.

—

So, What Should We Do?

I propose three steps.

1. First, try it out (Cost: 0 yen)
Install Ollama on your existing PC and run Qwen3-8B. It can operate with CPU inference even without a GPU. It may be slow, but you can experience “AI running on in-house PCs.”

2. Make a small investment (Cost: 100,000 to 250,000 yen)
Prepare one PC equipped with an RTX 4060 or 4070. Use LoRA adapters to create models specialized for your business. Start with routine tasks you perform daily, such as summarizing meeting minutes, drafting emails, and checking estimates.

3. Systematize it (Cost: Operational Effort)
Once it’s running, make it accessible to anyone in the company. Set up a web interface, create a Slack bot, etc. Don’t let it become personalized. This is the most important part.

FlashMap and edge SoC optimizations are still in the research stage for some aspects. However, the foundational technology for “running LLMs on in-house PCs” is already at a practical level as of 2025.

There’s no reason to wait. You can start building an AI infrastructure for around 3,000 yen per month today.

—

Additional Notes: Points to Be Aware Of

Of course, there are risks associated with in-house PC operation. I’ll be honest about it.

• Accuracy Limitations: Local models in the 8B class do not match GPT-4o or Claude 3.5 Sonnet. They are not suitable for complex reasoning or advanced coding. A compromise is necessary: “covering 80% of daily operations.”
• Operational Burden: You will need to handle model updates, security patches, and hardware failures in-house. Companies without IT personnel may find it more realistic to start with cloud APIs.
• Scalability: If the number of users increases to 50 or 100, a single PC may not be able to keep up with processing demands. At that stage, a hybrid configuration with the cloud should be considered.

Even so, for small and medium-sized enterprises with fewer than 10 employees, the cost advantage of in-house PC operation as a starting point for “using AI in-house” is overwhelming.

3,000 yen per month or 50,000 yen per month. If the same tasks can be accomplished, the answer is clear.