Monthly GPU Costs of Tens of Thousands of Yen: Will You Continue to Pay?

Small and medium-sized enterprises (SMEs) want to use LLMs in-house. However, GPUs are expensive, and cloud API costs keep piling up. This has been the biggest bottleneck in AI adoption for SMEs.

A structural change is underway. Technology has emerged that compresses LLMs by three times, allowing them to run on less than 50MB of RAM. No GPU. No cloud. We are moving towards a world where reasonably functional LLMs can operate on personal computers or smartphones.

This discussion should not end with a simple acknowledgment of “amazing technology.” When costs dramatically decrease, the structure of the industry itself changes. Let’s consider specifically what changes for SMEs.

What’s Happening — The Current State of Compression Technology

First, let’s clarify the facts.

Recently, the rapid evolution of model compression technologies such as quantization and distillation has garnered attention. A notable example is 4-bit quantization in GGUF format. For instance, Meta’s Llama 3.1 8B model (originally about 16GB) shrinks to approximately 4.7GB when quantized to 4 bits. Moreover, smaller models can be reduced to tens or hundreds of megabytes by narrowing down the number of parameters.

Tools like “SPECTRA,” which can compress models by three times with a single click, are also starting to appear. However, to be frank, the accuracy drop varies significantly depending on “what to compress” and “which tasks to use it for.” They are not universally applicable.

What matters is not the maturity of the technology, but the direction. LLMs are undoubtedly moving towards being smaller, lighter, and cheaper. This is an irreversible trend.

Changing Cost Structures — How Much Can Be Saved?

Let’s compare the AI utilization costs for SMEs across three patterns.

Pattern 1: Cloud API Usage (Current Mainstream)

• OpenAI GPT-4o API: Input $2.50/1 million tokens, Output $10.00/1 million tokens
• Monthly usage of 1 million tokens costs about $12.50 (approximately 2,000 yen)
• However, for processing large volumes of internal documents, monthly usage of 10 million to 100 million tokens is not uncommon
• Monthly costs range from 20,000 to 200,000 yen, totaling 240,000 to 2,400,000 yen annually.

Pattern 2: In-House GPU Environment (On-Premises)

• NVIDIA RTX 4090: Approximately 300,000 yen (initial investment)
• Electricity costs: 5,000 to 10,000 yen per month
• Personnel costs for operation and maintenance: Several tens of thousands of yen per month
• Annual total: 500,000 to 800,000 yen (800,000 to 1,100,000 yen in the first year).

Pattern 3: Compressed Model + CPU/NPU (No GPU Needed)

• Operates on existing PCs (business PCs costing around 100,000 yen)
• Additional hardware costs: 0 yen
• Incremental electricity costs: Negligible
• Software: Free if open-source
• Annual total: Effectively 0 to several tens of thousands of yen.

A company that was paying 1 million yen annually under Pattern 1 could effectively reduce this to zero under Pattern 3. This difference is equivalent to one employee’s bonus for a small business with five employees.

“So, Can It Actually Be Used?” — Realistic Use Cases

This is the most important part. A 50MB model cannot return responses on par with GPT-4. That is an illusion.

So, what can it be used for? Here are some use cases where SMEs can see actual benefits:

1. Generation and Proofreading of Standardized Texts
Cover letters for estimates, standardized email replies, drafts for internal reports. These do not require advanced reasoning abilities. Small models are sufficient.

2. Internal FAQ and Manual Search
By combining internal business manuals and FAQs with RAG (Retrieval-Augmented Generation), small models can function as a “window to extract internal knowledge.” The expertise of veteran employees, which was previously siloed, can be made accessible to anyone.

3. Data Classification and Tagging
Automatically classifying customer inquiries into “complaints,” “questions,” or “estimate requests.” Extracting amounts and dates from PDF invoices. Tasks that involve “simple judgments but large volumes” are the forte of small models.

4. Use in Offline Environments
Construction sites, factories, farms. Even in places with unstable internet connections, if the model is installed on the device, it will work. Being independent of the cloud can actually be a strength in many industries.

Conversely, it is less suitable for creating complex analytical reports, generating advanced code, or writing creative texts. For the time being, these will remain in the domain of large models or cloud APIs.

The Tailwind of Edge NPUs

Another point to note is the proliferation of edge NPUs (Neural Processing Units).

Intel’s Core Ultra series, Qualcomm’s Snapdragon X Elite, and Apple’s Neural Engine. Many PCs and smartphones released after 2024 will be equipped with AI processing dedicated chips.

This means that “additional investment to run AI will be zero.” When it’s time to replace a PC, an AI execution environment will naturally be available.

Recent benchmarks have confirmed that laptops equipped with NPUs can run 7B parameter quantized models at a rate of 10 to 20 tokens per second. This is nearly the same speed as human reading. Practically, it is sufficient speed.

The True Meaning for SMEs

Let’s summarize the discussion so far.

• The compression technology for LLMs is advancing, bringing models that can run without GPUs closer to practical levels.
• The proliferation of NPU-equipped PCs eliminates the need for additional hardware investment.
• For automating routine tasks and leveraging internal knowledge, small models can adequately handle the workload.
• Cost savings could amount to several hundred thousand yen annually.

To perceive this as “what large companies are doing is now affordable for SMEs” is half correct and half wrong.

The essence is this: Large companies build large-scale AI systems. SMEs embed small AIs in their operations. The approaches are fundamentally different.

Instead of investing tens of millions of yen to build a customer support AI, SMEs can install a 50MB model on their office PC to automatically summarize notes after phone calls. This alone can free up 30 minutes a day. That’s 10 hours a month. 120 hours a year. At an hourly wage of 1,500 yen, that translates to an increase in productivity worth 180,000 yen.

It may not be flashy, but when these small changes accumulate, they can transform a company.

So, What Should You Do Starting Today?

For those who think, “This sounds interesting, but it has nothing to do with us,” here are three suggestions.

1. First, Try Running a Local LLM
Install [Ollama](https://ollama.ai/) and try running something like Phi-3 Mini. It takes about 15 minutes and is free. Experiencing AI running on your own PC is more convincing than a hundred explanations.

2. Write Down One “Simple Task You Do Every Day”
Creating email reply templates, summarizing daily reports, classifying inquiries. Anything is fine. Consider, “What would happen if I let AI do this?”

3. Choose an NPU-equipped Model When Replacing Your PC Next Year
There’s no need to replace it immediately. But when you do buy a new one, make sure to check if it has an NPU. Just that will expand your future options.

Will Zero GPU Costs Become the “Standard”?

To be honest, not all AI processing will become GPU-free. Tasks requiring high precision will still need cloud APIs or high-performance GPUs for the time being.

However, the era where 80% of the AI used in daily operations by SMEs can run without GPUs is fast approaching. By the end of 2025, conversations like “Wait, are you still paying for GPU costs?” may become commonplace.

What happens when costs decrease? AI will shift from being something to “implement” to something that “exists there.” Just like electricity or the internet, it will be used unconsciously. When we reach that point, the true utilization of AI by SMEs will begin.

Technology won’t wait. But it’s never too late to start. I encourage you to take just 15 minutes to run an LLM on your own PC.