Conclusion First. Three Technologies That Will Change AI’s ‘Usage Fees’ by Orders of Magnitude Have Emerged Simultaneously

When small and medium enterprises (SMEs) integrate the ChatGPT API into their operations, the most significant factor is the monthly API usage fee. For a company with ten employees, this can range from tens of thousands to over a hundred thousand yen per month. While it is not unaffordable, it raises the recurring question, “Is this cost really justified?”

This structure may be fundamentally overturned. Three papers published in 2024 propose methods to reduce AI inference costs by orders of magnitude through different approaches.

“Prompt Compression,” “Optimal Distribution of Quantization,” and “Output Layer Acceleration” — the names of the technologies are secondary. What matters is how much the monthly expenses for SMEs will change. Let’s break it down with specific numbers for each.

—

1. A 3000-Token Prompt Becomes ‘1 Token’

Paper: “Learning a Single Token to Replace Long System Prompts in LLMs”

First, let’s clarify what happens when SMEs use the API today.

For example, a system prompt like “You are a polite customer service representative. Please respond according to the following rules…” can be long, totaling 2000 to 3000 tokens. The input token price for GPT-4o is $2.50 per million tokens. Sending a 3000-token prompt in one call costs about $0.0075 just for the prompt.

If called 10,000 times a month, the cost for the prompt alone would be $75 (about 11,000 yen). Since user input and output tokens incur separate costs, the actual monthly expense is higher.

The paper proposes a “Behavior-Equivalent Token (BE Token)” that compresses that 3000-token system prompt to just 1 token. The compression ratio can be as high as 3000 times, while maintaining approximately 98% of the original prompt’s performance.

What Would This Mean in Monthly Expenses?

Let’s recalculate under the same conditions. 3000 tokens → 1 token. The cost for the prompt drops from $0.0075 to $0.0000025. For 10,000 calls a month, this results in $0.025 (about 4 yen).

From 11,000 yen to 4 yen. This is not a rounding error but a structural change.

Of course, there are caveats. Training BE Tokens requires prior fine-tuning, and currently, they cannot be used directly with OpenAI’s API. However, if the API providers adopt the direction of “making prompt costs nearly zero,” it could rapidly gain traction. Whether Google will implement it first or OpenAI will follow is something to watch closely.

Moreover, more significant than the cost reduction is the liberation of the context window. The 3000 tokens previously consumed by the prompt will be entirely available. This allows for longer user inputs and increases the number of reference documents passed in RAG. In practical terms for SMEs, it means they can fit the entire manual into the prompt without running out of space.

—

2. Process 11% More on the Same GPU — Optimal Distribution of Quantization by Layer

Paper: “A Method for Layer Bit-Width Allocation in LLM Quantization via Performance Maximization Under a Quality-Degradation Constraint”

Next, we discuss hardware. This technology is aimed at SMEs hosting their own LLMs or considering local LLMs.

LLM quantization is a method that reduces the model’s precision (bit width) to decrease memory usage and computational load. While already widely used, traditional methods quantized all layers uniformly. This paper proposes assigning optimal bit widths for each layer to maximize inference speed while keeping quality degradation within acceptable limits.

As a result, inference latency was reduced by 11%.

What Would This Mean in Monthly Expenses?

Assuming a company is running an LLM on a rented GPU server, for example, using an NVIDIA A10G instance (g5.xlarge) on AWS, the monthly cost would be about $750 (around 110,000 yen).

An 11% increase in inference speed means that the same server can process 11% more requests. Conversely, if the processing capacity is reduced, fewer servers are needed for the same workload.

For a company that was renting two servers, after optimization, they would only need the processing power of 1.8 servers. The monthly cost would drop from $1500 (about 220,000 yen) to $1350 (about 200,000 yen), resulting in a monthly saving of 20,000 yen. Annually, this amounts to about 240,000 yen.

While not a staggering figure, this reduction can be achieved simply by “changing the settings.” No additional investment is required. For SMEs operating local LLMs, this is a sure win.

Even more importantly, this technology enables a downgrade of GPU quality. If processing that previously required an A100 (monthly cost of about $3000) can now be handled by an A10G (monthly cost of about $750) due to optimal quantization, the monthly cost would drop to a quarter. This effect of being able to operate on a “lower-tier GPU” fundamentally changes the decision-making process for SMEs.

—

3. Output Speed Increases by Up to 82% — Wait Times Cut by More Than Half

Paper: “Accelerating LLM Inference via Vector Index Based Output Embeddings”

The third technology increases the speed at which LLMs “write out” responses.

LLM inference consists of two phases: “understanding the input” and “generating the output.” This paper focuses on the latter, applying vector indexing to the output projection layer to narrow down candidates instead of scanning the entire vocabulary. As a result, the end-to-end decoding throughput for a batch size of 1 improved by up to 82%.

What Would This Mean in Monthly Expenses?

This is less about cost reduction and more about doubling the amount of processing that can be done at the same cost.

For instance, if an internal chatbot is operated via API with a monthly cost of 30,000 yen, and output token generation speeds up by 82%, it would be possible to handle 1.8 times the requests in the same amount of time. There would be no need to add servers even with an increase in users.

However, to be honest, the true value of this technology for SMEs is not in cost savings but in perceived speed.

The response time for the internal chatbot drops from 3 seconds to 1.6 seconds. Customer-facing FAQ bots provide immediate answers. This experience of “not having to wait” increases employee usage rates and reduces customer drop-off. More than cost reduction, this technology determines whether AI will be utilized.

The most common failure in AI adoption for SMEs is “we implemented it, but no one used it.” Response speed is one of the primary causes of this. An 82% increase in speed may be difficult to quantify in ROI calculations, but it directly impacts on-site adoption rates.

—

What Happens When You Combine the Three?

Let’s summarize the three technologies discussed so far.

These technologies are not mutually exclusive. They can be combined. Compress prompts, optimally quantize models, and accelerate outputs. If all are implemented, the AI operations currently costing 100,000 yen per month could theoretically drop to below 10,000 yen.

—

So, What Should SMEs Do Today?

To be honest, none of these technologies are immediately usable today. BE Tokens are waiting for implementation by API providers. Layer-wise quantization is for local LLM operators. Vector index output is still in the research phase.

However, the direction is clear. The phase of AI inference costs entering a period of “orders of magnitude reduction” has begun.

Here are three things to do today:

1. Understand the breakdown of your AI usage costs. How many tokens are used in prompts? How many times are calls made per month? How many output tokens are generated? Without this knowledge, it is impossible to determine which technology will be effective.

2. Review your system prompts. There are things that can be done before BE Tokens arrive. Simply reducing redundant prompts can lower costs starting today. Just ask, “Is this prompt really necessary at 3000 tokens?”

3. Keep options for local LLMs in mind. Advances in quantization technology are certainly lowering the barriers for SMEs to run LLMs on their own servers. Instead of relying solely on cloud APIs, consider the option of local operation.

When AI usage costs drop by orders of magnitude, what happens? The AI gap between large enterprises and SMEs narrows. Even if a large company reduces its AI operations from 5 million yen to 500,000 yen, the impact on management is limited. However, if an SME’s monthly cost drops from 100,000 yen to 10,000 yen, it transforms from “a company that couldn’t use it” to “a company that can use it.”

Dramatic cost reductions always create new players. For SMEs, this is a tailwind.