What Happens When Inference Costs Drop to 1/10

To get straight to the point: a seismic shift in the cost of using AI has begun.

AI chip startup Etched has doubled its valuation to $21 billion (approximately 3.2 trillion yen) in just one month. The reason for this investor enthusiasm is simple: the company’s Transformer-specific ASIC, “Sohu,” demonstrates overwhelming superiority in inference performance compared to NVIDIA’s general-purpose GPUs.

According to official figures, Sohu boasts over 20 times the throughput compared to the H100. When translated into inference costs, this means costs could drop to less than 1/10.

Whether these numbers will materialize as stated remains to be seen. However, the direction is clear: “inference costs will dramatically decrease.” This is not just a story about Etched. Companies like Groq, Cerebras, and even NVIDIA itself are striving to significantly enhance inference efficiency with their Blackwell architecture. A competitive landscape among chip manufacturers is forming, driving costs down.

I want to ask small and medium-sized enterprise (SME) owners: “If the cost of using AI drops to 1/10 of what it is now, what will change in your company?”

The Structural Reasons Why Dedicated Chips Will Outperform General-Purpose GPUs

Why can dedicated chips beat general-purpose GPUs? Understanding this will clarify future decision-making.

NVIDIA’s GPUs were originally designed as general-purpose processors for graphics processing. They can be used for both AI training and inference, but the fact that they can “do everything” also means they are “not optimized for specific tasks.”

Etched’s Sohu is different. It is an ASIC (Application-Specific Integrated Circuit) specifically tailored for inference processing of the Transformer architecture. By not having unnecessary circuitry, its processing efficiency per unit of power and area is extraordinarily high. To use a smartphone analogy, it’s the difference between a “compact camera that can take any picture” and a “dedicated camera optimized for specific subjects.”

What’s crucial here is that the current battleground in the AI market is shifting from “training” to “inference.” Training foundational models like GPT-4, Claude, and Gemini requires investments in the hundreds of millions to billions of dollars, which is the domain of large corporations and big tech. However, using pre-trained models—i.e., inference—is something all users, including SMEs, engage in regularly.

A decrease in inference costs represents a cost revolution for everyone on the “user side” of AI. This is the essence.

A Numerical Perspective: How Will AI Costs Change for SMEs?

Let’s visualize this concretely.

Currently, if you use a GPT-4 class model via API, the monthly inference costs are roughly as follows (depending on usage):

• Internal chatbot (handling 500 inquiries per day): 150,000 to 300,000 yen per month
• Document summarization and classification (processing 1,000 items per day): 200,000 to 500,000 yen per month
• Quality inspection including image recognition (one production line): 500,000 to 1,000,000 yen per month

What happens if these costs drop to 1/10?

• Internal chatbot: 15,000 to 30,000 yen per month
• Document summarization and classification: 20,000 to 50,000 yen per month
• Quality inspection: 50,000 to 100,000 yen per month

This is below the cost of hiring a single part-time employee. It will no longer be a question of “whether to implement AI” but rather “there’s no reason not to implement it.”

Of course, SMEs won’t directly purchase Etched’s dedicated chips. Improvements in chip performance will be reflected in the API prices of cloud services. AWS, Azure, and Google Cloud will provide cheaper inference infrastructure, and the benefits will reach end-users. In fact, in just the past year, the API prices for major LLMs have dropped to less than half. The rise of dedicated chips like Etched’s will further accelerate this price decline.

NVIDIA’s TRTMC — Lowering the Barriers to Entry

It’s not just about costs. The “effort of implementation” is also decreasing simultaneously.

NVIDIA’s recently announced TensorRT Model Connect (TRTMC) is understated yet significantly practical. In a nutshell, it allows you to “deploy AI models published on Hugging Face into a high-speed inference environment with just two commands.”

Previously, it required multiple steps: converting the model to ONNX format, optimizing it with TensorRT, and building the inference pipeline. This was a domain that was inaccessible without AI-specialized engineers.

TRTMC skips this entire process. Through a C++ task API, inference runs end-to-end. In other words, the time it takes to “select and run an AI model” has dramatically shortened.

The implications for SMEs are clear. There’s now a possibility to run external models quickly in their own environments without needing to hire AI engineers as full-time employees. With cloud GPU instances costing just tens of thousands of yen per month, they can run the same models at the same speed as large corporations. This is a way to bypass the biggest bottleneck for SMEs: the “talent gap.”

Surge in Physical AI Investments — The Next Wave Lies Outside Software

Another significant trend cannot be overlooked. Investments in physical AI—robotics, autonomous driving, drones—are surging. VC money is flowing into this sector at the scale of billions of dollars.

How does this relate to SMEs? It does.

The proliferation of physical AI will exponentially increase the demand for inference chips. Each robot needs to run inference in real-time. As demand increases, the cost per chip will decrease due to economies of scale. If chip costs decrease, so will cloud inference costs.

In other words, the more robotics companies invest in physical AI, the lower the costs for SMEs to use AI in the cloud. They can benefit from industry-wide investments without having to invest themselves. This is a favorable structure for SMEs.

So, What Should SMEs Do Now?

“Should we just wait if costs are going to drop?” — This is a misconception.

The companies that will benefit the most when costs drop are those that “already know how to use AI.” Even if costs drop to 1/10, it won’t matter if they don’t know how to utilize it.

Here are three things to do now:

1. Start Small
Begin with API usage costing 10,000 to 50,000 yen per month. Summarizing internal meeting minutes, drafting responses to inquiries, checking estimates—anything will do. The top priority is to create “time spent interacting with AI” within the company.

2. Be Aware of Inference Costs
Most of the costs of using AI come from inference costs. Check the pricing structure of the services currently used and understand “what the token price is” and “how many tokens are used monthly.” Knowing this will enable quicker decisions on scaling when costs drop.

3. Consider the Option of Having Your Own Model
As technologies like TRTMC mature, the barriers to running open-source models in-house will dramatically decrease. Even if it’s not immediate, it’s worth estimating “how much it would cost to run Llama-type models on local GPUs.” Moving away from API dependency is significant not only for cost reduction but also for data security.

Develop the Ability to Read Structural Changes

Etched’s valuation has doubled in just one month. More important than the fact itself is understanding “why investors are willing to bet so much.”

The answer is simple. There is a visible potential for the inference market to expand explosively and for NVIDIA’s monopoly to be challenged. If the monopoly breaks, competition will arise, and with competition comes lower prices. Lower prices will lead to more users, further expanding the market.

The ones who will benefit the most from this cycle are not large corporations. Large corporations already have their own GPU clusters. The most significant impact of cost reductions will be felt by “those who have been unable to fully implement AI due to cost constraints” — namely, SMEs.

What used to cost 3 million yen will become 300,000 yen. 300,000 yen will become 30,000 yen. With each reduction, the base of companies that can use AI will expand.

Local manufacturers will run quality inspection AI for 30,000 yen a month. Local accounting firms will automate document classification for 20,000 yen a month. Local retailers will run demand forecasting for 10,000 yen a month.

Such a world will arrive in the next 2 to 3 years.

The question is simple: “Is your company prepared for that time?”