Conclusion First: We Have Entered an Era Where the Inference Cost of LLMs Can Be “Visualized”

How much does it cost to run an LLM once?

Research has emerged that allows us to predict this question not just with a rough estimate, but at the power trace level. This is a significant development for small and medium enterprises (SMEs).

Why? Because it enables a numerical judgment on whether “using an API or running in-house inference is cheaper,” rather than relying on intuition. Should they continue to pay tens of thousands of yen per month for API usage, or should they run their own GPU servers? Making the wrong choice could result in a difference of several hundred thousand to millions of yen annually. For SMEs, this difference is essentially their profit.

The Breakdown of “Cost Per Token” is Becoming Clear

A recently published study (the LLM-PowerTrace dataset) has recorded and made public the power consumption when running LLMs on GPUs, with time resolution. The structure revealed here is simple yet crucial.

• The computation time and energy consumption increase almost linearly with the number of output tokens.
• For the number of input tokens, it increases linearly to quadratically.

In other words, the cost structure is entirely different when sending “a long instruction and getting a short response” versus “a short instruction and generating a long text.”

Let’s consider this concretely. For example, when using a GPT-4 class model via API, the cost per request for 1,000 input tokens plus 500 output tokens is approximately 3 to 5 yen (based on OpenAI’s pricing in late 2024). If this is done 500 times a day, the cost would be between 1,500 and 2,500 yen. This translates to about 45,000 to 75,000 yen per month.

On the other hand, with the latest models like GPT-4o or Claude 3.5 Sonnet, the cost per input token has significantly decreased. In some cases, it can be kept within 10,000 to 30,000 yen per month under the same conditions.

This raises the question: “Is it more cost-effective to continue paying 30,000 yen per month for API usage or to run an open-source LLM in-house?”

Breaking Down the “True Cost” of In-House Inference

Let’s roughly break down the costs of running an LLM in-house.

1. Initial Hardware Investment

• Workstation with NVIDIA RTX 4090: Approximately 500,000 to 700,000 yen
• NVIDIA A100 (Cloud GPU): About 300 to 500 yen per hour (for AWS or GCP)

2. Power Costs

• Power consumption of RTX 4090: Approximately 450W (it doesn’t run at full capacity during inference, but uses around 300W)
• Operating for 8 hours a day results in about 2.4kWh. With electricity priced at 30 yen/kWh, this amounts to about 72 yen per day, or approximately 2,200 yen per month.
• This is just for the GPU. The total for the server can be 1.5 to 2 times higher.

3. Operational and Maintenance Costs

• Setup labor costs (initial only, but at least several days to weeks)
• Model update management
• Fault handling

4. Constraints on Usable Models

• The RTX 4090 (with 24GB VRAM) can realistically run models in the 7B to 13B parameter class.
• To run a 70B class model, at least two A100s are needed, with initial investments in the hundreds of thousands of yen.

What becomes clear here is that when the monthly request count is low, using an API is overwhelmingly cheaper.

Let’s estimate the breakeven point.

• API usage cost: 50,000 yen per month (for a GPT-4o class model with moderate usage)
• In-house inference cost: 600,000 yen initial investment plus about 10,000 yen for monthly power and maintenance.

It would take over 12 months just to recover the initial investment. Moreover, the models that can be run in-house are open-source models in the 7B to 13B class, which do not match the quality of GPT-4o.

In other words, “just because it’s cheaper to run in-house” is a trap for most SMEs.

When Does In-House Inference Win?

However, there are cases where the tables can turn.

Case 1: When the Number of Requests is Explosively High
If processing several hundred thousand to millions of requests per month, the API costs can balloon to several hundred thousand yen. At this scale, in-house inference can lower the total cost. For instance, if a company chatbot is used 50 times daily by 500 employees, that amounts to 750,000 requests per month. The API cost could reach 200,000 to 500,000 yen. At this point, recovering the investment in in-house servers could take less than six months.

Case 2: When Data Cannot Be Exposed Externally
In industries dealing with medical data, personal information, or confidential information, sending data to an API poses risks. In such cases, in-house inference is the only option, regardless of cost comparisons.

Case 3: When Response Speed is Critical
Using an API introduces network latency. Processes that can return results in tens of milliseconds locally may take hundreds of milliseconds to seconds through an API. In scenarios requiring real-time responses, such as anomaly detection on manufacturing lines, in-house inference has an advantage.

The Real Focus Should Be on the “Future of Lower Costs”

Now we arrive at the main point.

The inference costs of LLMs have dramatically decreased over the past two years. The cost per input token for GPT-4 has dropped to less than one-tenth from its announcement in March 2023 to the end of 2024. The performance of open-source models has also rapidly improved, with Llama 3.1 70B delivering quality that surpasses GPT-3.5 in many tasks.

What this trend signifies is that the question of whether to use LLMs is no longer relevant. The question has shifted to “where and how to use them.”

Let’s summarize practical strategies for SMEs.

Fewer than 10,000 requests per month → API is sufficient. Using GPT-4o mini or Claude 3.5 Haiku can cost just a few thousand yen per month. Start here and evaluate the effectiveness.

Around 100,000 requests per month → API costs will rise to 50,000 to 100,000 yen. This is when consideration of switching to in-house inference begins. However, it’s essential to make a total judgment, including operational efforts.

Over 1,000,000 requests per month → The cost benefits of in-house inference become clear. However, SMEs of this scale are limited.

So, What Should Be Done?

I’ll say three things.

1. Start with APIs. Consider in-house servers only after API costs exceed 100,000 yen per month. Until then, the pay-as-you-go API model carries significantly lower risk.

2. Be mindful of token counts. The longer the input prompts, the higher the costs. Designing to pass only the necessary information instead of “just throwing in the full text” can sometimes cut costs by more than half.

3. Measure costs monthly. Record API usage and expenses every month. This alone will provide a basis for determining “when to switch to in-house inference.” Using datasets like LLM-PowerTrace allows for pre-simulation of power costs if transitioning to in-house inference.

The costs of LLMs have shifted from being perceived as “somehow high” to being “calculable.” If it can be calculated, it can be judged. If it can be judged, action can be taken.

For SMEs to succeed with AI, it’s not about chasing the latest models. It’s about accurately understanding their usage and cost structures and consistently making the most rational choices. This is where SMEs can actually have an advantage over larger companies due to their faster decision-making.