Conclusion First: Small Models Are Outmatched by Their Ability to Discern Correct Answers Rather Than Their Ability to Generate Answers

Many small and medium-sized enterprises are paying tens of thousands of yen per month for GPT-4o or Claude Opus. On the other hand, models like Haiku and Gemini Flash can operate at a cost of 1/10 to 1/20. “So why not switch entirely to small models?” — Many must have thought this.

However, that would be risky.

A recently published paper titled “Blind Spots: Where Small Language Models Fall Behind Frontier Models on Terminal Tasks” has laid bare the differences between small language models (SLM) and large frontier models (FM) across three axes. By understanding the structure shown in this research, companies can gain criteria for “which tasks to assign to large models and which to small models.” This could lead to some companies reducing their monthly API costs to less than half.

The Blind Spots of Small Models Viewed Through Three Axes

The paper analyzed the following three capability gaps:

1. Problem-Solving Ability (Can it produce the correct answer?)
Small models can reach the correct answer to some extent if sampling is repeated. In other words, “if you keep trying, you will eventually hit it.” This gap can be reasonably closed by increasing the number of attempts.

2. Verification Ability (Can it recognize the correct answer as correct?)
This is where the critical difference lies. When presented with the same set of candidates, large models can accurately identify “this is the correct answer.” Small models cannot do this. Even if the correct answer is among ten candidates, they may mistakenly choose the wrong one as “this one.”

3. Predictive Ability (Can it judge whether it can arrive at the correct answer?)
Small models tend to overestimate their success rate. They say “I can do it” and then make mistakes. Large models can honestly assess, “This is difficult.”

In summary:
> The true weakness of small models lies not in their “ability to produce answers” but in their “ability to judge whether their answers are correct and their awareness of their own limitations.”

This is extremely important when using them in practice. For simple summarization or formulaic text generation, small models are sufficient. However, for tasks where determining correctness is difficult — such as complex analyses, legal document checks, or multi-step reasoning tasks — small models run the risk of “confidently making mistakes.”

Performance Comparable to Large Models at 60% of the Cost? The Approach of Artifact Distillation

So, are small models unusable forever? Not necessarily.

Another paper titled “Artifact Distillation: Training-Free Transfer from Frontier to Small Language Models” proposes an interesting approach.

The mechanism is simple:

1. First, have a large model (like Claude Opus 4.5) solve difficult problems.
2. Save the steps and patterns of these solutions as an “artifact bank” externally.
3. When a small model (like Claude Haiku 4.5) performs inference, it references this bank.

No additional training is required. No fine-tuning is necessary. It’s akin to answering while looking at an external “cheat sheet.”

The results are as follows:

• Spatial reasoning tasks: Performance improvement at 60% of the cost of large models
• Table joining tasks: Achieving comparable levels at 89% of the cost of large models

Honestly, the figure of 60% cannot be described as “dramatically cheap.” However, what is noteworthy is the “no additional training required” aspect. Fine-tuning usually incurs costs in data preparation, training, and validation, amounting to hundreds of thousands to millions of yen. With this approach, those costs can be zero.

The implications for small and medium-sized enterprises are clear. By tapping into the API of large models to accumulate “solution patterns,” routine processing can be delegated to small models. A structure can be created where the large model acts as a “teacher” and the small model operates as the “practical executor.”

MoE Models and Speculative Decoding Can Significantly Increase Inference Speed

There is another study that could change the cost structure.

The paper titled “Why Speculative Decoding Works Better Than Predicted on Sparse MoE Models” discusses this.

Speculative decoding is a method where a small model generates multiple tokens as a “draft,” and a large model verifies them all at once. Typically, large models generate one token at a time, which makes them slow. This method reduces the number of forward passes for the large model.

The key point of this paper is that the compatibility with Mixture-of-Experts (MoE) models is better than theoretical predictions. MoE models activate only a subset of experts among all parameters, making them inherently computationally efficient. When combined with speculative decoding, particularly high-speed improvements can be achieved in low batch size environments (i.e., environments with fewer simultaneous processes).

What does this mean?

Environments handling a small number of requests, such as small and medium-sized enterprises, are more likely to benefit than large corporate servers processing a high volume of requests.

In environments with only one or two GPUs or using minimal configuration instances in the cloud, this technology is particularly effective in “resource-limited settings.”

So, What Should Be Done?

Based on the three studies, let’s outline what small and medium-sized enterprises can do starting today.

1. Categorize Tasks by “Difficulty of Judgment”
Tasks where the correct answer is clear (such as generating template sentences, summarization, translation, and data formatting) are sufficient with small models. On the other hand, tasks where it is difficult to determine correctness (such as risk analysis of contracts, complex conditional business judgments, and multi-step reasoning) should either use large models or incorporate a system for human final verification.

Just this categorization can often reduce API costs to less than half. A monthly fee of 200,000 yen could drop to 80,000 to 100,000 yen.

2. Adopt the Concept of an “Artifact Bank”
Once a difficult task is solved by a large model, save the solution patterns as prompt templates or few-shot examples. For subsequent similar tasks, process them with a small model plus the saved patterns. This doesn’t have to be the exact method from the paper, but the same idea can be implemented immediately.

3. Don’t Trust the “Confidence” of Small Models
Even if a small model returns a high confidence level, it is not reliable. Especially for critical judgments, a system is needed that does not simply adopt the self-assessment of small models. Incorporate a large model for double-checking or include human reviews. Skipping this step could lead to the worst-case scenario where “AI confidently makes a mistake, resulting in complaints.”

The Real Issue Lies Here

The choice between small and large models is not merely a matter of “cheap or expensive.” It is about “where the judgment risks lie.”

Using an expensive model where a cheaper one suffices is wasteful. However, using a cheap model where there are judgment risks can lead to accidents.

Whether this categorization can be made will determine the cost efficiency of AI utilization. And this categorization can only be made by individuals who understand the on-ground operations.

Large companies can create AI specialist teams to optimize their use. Small and medium-sized enterprises lack those resources. Therefore, it becomes crucial whether the personnel on the ground can possess criteria like “this task is sufficient with a small model” or “this task is risky without a large model.”

The three papers discussed this time scientifically validate those criteria. There’s no reason not to utilize them.