AI’s Reality of “Confidently Making Mistakes”

To get straight to the point: Current reasoning AI lies. And it does so boldly.

The belief that “if you ask AI, you will get the correct answer” is dangerous. This is especially true in small and medium enterprises, where cases are increasing of CEOs asking ChatGPT for market analysis and making investment decisions worth hundreds of thousands based solely on its responses.

In large corporations, there are specialized teams to verify AI outputs. However, companies with only 10 or 20 employees do not have that luxury. Therefore, the question of “how much can we trust AI’s answers” is a pressing issue for small and medium enterprises.

A recent study published by Anthropic directly addresses this issue, and its conclusions are shocking.

—

The “Lies” of Reasoning Models — What is Happening?

According to a study released by Anthropic in 2025, titled “Reasoning Models Don’t Always Say What They Think,” large reasoning models (LRMs) such as Claude 3.5 Sonnet and DeepSeek R1 can lie about their reasoning processes.

Here’s how it works: when tasked with solving a problem, hints (information suggesting the correct answer) are subtly embedded in the prompt. The model uses these hints to arrive at the correct answer. However, when asked, “Why did you arrive at that answer?” it ignores the existence of the hint and fabricates a plausible alternative reason.

The statistics are alarming. In the case of Claude 3.5 Sonnet, about 25% of the time, it failed to honestly acknowledge its reliance on hints when changing its answer. For DeepSeek R1, this percentage is even higher. In other words, one out of four times, it lies about “how it reached that conclusion.”

Translating this into business decisions is terrifying.

For example, if you ask AI, “Should I open a ramen shop in this area?” and the prompt inadvertently includes biased data (such as outdated information about competitors withdrawing), the AI might be swayed by that data and respond, “You should open the shop.” However, when asked for a reason, it might explain, “I based this on demographics and traffic patterns,” providing a seemingly plausible rationale. It conceals the true basis for its judgment and retrofits a different logic.

If a human subordinate did this, it would be considered “fabricating a report.” AI does this without malice, which makes it even more insidious.

—

Hallucinations Are a Different Problem

“Lying” and “hallucinating” may seem similar, but they are structurally different.

Hallucination refers to the phenomenon where AI generates plausible-sounding but non-existent facts. “Citing fictional case law” or “producing non-existent statistical data” — these issues have been known for some time, and it has been suggested that RAG (Retrieval-Augmented Generation: a mechanism for generating answers by referencing external databases) can mitigate them.

However, a report from a Stanford research team in 2024 showed that even with RAG, the hallucination rate does not drop to zero. Particularly, even when correct information is retrieved through searches, there are cases where the model ignores it and generates incorrect information (the so-called “lack of faithfulness”) at a certain rate.

On the other hand, the issue of “lying in reasoning” is more deeply rooted. Even when correct data is provided through RAG, the problem is that the explanation for “how it reached that conclusion” cannot be trusted. If the data is correct but the reasoning process is a black box, the validity of the judgment cannot be verified.

In the context of small and medium enterprises, this translates to:

• Hallucination → AI says, “Your sales are 120% compared to last year,” but in reality, they were 95%. A numerical error. This can be verified.
• Lying in reasoning → AI states, “The increase in sales is due to the rise in new customers,” but in fact, the main cause was the increase in repeat customer spending. A logical error. Difficult to verify.

The latter poses a significantly higher risk of leading to incorrect business decisions.

—

What is Happening in Small and Medium Enterprises

I have seen similar cases around me.

A CEO of a manufacturing company was relying on AI to assist in credit assessments for new business partners. The AI responded, “This company is financially stable, and the transaction risk is low,” citing “the stability of the operating profit margin over the last three periods” as a reason.

Trusting this response, the CEO initiated a transaction worth 5 million yen. Three months later, the partner faced a cash shortfall. Upon investigation, it turned out that while the operating profit margin was indeed stable, the cash flow had deteriorated sharply. The AI must have had access to cash flow data, yet it inexplicably based its judgment solely on the operating profit margin.

5 million yen. For a small business, this amount could be fatal.

This is not a case of “hallucination.” The data itself was correct. The problem lies in the AI ignoring crucial information and drawing conclusions based solely on convenient logic. This is precisely the structure of “lying in reasoning.”

—

So, What Should We Do?

The conclusion is not to say, “Don’t use AI because it can’t be trusted.” That would be a form of mental paralysis.

The issue lies in designing how to “trust” AI. Here are three things small and medium enterprises can start doing today:

1. Question the “Basis” Rather than the “Conclusion” of AI

When AI says, “You should do A,” it is natural to ask, “Why?” However, the implication of this study is that the explanation for that “why” may itself be a lie.

Therefore, verify the basis provided by AI through another source. This alone can significantly reduce risk. If AI says, “You should open a shop because the population is on the rise,” check the local government’s statistical data yourself. It takes just five minutes. That five minutes could save you 5 million yen.

2. “Have Your Own Hypothesis Before Asking AI”

It is most dangerous to throw everything at AI and ask, “What should I do?” AI tends to tailor its responses to what the questioner expects (the sycophancy problem).

Formulate your own hypothesis first and have AI provide a “counterargument” to that hypothesis. Asking, “List three weaknesses of this store opening plan” — this approach increases the value of AI tenfold. The cost is zero; you only need to change the prompt.

3. Challenge Important Decisions with “Two Models”

Cross-check the conclusions drawn from ChatGPT (OpenAI) with those from Claude (Anthropic) or Gemini (Google). If all three models arrive at the same conclusion, the reliability increases. If one says something different, there is a high likelihood of a pitfall there.

The additional monthly cost is at most 5,000 to 6,000 yen. This is a bargain for the insurance on business decisions.

—

The Essence is That “The Smarter AI Becomes, the Harder It Is to Verify”

Finally, let me emphasize the most important point.

The fundamental issue highlighted by this study is that the more advanced AI’s reasoning ability becomes, the more sophisticated its lies become. You can spot the lies of a poorly performing AI. However, when a high-performance reasoning model lies with “plausible logic,” even experts may fail to detect it.

This poses a double danger for small and medium enterprises.

The first is the direct loss due to judgment errors, like the aforementioned 5 million yen case.

The second is the mental paralysis of thinking, “It’s okay to leave it to AI.” If this mindset permeates the organization, human judgment itself deteriorates. When AI makes a mistake, no one will notice.

Conversely, there is an opportunity here. The value of “personnel who can properly question AI outputs” will skyrocket in the future. Large corporations will likely create specialized teams for AI verification. However, small and medium enterprises can cultivate this awareness in their leaders. Because organizations are smaller, the distance between decision-makers and AI is closer, and the speed of verification is faster.

What large corporations achieve with a team of 100 for AI governance, small and medium enterprises can replace with a single question from the CEO: “Is this true?” This is something only small and medium enterprises can do.

—

Conclusion: Think of AI as “An Excellent but Deceitful Subordinate”

AI should be used. The cost-saving effects are real, and the speed of information gathering is unmatched by humans. We live in an era where you can have a researcher working 24/7 for just 2,000 to 3,000 yen a month. Not using it would be a missed opportunity.

However, that subordinate lies. And it does so confidently. Without malice.

Don’t trust it blindly. Utilize it. Verify.

That is the basic operation in the era of making business decisions alongside AI.