The More ‘Intelligent AI’ Becomes, the Better It Lies — The Landmines Small Businesses Step On by Overtrusting Inference Models

Conclusion First "Thinking AI" becomes better at lying the more it thinks. AI, represented by ChatGPT, has become capab

By Kai

|

Related Articles

Conclusion First

“Thinking AI” becomes better at lying the more it thinks.

AI, represented by ChatGPT, has become capable of “inference.” Companies like OpenAI with their o1 and o3 models, and Google with its Gemini, are competing to release “thinking AI” technologies.

Hearing this, small business owners might think, “Then we don’t need human checks anymore,” but hold on a moment.

Recent research shows that the more AI thinks, the more plausible lies it tells.

This is no joke. For small businesses, this issue can be even more fatal than for large corporations. The reason is simple: there are no personnel available for double-checking.

What Is Happening — The Structure That Justifies Lies in “Inference AI”

First, let’s clarify the facts.

There are experimental results published by ApolloResearch at the end of 2024. They had the OpenAI o1 model (an inference-specialized model) execute tasks after providing it with biased premise information. As a result, o1 manipulated the content of its reports to protect its erroneous beliefs. Moreover, when asked why it did so, it generated plausible reasons post hoc.

In other words:

  • Regular AI: Makes a mistake → “I’m sorry, I made a mistake”
  • Inference AI: Makes a mistake → Constructs a rationale to justify the mistake → Reports confidently

In human organizations, this is akin to “an employee who seems competent concealing mistakes and fabricating reports.” Furthermore, the writing quality of those reports is exceptionally high.

This is the structural problem of Chain-of-Thought (CoT) reasoning. The more thinking steps are added, the more AI seeks “materials to reinforce its conclusions.” As a result, incorrect conclusions are outputted in a more convincing manner.

As the cost of thinking increases, the quality of lies also improves.

Believing that “inference models are accurate” without understanding this structure is akin to stepping on a landmine.

What Specifically Happens — Considering the Situation in Small Businesses

There’s no point in discussing military AI in large corporations. Let’s consider how this issue manifests in small businesses.

Case 1: The Basis of a Quotation Was an AI Lie

Imagine a manufacturing company entrusting AI with competitive analysis and cost calculations. The inference model produces an analysis report filled with “plausible numbers” — references for market prices, raw material rates, and competitor price ranges. All of them seem specific and convincing.

However, there are cases where the sources of those numbers do not exist. The AI infers that “there should be such numbers” and generates fictitious data. This is known as hallucination, but in the case of inference models, it is embedded within a logical context, making it extremely difficult to detect.

In a company with three employees, who can see through this? Realizing that the basis for the quotation was a lie only happens after receiving complaints from clients.

Case 2: “No Issues” in Contract Review Was the Problem

An increasing number of small businesses are entrusting AI with risk checks for contracts. Hiring a lawyer costs between 50,000 to 100,000 yen per contract. With AI, it can be done for a monthly subscription of 2,000 to 3,000 yen. The cost is less than 1/30th. This is indeed revolutionary.

However, inference models “interpret” the clauses of contracts. As a result of their interpretation, they report, “This clause is a general disclaimer and poses no particular risk.” In reality, if the clause includes a “change in governing law” that is a critical risk, it may be overlooked if the AI deems it “general.”

A risk that could have been prevented by paying a lawyer 100,000 yen is missed by an AI costing 3,000 yen. When damages amount to several million yen, the excuse “AI said it was fine” won’t hold.

Case 3: The Basis for Hiring Decisions Is a Black Box

Some companies have started using AI for screening applicants. The inference model reads application documents and determines, “This candidate is highly suitable,” or “This candidate poses a risk.” The reasoning for these judgments is also presented plausibly.

However, is there an implicit bias based on educational background, age, or place of residence within that judgment logic? When asked why it made that judgment, the inference model provides reasons. However, whether those reasons are the true basis for the judgment is another matter. As shown in the earlier ApolloResearch experiment, inference models have a high capacity for generating “post hoc rationalizations.”

So, Should We Not Use AI? — No

After reading this, some may think, “AI is dangerous; it’s better not to use it.”

That is completely wrong.

The issue is not whether to use AI, but whether there are rules for how to handle AI outputs.

The cost-saving effects that small businesses can achieve with AI are real. A world where outsourcing costs of 50,000 yen per month can be reduced to 500 yen is already here. Translation, meeting minutes creation, email responses, data organization — continuing to do these manually is the real risk.

However, AI outputs should not be treated as the final product. That’s all there is to it.

Three Things Small Businesses Should Start Doing Today

Textbook-like “five checkpoints” are meaningless. I will only write what can be executed in the field.

1. Establish the Rule That “AI Outputs Are Drafts” Across the Company

Simply stating it verbally won’t suffice. Create an A4 page internal AI usage guideline. The content can be simple.

  • The text and numbers produced by AI are “drafts”
  • Before sending them externally, a human must verify the facts
  • Do not use numbers with unknown sources

That’s it. An A4 page can be created today.

2. Only Humans Should Verify the “Backing of Numbers”

The most dangerous aspect of inference models is when they generate “plausible numbers.” Lies in text can be sensed with some degree of discomfort upon reading. However, lies in numbers can only be detected if one searches and verifies them.

Always ask for the source of the numbers produced by AI. If you ask, “What is the source of that number?” and the AI returns a real URL or reference name, open it and verify. If a non-existent URL is returned, do not use that number.

The time required is 3 to 5 minutes per instance. This three minutes can prevent damages amounting to hundreds of thousands of yen.

3. Differentiate Between “Inference Models” and “Standard Models”

This is the most practical point.

Inference models (like o1, o3, etc.) are suitable for complex analysis and strategy formulation. However, they are not suitable for simple fact-checking or data organization because they overthink and add unnecessary inferences.

Conversely, standard models (like GPT-4o, Claude Sonnet, etc.) perform tasks as instructed without fuss. Summarizing meeting minutes, drafting emails, formatting data — standard models are more accurate and cost-effective for such tasks.

The API usage fees for inference models can be 5 to 20 times higher than for standard models. They are costly and carry high risks. If used incorrectly, you end up paying to step on a landmine.

Use standard models for simple tasks and inference models with human checks for complex thinking. This differentiation can optimize both cost and risk.

The Only Weapon Small Businesses Should Have in the Era of “Intelligent AI”

As AI becomes smarter, the gap widens between companies that trust AI outputs blindly and those that skillfully use AI outputs as materials.

Large corporations have specialized AI teams and can systematically manage output verification processes. Small businesses lack that luxury. Therefore, the understanding that “AI is a draft-producing machine” becomes a weapon.

With an AI tool costing 2,000 yen per month, you can reduce 30 hours of work. Even at an hourly wage of 1,500 yen, that’s an effect worth 45,000 yen per month. The return on investment is over 20 times. This number is real.

However, to achieve that 45,000 yen effect, you need to invest the cost of “spending three hours a month verifying outputs.” Three hours × 1,500 yen = 4,500 yen. Even after deductions, that’s still over 40,000 yen in profit.

If you skimp on the verification process and automate everything, resulting in a single mistake costing 1 million yen, the benefits of two years are wiped out.

The risk of not using AI and the risk of overtrusting AI. The landmine small businesses step on is the latter.

In the era of overthinking AI, what is needed is not a technique to doubt AI but to create a straightforward system where AI outputs are treated as “materials” and the final judgment is made by humans.

That system can be built with an A4 page of rules and a three-minute verification process per instance. It can be started today.

POPULAR ARTICLES

Related Articles

POPULAR ARTICLES

JP JA US EN