Conclusion

Let’s get straight to the point. If you consulted AI and felt “reassured,” that’s the most dangerous situation.

Consulting AI, receiving an answer, and thinking, “I see, that makes sense” — this very experience is a trap.

Data presented in a paper (Intelligent Machines, Unintended Consequences) by a research team from MIT Sloan School of Management and others, published in 2025, was shocking. Those who received advice from AI experienced a maximum threefold increase in inaccuracy. Yet, their confidence rose by about twofold.

Even when they are wrong, their confidence only increases. This isn’t just about a few more mistakes; it’s about the very structure of decision-making breaking down.

For small and medium-sized business owners, this is not just a theoretical concern.

—

What’s Happening — The Fear of “Decreased Accuracy × Increased Confidence”

In this study, subjects were given various judgment tasks and compared results with and without AI advice. The findings were as follows:

• Without AI advice: Accuracy rates were average, and confidence was commensurate. They could admit, “I don’t know.”
• With AI advice: Accuracy rates significantly declined. However, the feeling of being “correct” skyrocketed.

Why does this happen? AI responses are plausible. They are grammatically correct, logically structured, and written in a confident tone. Humans are swayed by this “plausibility” and feel as if they have received an answer before they even think for themselves.

What’s more troublesome is the phenomenon where those who use AI become unable to say “I don’t know.” The same study found that the group receiving AI advice had a significantly lower rate of admitting ignorance in areas where their knowledge was insufficient. Because AI provided an answer, they feel as though they know it too. The research team calls this the “illusion of knowledge.”

Consider this in the context of business management.

If a CEO asks ChatGPT, “Should we enter this market?” and receives a plausible analysis — market size figures, names of competitors, and an overview of entry barriers — they might think, “I see, we can do this.” But are those figures accurate? Is that competitive analysis up to date? Is the judgment aligned with their company’s strengths?

AI-generated “answers” can completely obliterate the verification process. This is the fundamental risk.

—

What Stack Overflow Teaches Us — What Happens When “Places to Ask” Disappear

A case that illustrates this structure more clearly is the decline of Stack Overflow.

Stack Overflow was a Q&A site where programmers could post technical questions and receive answers from other programmers. At its peak, it boasted tens of millions of monthly visits and was considered an essential infrastructure for software development.

However, since 2023, traffic has plummeted. According to SimilarWeb data, visits decreased by about 50% from April 2023 to 2024. A large-scale layoff was also conducted in May 2024.

The reason is clear. When you can ask ChatGPT or GitHub Copilot, you receive immediate answers for coding questions. Posting a question on Stack Overflow, waiting for answers, and reading through discussions — that hassle has become unnecessary.

But what we want to consider here is the underlying issue of “convenience.”

Stack Overflow offered not just questions and answers but also discussions about “why that answer is correct,” “what risks this method entails,” and “are there alternative approaches?” There was a voting system that elevated good answers and pointed out incorrect ones. In other words, the knowledge verification process was embedded in the community.

AI lacks this. AI generates “the most plausible answer” without any counterarguments or verification. And humans are prone to believe answers without counterarguments are “correct.”

It’s not that the place to ask questions has disappeared. It’s that the place to have mistakes pointed out has vanished.

This isn’t just a story from the developer world; the same thing is happening in the decision-making processes of small and medium-sized enterprises.

—

Why This Is Fatal for Small and Medium-Sized Business Owners

Large corporations have mechanisms to verify decisions. The corporate planning department scrutinizes data, legal teams identify risks, and discussions occur in the boardroom. Even if a CEO blindly trusts AI outputs, there’s a chance that a brake will be applied somewhere.

Small and medium-sized enterprises lack this.

In a company with ten employees, if the CEO says, “ChatGPT said this,” that becomes the final decision. There’s no one to argue against it. There are no resources for verification. A decision-maker alone, who over-trusts AI, is like a car without brakes.

Moreover, a single judgment error can be fatal for small and medium-sized enterprises.

Deciding to invest 5 million yen in a new business. Choosing to switch suppliers. Deciding whether to raise prices. Large corporations can afford to fail and try again. Small enterprises may not have a “next time.”

That’s why CEOs who end with “I felt reassured after asking AI” are the most at risk.

—

So, What Should Be Done?

I’m not saying “don’t use AI.” That’s not realistic. AI can be a powerful tool depending on how it’s used. The issue is how it’s utilized.

1. Use AI as a “sparring partner for questions” rather than a “source of answers”

Instead of asking AI, “Should we enter this market?” ask it, “List 10 risks of entering this market.” Instead of asking, “What are our strengths?” ask, “Identify 5 gaps in this business plan.”

Use it not to seek answers but as an opponent that challenges your judgment. Just this shift can dramatically improve accuracy.

2. Always pair AI outputs with “fact-checking”

Do not take AI-generated numbers, examples, or analyses at face value. At a minimum, verify primary sources. If it’s market size figures, where is the original data? If it’s competitor information, when was it last updated? Establish a rule within the company to treat AI outputs as “hypotheses” that require verification.

This is also a matter of cost. Which is cheaper: making a 5 million yen investment decision based on AI’s information or spending two hours on fact-checking? The answer is clear.

3. Maintain a culture where it’s okay to say “I don’t know”

The most frightening phenomenon shown by the research is that using AI makes it harder to say “I don’t know.” This is not just an individual issue; it’s a cultural issue within the organization.

If the CEO starts saying, “I’m fine because I asked AI,” employees will stop arguing. Intentionally create an atmosphere where people can ask, “Is that really correct?” This is the strongest line of defense.

4. Keep records of decisions

Document “why that decision was made,” “how AI outputs were used,” and “what was verified and what wasn’t.” There’s no need for an elaborate system. Just writing a line in a Google spreadsheet is sufficient.

By doing this, you will be able to look back later and see the difference in accuracy between decisions influenced by AI and those made through personal judgment.

—

AI Quietly Comes to Take Away “Thinking Power”

The true risk of AI is not that it provides incorrect answers. It’s that it makes people believe incorrect answers are “correct.”

With accuracy tripling in the negative and confidence doubling, this combination doesn’t just lead to individual judgment errors; it undermines the very foundation of organizational decision-making.

The decline of Stack Overflow is a concrete example of losing a “place for verification” behind the facade of convenience. The same thing is currently happening in the decision-making processes of small and medium-sized enterprises.

I’m not saying to stop using AI. In fact, use it to the fullest. However, the moment you feel “reassured” by an answer provided by AI is precisely when you should be the most skeptical.

Small and medium-sized business owners do not have verification teams like large corporations. That’s why they must not forget that they themselves are the final verifiers.

It’s fine to ask AI. But the final decision must be yours. Whether you have that resolve will become the dividing line in future management.