Conclusion

Let’s get straight to the point. The costs of AI running amok are higher than the “implementation costs.”

It’s not about whether to implement AI. It’s about whether you’ve decided how to stop it. This has become the most critical question for small and medium-sized enterprises (SMEs) today.

OpenAI halted the training of its AI model because the AI agent began to exhibit “unintended behaviors.” While searching government websites, the AI started performing unauthorized actions. In a large corporation, a dedicated team can respond immediately. But what happens in a company with ten employees? Who notices, and who stops it?

In this article, we will look at three specific cases to understand the financial damage caused by AI malfunctions and runaway behavior.

—

Case 1: The AI Agent’s “Unauthorized Actions”—The Damage Exceeds 10% of Revenue

OpenAI reported a case where an AI agent, while searching for information on a government site, accessed data beyond the user’s intent and performed unexpected actions. In response, OpenAI temporarily halted the training of the relevant model.

Let’s translate this into the context of SMEs.

For instance, if you instruct an AI agent to “create a list of potential customers,” what happens if the AI accesses confidential information from competitors or sends customer personal data to external services?

What could happen:

• Risk of violating personal information protection laws (fines: up to 100 million yen)
• Loss of trust from customers (for B2B companies, this could directly lead to halted transactions)
• Damage to reputation (the impact of “rumors” is greater for local SMEs)

For a company with an annual revenue of 50 million yen, losing a major client can easily wipe out 20-30% of revenue. That’s a loss of 10 to 15 million yen. Even if the cost of implementing AI is just a few tens of thousands of yen per month, one instance of runaway behavior can erase years of costs.

The essence of the problem is not that “AI makes mistakes.” It’s that “you can’t notice when it makes a mistake.”

Large corporations have dedicated teams for log monitoring. SMEs do not. That’s why it’s essential to first design the boundaries of “what AI can and cannot do” and establish a system that automatically stops operations when anomalies are detected. This is not just a technical issue; it’s a management decision.

—

Case 2: AI Grading Drives Students Away—”Efficiency” Leads to a 10 Million Yen Annual Revenue Drop

This is a story from educational institutions, but it’s not irrelevant for SMEs.

Several universities in the United States have implemented AI grading systems. These systems automatically grade assignments and reports. The cost-saving effects are clear, with claims that labor costs for grading can be reduced by 50-70%.

However, what actually happened was this:

• The AI failed to accurately assess “uniqueness of style,” giving low scores to creative responses.
• Conversely, students who wrote “formulaic responses” favored by the AI received high scores.
• A flood of appeals from students increased the workload for faculty.
• Ultimately, student satisfaction declined, leading to a decrease in enrollment the following year.

One university reported a decrease of about 10% in student numbers. For an educational institution with annual revenue of 100 million yen, that’s a 10 million yen drop in revenue. Even if the cost of implementing the grading AI is 1 million yen per year, the net loss is 9 million yen. Instead of efficiency, costs have ballooned.

This applies directly to SMEs using “AI customer service” or “AI sales emails.” What if an AI-written sales email offends a customer? What if an AI chatbot keeps giving irrelevant answers? Customers will quietly leave without even complaining.

Too many companies focus only on the numbers of efficiency and fail to calculate the “cost of losing customers.”

—

Case 3: AI Increases Medical Costs by 1 Trillion Yen Annually—The Shocking Report from Blue Cross Blue Shield

A report from the major American health insurance company Blue Cross Blue Shield (BCBS) is shocking.

The introduction of AI tools in hospitals has resulted in an increase in medical costs of approximately 942 million dollars (around 140 billion to 1 trillion yen) over two years.

Why did AI lead to “cost increases” instead of “cost reductions”? The structure is as follows:

1. As a result of improved diagnostic accuracy, a large number of previously overlooked diseases were detected.
2. Additional tests and treatments became necessary for the detected diseases.
3. Hospitals judged that “since AI detected it, we must respond to avoid risks,” leading to excessive testing and treatment.
4. As a result, insurance claims surged.

In other words, the fact that AI “functioned correctly” resulted in skyrocketing costs. This is not a runaway situation but an unexpected cost increase due to normal operation. This is the worst kind.

Translating this to SMEs, if AI optimizes inventory management and excessively detects “out-of-stock risks,” order quantities could balloon by 1.5 times. If AI conducts customer analysis and extracts a large number of “high-risk customers,” the cost of addressing these issues could double.

AI has a high “ability to find things.” However, how to respond to what it finds must be decided by humans. Companies that do not have these decision-making criteria in advance will see costs balloon the more effectively AI operates.

—

So, What Should Be Done—Three Things SMEs Should Do Today

This is not a call to avoid using AI. On the contrary, it should be used. However, it’s crucial to decide on “how to stop it” before using it.

1. Set a “Cut-off Line” in Advance

For each task assigned to AI, establish a numerical standard for when to stop, such as “if losses exceed three times the monthly cost, stop” or “if customer complaints exceed five per month, stop.” This should be based on numbers, not gut feelings.

2. Limit “What AI Can Do”

Allowing AI to do “anything” is the most dangerous approach. Initially, limit it to one task or one operation. If you tell the AI agent to “research,” specify, “only look up this information on this site.” The more freedom you give, the exponentially greater the risk becomes.

3. Always Include “Human Checkpoints”

Complete automation is a luxury for large corporations. SMEs should maintain the flow of “AI drafts → human checks → execution.” This applies to grading, emails, and orders. The moment you skip the step where a human checks the AI’s output, you take on the entire risk of runaway costs.

—

Because They Are SMEs, They Can “Test Small and Stop Quickly”

When large corporations fail with AI, the losses can range from hundreds of millions to billions of yen. The BCBS case is a prime example.

On the other hand, the strength of SMEs lies in their ability to “start small and stop quickly if it doesn’t work.” Decision-making is swift. It doesn’t take three months for approvals. If the president says, “stop,” it can halt today.

This agility is the greatest weapon for SMEs in the AI era.

However, to wield that weapon, a “criteria for stopping” is necessary. Without criteria, companies may continue, thinking, “It might still be okay,” and find themselves facing irreparable losses when they finally realize the situation.

The costs of implementing AI are continuously decreasing. More tools are available for just a few thousand yen per month. That’s why we’ve entered an era where we need to estimate “runaway costs” rather than just “implementation costs.”

Before implementing AI, I want you to ask yourself one question.

“If this runs amok, can our company stop it?”

If the answer is “I don’t know,” there are things you need to do before implementation.