AIの出力を「そのまま使った」人たちが、いま何を失っているか

The cost of using AI has dramatically decreased. Ask ChatGPT, and you’ll get text in seconds. Images, summaries, reports—everything is at your fingertips. However, behind the falling “cost of generating outputs,” there is a rapidly rising cost.

The cost of “damages incurred when AI outputs are blindly trusted.”

In the past few weeks, three incidents have occurred that illustrate this point. A police officer is under investigation for allegedly fabricating evidence using AI. KPMG has withdrawn an AI-generated report. A court has held Google accountable for false outputs generated by AI. The common thread in all these cases is that they involved using AI outputs without verification.

This issue is not limited to large corporations or public institutions. In fact, small and medium-sized enterprises, which often lack robust verification systems, are at a higher risk of stepping on the same landmines.

—

事件1：警察官がAIで「存在しない証拠」を作った

A police officer in Derbyshire, England, is under investigation for allegedly creating evidence for multiple cases using AI.

According to reports, this officer used documents generated by AI as evidence in investigations without any verification. The problem is clear: no one checked whether the “plausibly crafted text” by AI was based on facts.

What are the consequences? Innocent individuals may be pursued. Wrongful convictions can emerge. In the UK, state compensation for wrongful convictions can reach hundreds of thousands of pounds. Moreover, all past cases involving the officer could be subject to re-investigation. The act of one person “trusting AI” can undermine the trust of an entire organization.

Here, one should consider the comparison between the cost of “writing evidence oneself” and the cost of “having AI write it without checking.” The former takes time, while the latter is instantaneous, but the cost of failure is three orders of magnitude greater.

—

事件2：KPMGがAI生成レポートを撤回——「Big4」でも防げなかった

KPMG has withdrawn a report created using AI due to the phenomenon known as AI “hallucination.” In other words, AI generated plausible but false information that ended up in the report.

KPMG is one of the Big Four accounting firms, a group of professionals specializing in compliance and risk management. Even KPMG was unable to circumvent the need for checking AI outputs.

This incident illustrates that a “system for verifying AI outputs” will only function if intentionally designed by those implementing it. AI does not inform you of its mistakes; it outputs confidently even when incorrect. Therefore, without a verification mechanism on the human side, the same issues can arise in any large corporation.

Consider the damages resulting from KPMG’s report withdrawal. Direct costs include re-creating the report, explaining to clients, and legal responses. However, far more significant is the spread of doubt in the market regarding the trustworthiness of KPMG’s reports. For an auditing firm, trust is the product itself. The cost of that trust being damaged is not something that can be measured in mere hundreds of millions of yen.

—

事件3：裁判所がGoogleに「AIの虚偽出力」の責任を認定

A court has ruled that Google is legally responsible for false statements generated by AI.

This is a landmark ruling. Until now, it was ambiguous to what extent companies providing AI would be held accountable for what AI “spontaneously said.” This ruling indicates that AI outputs can be treated as “statements of the provider.”

For small and medium-sized enterprises, the implications of this ruling are significant. For instance, if a company installs an AI chatbot on its website and that bot provides incorrect product descriptions, the defense of “AI said it on its own” will not hold. That output may be treated as a statement from your company.

Violations of the Act Against Unjustifiable Premiums and Misleading Representations, false advertising, and damage to reputation—one AI output can suddenly bring these risks to reality.

—

「検証コスト」と「検証しなかったコスト」を数字で比較する

Now to the main point. For small and medium-sized enterprises, what is the cost of verifying AI outputs? And what are the damages if verification is not conducted?

検証するコスト（月額ベースの目安）

検証しなかった場合の損害（実際の事例ベース）

Monthly verification costs of 50,000 to 80,000 yen compared to damages exceeding several hundred thousand yen per incident. The difference is clear.

Verification is not a “cost” but rather “insurance.” Moreover, it is not insurance against risks that “may not happen” like fire insurance. AI hallucinations are probabilistically guaranteed to occur. Failing to prepare for known risks is akin to not wearing a seatbelt, not merely skipping insurance.

—

中小企業が「今日からできる」3つの検証ルール

You don’t need a specialized team like large corporations. Just decide on the following three rules.

1. 「AIの出力は下書き」ルールを全社に徹底する

Do not release AI outputs externally as they are. Document this as an internal rule. Emails, proposals, social media posts, chatbot responses—all should adhere to the principle of “human verification before release.” Creating this rule will take one hour and cost nothing.

2. 「固有名詞・数字・法的表現」は必ずソースを確認する

AI is most prone to errors with proper nouns, numbers, and legal or regulatory descriptions. Always refer back to the original source for these three categories. There is no need to check the entire text; the landmines are located in specific areas.

3. 「誰がチェックしたか」を記録に残す

In the event that an AI output causes an issue, having a record that the “verification process was followed” can change the weight of legal responsibility. Simply record “date, content, verifier” in a Google Spreadsheet. This can be done in five minutes.

—

「AIを使うな」ではない。「AIの出力を信じるな」だ

Do not misunderstand. This article is not saying “do not use AI.” AI is a powerful tool for small and medium-sized enterprises, helping to alleviate labor shortages, reduce costs, and increase speed. Its value remains unchanged.

However, weapons require safety mechanisms.

The police officer used AI outputs as evidence without verification. KPMG submitted AI outputs as reports without checking. Google displayed AI outputs as search results without scrutiny. The commonality among these three incidents is that they were used “as is.”

The greatest advantage for small and medium-sized enterprises in using AI is “speed” and “cost-effectiveness.” However, the moment they take shortcuts on verification, relying on that speed and cost, AI transforms from a weapon into a landmine.

By skimping on a verification cost of 50,000 yen, they risk incurring damages in the hundreds of thousands. This is not an “AI risk”; it is a “business judgment error.”

The cost of generating AI outputs is approaching zero. Therefore, the ability to verify outputs will become the competitive edge moving forward.

Start today by deciding just one thing within your organization: “AI outputs must always be verified by a human before being released externally.” By doing so, your company will have made a far wiser decision than those involved in the three incidents.