AIに仕事を任せたら、嘘をつかれた。さて、どうする？

Many small businesses have created internal chatbots believing that incorporating RAG will ensure accuracy.

However, recent research reveals a contrary reality: RAG can actually increase hallucinations.

The issue of “AI lying” is no longer just a concern for technologists. Mistakes in estimates, fabricating employment regulations, citing non-existent laws—the more businesses use AI, the greater the damage from these lies.

This article organizes three recent findings and illustrates how small businesses should deal with ‘AI lies’ with specific figures regarding cost and acceptable error rates.

—

RAGが幻覚を「増やす」メカニズム——検索できても正しいとは限らない

RAG (Retrieval-Augmented Generation) is a mechanism that “searches external documents and generates answers based on them.” It was supposed to provide accurate answers when fed internal documents.

However, recent studies have confirmed that when RAG systems have a “knowledge gap”—meaning the searched documents do not contain the correct answer—they tend to generate answers anyway.

The seriousness of this issue is evident in the numbers:

• Correct answer rate of commercial RAG systems: 97-98% (when the correct answer is in the database)
• Rate of “lying” when there is a knowledge gap (violation rate): 16.7% to 98.1%

In other words, when the answer to a question is not in the database, there is up to a 98% chance of returning a “plausible lie.”

Why does this happen? The cause lies in the evaluation structure of RAG. Even with an asymmetrical penalty design—where a correct answer scores +1 and an incorrect answer scores -4—the model does not have the option to “not answer.” As a result, it answers even when it doesn’t know.

This is a pitfall for small businesses. Even if they think, “We’ve included all our internal manuals, so we’re fine,” the moment a question arises that isn’t covered in the manuals, the AI will not remain silent and will lie. Moreover, that lie will appear plausible, written in the style of internal documents, making it difficult for humans to discern.

The first step in countermeasures is to clearly define the “areas where AI cannot answer.” Instead of expecting omniscience, it is essential to design constraints such as, “Only answer questions within this scope; for anything else, respond with ‘I don’t know.'”

—

LLMの嘘には「一生」がある——どこで生まれ、どこで育つのか

LLM hallucinations do not occur suddenly; they have a life cycle. Understanding this is crucial to effectively implement countermeasures.

1. データ段階（生まれ）

Errors and inconsistencies are present in the training data itself. Since the model learns by gathering information from the internet, the seeds of lies are planted at this stage. It is not realistic for small businesses to intervene at this stage.

2. トレーニング段階（育ち）

During the fine-tuning and RLHF (Reinforcement Learning from Human Feedback) processes, the model is optimized to “answer fluently at all costs.” There is a structure where providing a plausible answer is rated higher than responding with “I don’t know.”

3. 推論段階（社会に出る）

This is the stage where the model answers actual user questions. This is the most critical point for small businesses because this is where intervention is possible.

Recent research has focused on an approach where the model itself assesses its “confidence level.” By asking the same question multiple times, the variability in responses can be observed. If the variability is high, it can be determined that “there is low confidence = high likelihood of lying.”

The greatest advantage of this method is that it does not require labeled datasets. In other words, there is no need to have experts create correct data. It can be implemented with just API usage fees. Asking the same question five times with GPT-4o incurs a cost of only a few yen to several tens of yen.

—

幻覚検出は「無料」でできる時代に

“Detecting AI lies requires an expensive verification system”—this is an outdated notion.

Methods for checking the consistency of the model’s own outputs do not require additional specialized tools. Specifically:

• Multiple generations of the same question: Generate five responses by changing the temperature (randomness) and check the consistency of the answers.
• Self-contradiction check: Ask the same model, “Is this answer correct?” regarding the generated response.
• Enforcing source citation: Instruct the model to “always cite the name of the source document and the relevant section” and reject responses that do not include citations.

These can be achieved through prompt design alone, with zero costs for external tool implementation. Small businesses can start trying these today.

—

中小企業が設定すべき「許容誤答率」——業務別の目安

“Zero error rate” is a fantasy. Humans make mistakes too. The question is whether you have decided how much error rate to allow for each business operation.

If you haven’t decided, you won’t notice AI lies. If you don’t notice, you will end up reacting only after damage has occurred.

Below are guidelines based on practical operations for small businesses.

It is noteworthy that the verification cost for internal FAQs can start at nearly zero. Conversely, if AI is used for legal matters or customer-facing communications, it is necessary to anticipate verification costs of over 100,000 yen per month.

What should be considered here is the comparison with the “cost of not using AI.” If a person handles internal FAQs, the personnel cost for the responsible party is 200,000 to 300,000 yen per month. Automating 80% with AI and spending 10,000 yen on verification results in a cost reduction of over 190,000 yen.

On the other hand, if you assign AI to contract reviews and incur verification costs of 300,000 yen per month, it may be cheaper to hire a lawyer directly. Comparing “AI + verification costs” with “human costs” is the correct decision-making criterion for small businesses.

—

「AIの嘘」と付き合う3つの実践ルール

Finally, here are three practical rules for small businesses to deal with AI hallucinations.

ルール1：AIに「黙る権利」を与えろ

Clearly state in the prompt, “If you are not confident, please respond with ‘I cannot answer this question.'” Just this one line significantly reduces hallucinations. For RAG, set it so that if the search score is below a threshold, it does not generate a response. The cost is zero.

ルール2：「嘘が致命傷になる業務」と「嘘が笑い話で済む業務」を分けろ

Demanding the same accuracy across all operations is a waste of resources. Referencing the table above, determine acceptable error rates for each operation. Simply deciding that “10% is acceptable for the internal chat bot, and below 1% for estimates” clarifies the allocation of verification resources.

ルール3：検証は「仕組み」にしろ。属人化させるな

“Veteran Tanaka is checking the AI’s responses”—this is the worst-case scenario. If Tanaka is absent, no one will check. Create a checklist, incorporate consistency checks for multiple generations into the routine, and measure the error rate monthly. By systematizing it, the same quality can be maintained regardless of who performs the checks.

—

で、結局どうすればいいのか

AI lies. Even with RAG, AI lies. This is not a limitation of technology but the current specification.

However, by calculating the cost of lies, drawing acceptable lines for each operation, and systematizing verification, small businesses can effectively utilize AI.

Large corporations spend millions on dedicated systems to combat hallucinations. Small businesses do not have that budget. Therefore, starting with “zero-cost countermeasures” such as prompt design and operational rules is essential. This is how small businesses can compete.

I encourage you to start with an internal FAQ bot. “When you don’t know something, say ‘I don’t know'”—this simple addition is where the journey of dealing with AI lies begins.