A Day Will Come When We Are Free from the Fear of Business Stopping When a Veteran Leaves

What is the most serious management risk for small and medium enterprises in rural areas? It is not cash flow or competition from new entrants. “If that person leaves, the business will come to a halt”—this issue of reliance on specific individuals is paramount.

The intuition for estimates, the strategies for handling complaints, the ability to detect machine malfunctions from unusual sounds—these know-how exist only in the minds of veterans and are disappearing along with them, unrecorded in manuals. A survey indicates that 70% of small and medium enterprises cite the “inheritance of skills and know-how” as a management challenge.

In response to this issue, a structural change is occurring in the world of AI. The cost of Knowledge Distillation has collapsed.

To put it simply, what used to cost 3 million yen to create specialized AI can now be done for 50,000 yen. That’s one-sixtieth of the original cost. At this price point, small and medium enterprises can experiment with transferring the “veteran’s knowledge” into AI.

What is Knowledge Distillation?—The Essence Explained in 30 Seconds

First, let’s break down the term “Knowledge Distillation.” It’s not a complicated concept.

It is a technique to copy the “thinking patterns” of a large and intelligent AI (teacher model) into a smaller and lighter AI (student model). The judgment patterns of a massive model like GPT-4 are distilled into a lightweight model that can run on a smartphone. Think of it as extracting only the essence, like distilling liquor.

Why is this important? Continuing to use a massive model directly incurs API usage fees that can run into hundreds of thousands of yen each month. If you try to run it on your own servers, you need GPUs costing several million yen. This is not realistic for small and medium enterprises.

However, a distilled lightweight model can operate in a cloud environment for just a few thousand yen per month. Moreover, if it is specialized for specific tasks, it can even achieve higher accuracy than a general-purpose massive model. Big does not always mean better. Smaller and more specialized can be stronger. This is the very essence of how small and medium enterprises can compete.

What Has Changed?—Research from Hugging Face Shows Cost Collapse

The research published by Hugging Face, titled “Making Knowledge Distillation Cheap Enough to Run at Scale,” has fundamentally changed this cost structure.

Previously, knowledge distillation required both the teacher model and the student model to be run simultaneously during training. This was the root cause of high costs. Just running inference on a massive teacher model required high-performance GPUs, and this had to be repeated thousands of times. Occupying 4 to 8 GPUs for several days could lead to computational costs of 2 to 3 million yen. Adding in engineering labor costs, some projects easily exceeded 5 million yen.

Hugging Face’s approach tackled this issue. By pre-caching the outputs of the teacher model and separating the distillation process, they established a method that completes equivalent distillation using just 2 GPUs in a few hours. The computational cost is around 50,000 yen. That’s one-sixtieth of the original cost.

Consider the significance of this “one-sixtieth” figure. A budget of 3 million yen would likely not get approved. At 50,000 yen, it becomes feasible to test. For small and medium enterprises, the ability to “test” is almost synonymous with the ability to “do.”

Agent Memory Distillation—Copying the ‘Veteran’s Work Style’ into AI

Another noteworthy technology is Agent Memory Distillation.

While traditional knowledge distillation transfers the “content of knowledge,” agent memory distillation transfers the “work processes.” This is a crucial difference.

Specifically, a large-scale teacher agent records the history of task completion in three hierarchical layers:

• Workflow Memory: The overall flow of work. The steps of “first check this, then judge that, and finally process like this.”
• Subtask Memory: The specific criteria for judgment at each step. Rules like “if this value falls within this range, it’s acceptable; if not, here’s how to respond.”
• Episode Memory: Past success and failure cases. Insights like “when I did this, it worked well” or “when I did that, it failed.”

Knowledge recorded in this three-layer structure is transferred to a smaller student agent. In other words, the veteran employee’s “work style” is structured and taught to AI.

Let’s apply this to a small and medium enterprise context.

Imagine a veteran inspector with 30 years of experience in manufacturing. He can instantly judge that a product is “NG” (not good) upon seeing it. However, when asked to verbalize “why it’s NG,” he can only say, “I just know” or “it’s based on experience.” This is the essence of reliance on specific individuals.

With the agent memory distillation approach, the veteran’s judgment process is first mimicked and structured using a large-scale model, and then distilled into a lightweight model. “Look at this part on the surface → if the color deviates from this range, be cautious → if the light casts a shadow at this angle, it’s NG”—such tacit knowledge is transformed into reproducible procedures.

“So, what should we do?”—What Small and Medium Enterprises Can Start Doing Today

Having discussed the technology, let’s talk about practical steps.

Small and medium enterprises do not need to jump straight into advanced AI development to utilize knowledge distillation. They can take it step by step.

Step 1: Record the Veteran’s Judgments (Cost: Almost Zero)

First, record the work of veteran employees through video and audio. Ask them “why did you do that?” at the moment of judgment. Spending 30 minutes a day for two weeks can yield a significant amount of tacit knowledge verbalized. This can be done with just a smartphone.

Step 2: Structure with ChatGPT (Cost: A Few Thousand Yen per Month)

Transcribe the recorded content and use ChatGPT or Claude to categorize it into “workflow,” “judgment criteria,” and “exception handling.” This alone can create 80% of the operational manual that would traditionally cost 1 million yen to develop with external consultants.

Step 3: Distill a Specialized Lightweight Model (Cost: 50,000 to 200,000 Yen)

Using the structured data, distill a lightweight AI model specialized for your business operations. Using Hugging Face’s open-source tools, this can be done in a few hours in a cloud environment with just 2 GPUs. Increasingly, projects that can be outsourced for around 200,000 yen are becoming available.

Step 4: Implement and Operate in the Field (Cost: A Few Thousand Yen per Month)

Integrate the distilled model into internal chatbots or operational support tools. When a newcomer asks, “What should I do in this case?” they receive answers based on the veteran’s judgment criteria. They can ask as many times as they want, 24/7, without any annoyance.

With these four steps, the total cost is under 300,000 yen. Considering that implementing a knowledge management system used to cost 5 million yen and required 1 million yen annually for operation, the cost is now less than one-tenth, while accuracy improves.

What Truly Changes is Not the ‘Cost’ but the ‘Structure’

At this point, I want to take a step back and consider what happens when the cost of knowledge distillation decreases.

The value of “possessing knowledge” decreases. The value of “being able to structure and transfer knowledge” increases.

Until now, the very presence of veterans was a source of competitive advantage for small and medium enterprises. However, as this knowledge becomes transferable to AI, the source of competitive advantage shifts from “who knows what” to “how it is systematized.”

This is actually an opportunity for small and medium enterprises. Larger corporations take longer to structure tacit knowledge due to their size. Barriers between departments, approval processes, and security reviews slow them down. In contrast, small and medium enterprises can start next week based on the president’s decision. The distance between veterans and newcomers is also shorter. “Being small means being fast”—this structural advantage will be beneficial in the era of knowledge distillation.

Caution—It’s Not a ‘Magic Wand’

However, there are points to consider calmly.

Specialized AI created through knowledge distillation can only respond within the range of the training data. The veteran’s ability to handle “unexpected situations”—this is still difficult to transfer to AI. What can be distilled is the “form,” not the “adaptability.”

Moreover, distributed learning platforms like DGX Spark are currently over-spec for small and medium enterprises to implement directly. Their price range is in the hundreds of thousands of yen. This is not something to dive into immediately; it would be more realistic to wait for it to be offered as a cloud service.

What’s important is to not aim for perfection. There is no need to transfer 100% of the veteran’s knowledge. If you can systematize 80%, the onboarding process for newcomers will be accelerated. The remaining 20% can be taught by people.

Conclusion—Ending the Dependency on ‘That Person’

The cost of knowledge distillation has dropped from 3 million yen to 50,000 yen. What this means is that small and medium enterprises can now experiment with transferring the “veteran’s knowledge” into AI.

With population decline, recruitment challenges, and aging, the issue of “eliminating reliance on specific individuals” is an unavoidable theme for small and medium enterprises in rural areas. And the tools for this have finally become affordable for them.

I encourage you to start by recording the work of veterans. With just a smartphone, you can begin today. There’s no need to wait for technological advancements. Systematization begins with recording.