Skip to content

Optional

Distillation

Understand how knowledge distillation trains a smaller or cheaper model to imitate useful behavior from a larger teacher model.

4 min read

After this, you will understand

How Distillation helps you see why the model and retrieval machinery behaves the way it does, and which tradeoffs it creates.

Article guideprerequisites, mental models, and concepts

Article overview

intermediateMechanicsModelingInference

Three useful mental models

In plain terms

Treat the idea as a definition to memorize.

Production pressure

Real systems force the idea to handle Distillation, Knowledge Distillation, and Teacher Model.

Better reasoning

Use the concept to decide what the system guarantees, what it risks, and what it costs to operate.

Think before reading

Where would Distillation appear in a real production system, and what failure or bottleneck would it help you reason about?

As you read, look for the pressure that creates the idea first. The mechanics matter more once the reason is clear.

Optional depth

Follow these links when you want to understand the internals more deeply.
  1. 1Mixture Of ExpertsUnder The Hood

Concepts Covered

  • Distillation
  • Knowledge distillation
  • Teacher model
  • Student model
  • Soft targets
  • Model compression
  • Deployment tradeoffs
  • Quality evaluation

Definition

Distillation is a training approach where a smaller or cheaper model learns from the behavior of a larger, stronger, or more expensive model.

The plain-English version:

teacher model shows useful behavior
student model trains to imitate enough of it

The goal is often to make a deployable model that keeps much of the teacher's usefulness while being cheaper, faster, smaller, or easier to serve.

Why This Concept Exists

The best model for quality may not be the best model for production.

A large model can be:

  • expensive to run
  • slow for a latency-sensitive product
  • too large for edge devices
  • hard to batch efficiently
  • unnecessary for simpler tasks

Distillation exists because teams often want some behavior from a powerful model in a smaller serving package.

The Beginner Mental Model

A beginner may think:

Distillation copies the big model into a small model.

That is too strong.

The student does not become the same model. It learns from examples of the teacher's behavior. The student may imitate useful output patterns, but it has its own architecture, capacity, limits, and failure modes.

Better:

distillation transfers behavior signals, not the teacher's entire mind

Teacher And Student

The teacher is the model that provides the learning signal.

The student is the model being trained.

In a product setting:

large teacher -> high quality, expensive
smaller student -> cheaper, faster, maybe weaker

The teacher may produce labels, rankings, probabilities, explanations, traces, or output examples depending on the distillation setup.

The student trains on those signals and tries to reproduce the behavior that matters.

Soft Targets

In ordinary supervised training, a target may be a hard label:

support ticket -> billing

Distillation can use richer signals from the teacher.

For example, the teacher may indicate:

billing: 0.72
account access: 0.18
technical issue: 0.08
other: 0.02

Those probabilities carry more information than only saying billing.

They reveal what the teacher found plausible and what it rejected. That extra structure can help the student learn a smoother decision boundary.

A Small Product Example

Imagine a company has a high-quality support classifier built with a large model.

The product needs to classify millions of short support messages cheaply.

The team might:

  1. run the large teacher on many representative messages
  2. store the teacher's outputs
  3. train a smaller student on those outputs
  4. evaluate the student against real support outcomes
  5. deploy the student for the high-volume path

The teacher may still be used for harder cases, audits, or new data generation.

Distillation vs Fine-Tuning

Fine-tuning adapts a model through additional training on a task or data.

Distillation uses another model's behavior as part of the training signal.

They can overlap.

A team may fine-tune a student using teacher-generated outputs. The distinction is the source of the supervision:

fine-tuning -> train on task data
distillation -> train from teacher behavior

Why Distillation Needs Evals

Distillation can preserve useful behavior, but it can also preserve mistakes.

The student may:

  • imitate teacher bias
  • lose rare-case behavior
  • become overconfident
  • handle simple cases well and hard cases poorly
  • fail on production data that was missing from the distillation set

This is why distillation should be tied to task evals, safety checks, and production monitoring.

The question is not:

Did the student imitate the teacher?

It is:

Does the student meet the product contract at the cost and latency we need?

Product And Infrastructure Pressure

Distillation is useful when a product has a clear serving reason:

  • lower cost per request
  • lower latency
  • offline or edge deployment
  • smaller specialized model for a narrow task
  • faster fallback path
  • cheaper batch processing

It is less useful when the product truly needs the full capability of the larger model or when the task keeps changing faster than the student can be refreshed.

Common Confusions

Distillation is not a perfect clone.

The student can fail differently from the teacher.

Distillation is not the same as quantization.

Quantization changes numeric representation. Distillation trains a model from another model's behavior.

Distillation is not only for LLMs.

The idea applies broadly across machine learning, though it is very relevant to modern AI serving.

A distilled model still needs evaluation.

Teacher quality does not automatically guarantee student quality.

What This Does Not Mean

Distillation does not remove the need for data, objectives, or product design.

It gives another way to produce a deployable model, but the team still needs representative examples, careful evaluation, and a clear definition of acceptable behavior.

Finished reading?

Your reading history is saved in this browser so you can continue later.

Recommended Next

What Is AI?AI Foundations12 min read

Return to the recommended AI Engineering journey here. Start with the model and inference vocabulary every practical AI system depends on.

Optional exploration

These links add context, but they do not replace the recommended next lesson.

Arcflow Plus is coming — review drills, research breakdowns, more AI. Get one email at launch.

One email at launch. That's it.