What Is Model Distillation in AI? Teacher-Student Training Explained
A clear breakdown of AI model distillation: soft labels, Hinton's original method, on-policy distillation, and why the term gets misused in AI news.
Drafted with Claude from source material, checked by automated verification, and reviewed before release. How we make these.
What is model distillation in AI?
Model distillation is the process of training a smaller “student” model to reproduce the behavior of a larger, more capable “teacher” model. Instead of just copying the teacher’s final answers, true distillation trains the student on the teacher’s full probability distribution over possible outputs, known as soft labels. This lets the student absorb patterns of reasoning and generalization the teacher learned, not just its surface-level responses, which is why distillation can compress expensive frontier models into cheaper ones without losing as much capability as you’d expect.
TL;DR
- Distillation trains a smaller student model to mimic a larger teacher model’s output distribution, not just its final answer.
- Soft labels, introduced in Geoffrey Hinton’s 2015 paper on knowledge distillation, capture the teacher’s full probability spread across every possible class or token, including small probabilities on “wrong” answers that reveal how the model generalizes.
- A temperature parameter flattens the teacher’s logits so those small, informative probabilities become large enough for a student to actually learn from.
- True distillation requires access to logits or log probabilities, something most commercial APIs from major labs deliberately withhold, exposing only the single most likely output.
- Sequence-level distillation (sometimes called output harvesting) just trains on a teacher’s finished text outputs and transfers style, not deep capability, which is a much weaker technique than the textbook method.
- On-policy distillation has the student generate its own answers while the teacher scores each token, acting like a reward model, and research shows this can match reinforcement learning results using an order of magnitude less compute.
- Confusion between these different techniques is fueling a lot of the recent public debate over whether certain open models were “distilled” from Western frontier models.
One coffee. One working app.
You bring the idea. Remy manages the project.
Where does distillation actually come from?
The foundational method traces back to a 2015 paper by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean called “Distilling the Knowledge in a Neural Network.” The core insight: a large model or ensemble of models learns rich internal representations during training, but deploying that large model everywhere is expensive. Their solution was to train a compact model that reproduces the larger model’s behavior, described in the paper’s own framing as compressing an ensemble’s knowledge into a single, deployable model.
The mechanism relies on the difference between hard labels and soft labels. In a simple image classification example, a handwritten “2” gets a hard label: 100% probability on “2,” zero everywhere else. This is one-hot encoding, and it’s how you’d label training data by hand. But a trained teacher model, when it classifies that same image, doesn’t just output “2.” It outputs a full distribution: maybe 90% for “2,” but small non-zero probabilities for other digits too, say a bit more probability mass on “7” than on “3” because the “2” has a flat top that resembles a “7.”
Hinton called those small, seemingly “wrong” probabilities dark knowledge. They’re not noise. They encode how the model perceives similarity and structure across classes, which is exactly the kind of generalization a student model benefits from learning. The problem is those probabilities are often too small to train on directly. Hinton’s fix was a temperature parameter that divides the logits before the final output layer, flattening the distribution so the relative relationships between classes become learnable signal instead of vanishing noise.
How does this apply to large language models?
Language models are next-token predictors, but at each step they’re solving a classification problem across their entire vocabulary, assigning a probability to every possible next token. That means the same soft-label approach applies directly: instead of training a student only on the single token the teacher would output, you train it on the teacher’s full probability distribution across all candidate tokens at each position.
This is where the practical wall shows up. Reproducing that distribution requires access to the teacher’s logits, the raw, pre-softmax scores for every token in the vocabulary. Consumer-facing chat products and most commercial APIs don’t expose this. You get the model’s chosen output, sometimes with a limited set of top log probabilities, but not the full distribution needed for genuine distillation. Frontier labs that build products like Anthropic’s API generally return finished sequences and, in the case of chain-of-thought reasoning, a summarized version of the reasoning trace rather than the raw one. That makes real distillation from those APIs structurally difficult for anyone outside the lab itself.
Built like a system. Not vibe-coded.
Remy manages the project — every layer architected, not stitched together at the last second.
Inside frontier labs, though, distillation is routine. Because you generally can’t compress a multi-trillion-parameter model straight down to a few billion parameters in one step, labs use intermediate stages: a huge model distills into a medium-sized model, which then distills into something smaller still. DeepSeek’s R1 release is a public example of this pattern, where the full R1 model was distilled into several smaller model variants.
What is sequence-level distillation, and why is it weaker?
Sequence-level distillation, described in a 2016 paper by Kim and Rush, sidesteps the logit-access problem entirely. If you can’t see the teacher’s probability distribution, you sample a large volume of its finished text outputs and fine-tune your model on those outputs as ordinary training data. This is functionally close to supervised fine-tuning.
At scale, this looks like automated harvesting: querying a frontier model repeatedly through accounts or proxies, collecting responses, deduplicating them, and using the resulting dataset to fine-tune a separate model. Early open efforts like Alpaca and Vicuna used exactly this approach with outputs sourced from OpenAI’s API.
The catch is that this method transfers style and surface patterns, not the deeper reasoning capability that comes from the teacher’s actual probability distribution. It’s imitation, not capability transfer. It’s also generally a violation of API terms of service rather than a scalable path to building a genuinely competitive model. Anthropic reported in February detecting large-scale automated querying patterns consistent with this kind of harvesting, tied to a large number of flagged accounts, which illustrates how common the practice is, even though its actual capability payoff is limited.
What is on-policy distillation and why does it matter now?
On-policy distillation flips the direction of who generates the text. Instead of training on the teacher’s outputs, the student generates its own answer, token by token, and the teacher evaluates that answer using its own log probabilities, essentially acting as a judge or reward model. This is sometimes grouped under reinforcement learning through AI feedback.
The appeal is efficiency. Research on this technique has shown it can lift benchmark performance (one cited case involved AIME 2024 scores) using dramatically less compute than standard large-scale reinforcement learning, on the order of roughly ten times less GPU time in the comparisons discussed. The student learns from being corrected on its own actual mistakes rather than absorbing a teacher’s independently generated text.
The limitation is the same one that shows up everywhere in distillation: you still need access to the teacher’s log probabilities, which paid commercial APIs typically don’t expose. A full reinforcement learning run for a frontier-class model can require tens of millions of completed rollouts, a scale that’s simply not achievable through metered public API access.
Why does this matter for how people talk about distillation in AI news?
Other agents ship a demo. Remy ships an app.
Real backend. Real database. Real auth. Real plumbing. Remy has it all.
Most public discussion collapses all of this into one word. When a new open-weight model scores close to a proprietary frontier model shortly after that frontier model launches, the immediate assumption is often “they must have distilled it.” But the mechanics matter. Copying outputs from an API gives you style transfer at best. Real capability transfer, the kind that would let a smaller lab match a frontier model’s reasoning ability, requires access to logits or log probabilities that commercial APIs don’t provide, plus a training pipeline (pretraining, mid-training, supervised fine-tuning, reinforcement learning) that takes months to execute and typically has its data locked in well before a competing frontier model even launches.
That timing detail is often the strongest evidence against simplistic distillation claims: if a competing model’s training data was finalized months before a rival frontier model’s public release, there’s no window in which output harvesting from that rival could have shaped the earlier model’s core capabilities.
Frequently Asked Questions
What’s the difference between distillation and fine-tuning?
Fine-tuning adjusts a model using labeled examples of desired input-output pairs. Distillation specifically trains a student model to match a teacher model’s output distribution (soft labels or log probabilities), which can transfer more nuanced generalization than simple example-based fine-tuning.
Can you distill a model without access to its logits?
You can approximate style through sequence-level distillation, training on a teacher’s finished outputs, but this transfers surface patterns rather than deep capability. Genuine capability transfer requires the teacher’s probability distribution, which most commercial APIs don’t expose.
Why do frontier labs bother distilling models at all?
Serving a massive, multi-trillion-parameter model to every user is prohibitively expensive. Labs distill smaller student models that approximate the teacher’s capabilities at a fraction of the inference cost, which is why most consumer-facing model variants are distilled versions of larger internal models.
What is on-policy distillation used for?
It’s used to improve a student model efficiently by having the student generate its own answers while the teacher scores each token as a reward signal. Research suggests it can achieve reinforcement-learning-level gains with substantially less compute, though it still depends on having access to the teacher’s log probabilities.
Does using an AI model’s output to train another model count as distillation?
Only in a limited sense. That’s sequence-level distillation, and it mainly transfers style and phrasing rather than the reasoning capability that comes from training on a full probability distribution. It’s also typically a violation of API terms of service.
