Distillation
Training a small model to imitate a large one, including what the large one nearly said.
Read first:Sampling and decodingFine-tuning and LoRA
Step 1 of 5· Illustrative teacher logits and a toy six-number student; the temperature softmax, T²-scaled loss and gradient steps are real maths; results as published
Distillation trains a small, cheap student model to copy a large teacher model. The key idea is to learn from the teacher's full set of probabilities, not just the right answer. If the teacher thinks the next word is “mug” at 56% but “cup” at 34% and “car” at almost zero, that tells the student which mistakes are reasonable. Raising the softmax temperature makes those small probabilities easier to learn from. The student ends up much smaller and faster while keeping much of the teacher's skill. With language models, the student can also simply be fine-tuned on text the teacher wrote, as DeepSeek did for its small reasoning models.
Why it matters for your product
A model's running cost is paid every time it answers, so a small model that behaves like a large one saves money and latency on every request, and may fit on a single GPU or a phone. DistilBERT and the DeepSeek-R1 distilled models were made this way. For a team, it is a route from a prototype that works on a large model to a product that is cheap to run: collect the large model's outputs on real inputs, train a small model on them, and evaluate carefully.
For engineersShow the maths
qᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)
Softmax with temperature T: dividing the logits by T above 1 squeezes them together and flattens the probabilities.
Worked example: Logits of 4.0 for “mug” and −4.0 for “car” differ by 8: at T = 1 “mug” is e⁸ ≈ 2,981 times likelier; at T = 4, only e² ≈ 7.4 times.
L = α · T² · CE(p_T, q_T) + (1 − α) · CE(y, q₁)
A weighted mix: match the teacher's softened distribution p_T (scaled by T² so gradients keep their size), plus a considerably smaller weight on the ordinary loss against the true label y at T = 1.
Worked example: At T = 4 the soft term is multiplied by 16, cancelling the 1/16 by which temperature shrinks its gradients.
∂C/∂zᵢ ≈ (zᵢ − vᵢ) / (N · T²)
At high temperature, with N classes and zero-mean logits, distillation reduces to making the student's logits zᵢ equal the teacher's vᵢ.
A student cannot learn what the teacher does not know, and it copies the teacher's mistakes and biases along with its skills. Smaller students keep less: in DeepSeek's results the 1.5B student scored well below the 7B and 32B students on AIME.
The words you will hear
Teacher
Primary sources
- 01Distilling the Knowledge in a Neural NetworkHinton et al. · 2015
- 02DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighterSanh et al. · 2019
- 03DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI · 2025
Connected ideas
Speed and cost
Quantisation
Storing a model's numbers with fewer bits, and keeping the answers nearly the same.
ExploreTraining and alignment
Pretraining and scaling laws
Why loss falls predictably with compute, and how to split a budget between model size and data.
ExploreGenerating text
Speculative decoding
A small model guesses a few tokens ahead and the big model checks them all in one pass: faster, with the same output distribution.
ExploreTraining and alignment
Fine-tuning and LoRA
Teaching a pretrained model a new job by training a small add-on instead of every weight.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.