How a big model teaches a small one

A model with billions of weights (the numbers it adjusts as it learns) is too big for many jobs: it needs a lot of memory, and every answer takes time. Distillation gets a smaller model that behaves like it. First a large teacher is trained. Then a small student is trained to copy the teacher’s doubts as well as its answers. Hinton, Vinyals and Dean’s 2015 paper set out the method.

It is not quantization. Distillation removes weights: the student is a different, smaller model. Quantization removes bits: the model stays and each weight is stored more coarsely.

Every figure is a toy and says what is invented. This post makes no claim about how any RuneLab model was made.

By the end of this post you will know:

• why a teacher’s wrong answers are worth teaching, and what the temperature has to do with it;
• how a tiny student learns from a few examples with and without its teacher’s soft targets, and why they help;
• what a student sees when the teacher is a language model, and how this differs from quantization.

A classifier shown a photo of a ginger cat says “cat”, and so does a label. But before it picks a winner it gives every class a score, and the wrong classes’ scores are not equal: a cat looks more like a fox than like a bus.

Softmax turns scores into shares that add up to 100 percent, and a setting called the temperature, T, decides how sharp they are. At T = 1 a confident teacher is very sure. Drag T up and watch the four wrong classes.

This figure needs JavaScript. It shows a teacher’s invented scores for a photo of a ginger cat (cat 9, fox 5.5, dog 5, bus 1, carrot 0) turned into shares by softmax with temperature T. At T = 1 the teacher says cat with 95 percent and the other four classes share under 5 percent. At T = 4 it says cat 49, fox 21, dog 18, bus 7 and carrot 5 percent, so the wrong answers that a cat is closest to become visible. At T = 10 the shares are nearly flat, with cat at 31 percent. The order of the classes never changes.

Around T = 4 fox gets 21 percent and dog 18, against 7 for bus and 5 for carrot: the teacher says which wrong answers are nearly right. Temperature only flattens the shares; the order never changes, and the specks grow big enough for a student to notice. Shares used as training targets are called soft targets.

Why not just use the shares at T = 1? The wrong answers are still there. They are: even at T = 1 fox is 90 times as likely as bus. But a share of 0.03 percent barely registers in training. A raised temperature makes the wrong answers easier to read; the student is used at T = 1 afterwards.

That is the 2015 paper’s idea: soft targets carry more information per training example than hard labels. On MNIST digits a small network trained on the labels, with no regularisation (no guard against overfitting), made 146 test errors; the same network, also trained to match a large network’s soft targets (T = 20), made 74. The large network made 67.

Three kinds of dots sit on a plane in overlapping clouds, so no border gets every dot right. A teacher network (99 weights), trained beforehand on 3,000 labelled dots, draws the thin lines. Two identical tiny students (33 weights each) start from the same random weights and see the same 15 labelled dots. One is trained on the labels alone. The other is nudged until its own shares, at the same T = 3, match the teacher’s for those dots. (Real recipes usually keep the labels too; this student never sees them.)

Press “Train both” to watch 200 real steps of gradient descent (each nudges every weight a little, so the answers fit the targets better), then drag the learning step to scrub it. A ring marks a dot whose label the teacher disagrees with.

This figure needs JavaScript. It trains two tiny networks, in the page, on the same 15 labelled dots of a toy problem with three kinds of dots in overlapping clouds. One student learns from the labels alone; the other learns from a teacher network’s soft targets for the same dots. Over 100 different samples of 15 dots, the labels-only student averages 88 percent correct on 900 fresh dots and the soft-target student 92 percent; the teacher gets 93. The soft-target student was ahead in 94 of the 100 samples. It is a toy of dots on a plane, not a language model.

Almost everything that matters happens in the first 60 steps. The clouds overlap, so some labels sit on the wrong side of the teacher’s borders: the ringed dots. The labels-only student takes them at their word. The soft-target student does not, because the teacher’s shares for a ringed dot lean to the other kind.

One sample can mislead either way, so press “Other 15 dots” a few times, then “Repeat on 100 samples”. At T = 3 the soft-target student averages 92 percent against 88 for labels alone (the teacher: 93) and is ahead in 94 of the 100 samples (tied in 1, behind in 5). The first sample is the most typical. The settings are tuned for neither student.

Both students saw the same 15 dots. Where did the extra information come from? From the teacher, which had seen 3,000 dots. A label says “square”, even for a dot out among the triangles. The teacher’s shares say “probably a triangle”, or “square, but close to the triangles”: a position as well as a name.

Where does the gain (92.2 against 88.1) come from? Give a student only the teacher’s best guess as a label, with no shares, and it reaches 89.9: that closes 1.8 of the 4.1 points, mostly by overriding misleading labels. Spreading each label evenly over the three kinds, with no teacher, reaches 88.4. The rest needs the shares, which say how near a dot is to a border. Split the samples by ringed dots and the labels-only student falls from 90.5 percent (none ringed, 39 samples) to 83.8 (three or more, 9 samples); the soft-target student stays near 92.

Temperature matters less here than figure 1 suggests. Pick T = 1, 3 or 10 and repeat: soft targets average 91.7, 92.2 and 92.3 percent, all above the labels and within a point of each other. A toy this tidy flatters the method.

In the figure both students saw the same 15 dots, yet the soft-target student did better. What made the difference?

1. It trained for more steps than the other student, so it had longer to improve and to fit the dots.
2. The teacher added more dots, so it had more examples to learn from.
3. The teacher’s shares overrode misleading labels and showed how near a dot is to a border.

Show answer

Both, in the toy. The teacher’s best guess alone closed 1.8 of the 4.1 points, by overriding misleading labels (the ringed dots); the rest needed the shares, which say how near each dot is to a border.

Not A: both students took exactly 200 steps from the same starting weights.

Not B: the dots were the same 15. The teacher added shares to them, not dots.

A language model plays the same game at every position of a text: given the words so far, how likely is each word of its vocabulary to come next? Those probabilities are the teacher’s shares. A student can be given what the teacher knows in three ways: one word the teacher wrote (training on text it generated), its top three shares, or every share. The figure shows one step with eight words and invented probabilities.

This figure needs JavaScript. It shows one step of a toy language model with eight words and invented probabilities: cup 46 percent, mug 22, glass 12, bowl 7, pot 5, jar 4, shoe 3 and moon 1. With one written word the student hears about one word of eight, and if the teacher wrote “cup” then 54 percent of its belief never reaches the student; asked again it may write “mug”. With the top three shares the student hears about three words and 20 percent of the belief is left out. With every share nothing is left out.

With one word, the target says 100 percent for a word the teacher never fully believed; ask again and it may write another. Over very many examples it still works, and it is the cheapest to get: it needs only the teacher’s text. (The figure draws a random word to show the noise; real recipes often use best-scoring or filtered outputs.)

In 2016 Kim and Rush trained translation students on their teacher’s best outputs, found with beam search, a slower search that keeps several candidate sentences alive. Their best student ran about ten times faster than its teacher (partly by needing no beam search) with little loss in quality.

The DeepSeek-R1 report fine-tuned (trained further) six smaller open models on about 800,000 samples curated with R1, with supervised fine-tuning only.

The top three name the runners-up, but a fifth of the belief is still missing. Every share needs the teacher’s own outputs, not only its words: the Gemma 2 report trains its 2B and 9B models on the teacher’s whole distribution for each next word, instead of only the word that came next.

So can I ask a big model’s API a million questions and train a small model on the answers? Technically that is easy. Whether you may is another matter: a provider’s terms, or a model’s licence, can restrict using its outputs to train another model. Read them first.

A service returns only the text of a big model’s answers. Which of the figure’s three kinds of supervision can you build?

1. One written word per position: the words of its answers.
2. The top three shares, because every service returns them.
3. Every share, since the text is the distribution written out.

Show answer

The words it wrote, the first kind. Text is a sample from the teacher’s distribution, not the distribution itself.

Not B: some services return a few top probabilities and many do not, and this one returns text only.

Not C: a written word is one draw from the teacher’s shares; ask again and it may differ, as in the figure above.

Distillation and quantization shrink a model along different axes. A model’s weights take (number of weights) × (bits per weight) of memory. Distillation cuts the first factor, since the student has fewer weights; quantization cuts the second. So the savings multiply. Pick sizes and bits below (round examples, not any product’s).

This figure needs JavaScript. It is a calculator for the memory of a model’s weights, which is the number of weights times the bits per weight, divided by eight. A teacher of 70 billion weights at 16 bits takes 140 GB; a student of 8 billion weights at 16 bits takes 16 GB, 8.75 times less; the same student at 4 bits takes 4 GB, which is 35 times less than the teacher. The sizes are round examples, only the weights are counted, and quality is not shown.

The figure leaves out quality, and picking a teacher more than ten times the student’s size adds a caution: large gaps are where students suffer (next section).

For quality, one verified example: the DistilBERT authors report 40 percent fewer weights than BERT (66 million against 110 million), 60 percent faster inference and 97 percent of its language-understanding score, measured on a CPU, one example at a time. Those numbers belong to that model and those tests. Doing both steps can cost quality twice, and the costs need not add up: test the combined model on your own task.

Distillation gives a small model a good teacher’s habits, and nothing more:

• The gap. A student much smaller than its teacher absorbs less. Mirzadeh and colleagues report worse students when the gap is large, and add a middle-sized “teacher assistant”.
• Inherited mistakes. The student sees the problem only through the teacher. If the teacher is wrong or biased, the copy is too (in the toy, the soft-target student stayed a little behind its teacher on average).
• The questions you ask. A teacher only labels what you show it: inputs unlike real use give a student good at the wrong thing.

Distillation also helps speculative decoding: DistillSpec distils the large model into the small guessing model so that more guesses are kept, and reports 10 to 45 percent speed-ups over standard speculative decoding on its tests.

The teacher gets 7 of every 100 dots wrong. What happens to a student that learns to copy it?

1. It learns to be right where the teacher is wrong, because a smaller model is always simpler.
2. It can copy the teacher’s mistakes, since it learns the problem through the teacher.
3. The mistakes drop out, because soft targets always put most weight on the right kind.

Show answer

It can: a student trained on the teacher cannot tell where the teacher is wrong. In the training figure it ended a little behind on average (92 against 93).

Not A: being smaller does not tell a student where its teacher is wrong.

Not C: soft targets are the teacher’s own opinion. Where the teacher is wrong, they put weight on the wrong kind.

• A teacher’s wrong answers carry information, and a raised temperature makes it easier to read.
• Soft targets can teach more per example than labels (in small-data cases like this one), and a student that sees only the teacher’s text gets one word per step.
• Distillation removes weights and quantization removes bits: the savings multiply, and the quality cost is yours to measure.
• Check the teacher’s terms and licence, and test the student on your own task.

Every outside number and claim, checked on 14 October 2026.

1. Hinton, Vinyals and Dean (2015), “Distilling the Knowledge in a Neural Network”.
2. Kim and Rush (EMNLP 2016), “Sequence-Level Knowledge Distillation”.
3. Sanh, Debut, Chaumond and Wolf (2019), “DistilBERT, a distilled version of BERT”.
4. DeepSeek-AI (2025), “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”, section 2.4 of version 1.
5. Gemma Team (2024), “Gemma 2: Improving Open Language Models at a Practical Size”.
6. Mirzadeh and colleagues (AAAI 2020), “Improved Knowledge Distillation via Teacher Assistant”.
7. Zhou and colleagues (2023), “DistillSpec: Improving Speculative Decoding via Knowledge Distillation”.