How a small model speeds up a big one

Ask a chatbot a question and watch the answer arrive, a few words at a time. A language model writes one token at a time: it reads everything written so far, picks the next token, and starts again. (A token is a small piece of text, often a word or part of one; the figures say “word”.)

Every one of those steps is a full pass through the model, and for a large model a pass touches a great many numbers. Speculative decoding lets the large model write several tokens in one pass without changing the odds of what it writes. It was described in 2022 by Leviathan, Kalman and Matias, and in a closely related form in 2023 by Chen and colleagues.

Everything inside the figures is a toy: the probabilities are invented and the durations are made up, never measured on a machine. The procedure and the formulas are the papers’, and any number from a paper is named as theirs.

By the end of this post you will know:

• why writing a token costs a whole pass while checking several existing tokens costs about one;
• how a small model’s guesses are checked in one go, and why the text keeps the same odds;
• how to estimate the speed-up, and when it is small or even negative.

A large model is a very long list of numbers, its weights. To write one token, the chip brings those numbers in from memory, uses them on the text so far and works out the odds for the next token. The arithmetic is quick; the trip from memory is the slow part. People say decoding is memory-bound: for one stream of text, speed is set by how fast the weights move, not by how fast the chip adds and multiplies.

Each box in the figure is one pass: the blue strip is the weights travelling, the hatched strip is the arithmetic, and the box lasts about as long as the longer of the two. Press “Write five tokens”, then “Check five at once”.

This figure needs JavaScript. It compares two ways of handling five tokens with illustrative durations. Writing five tokens takes five passes of the large model, one after the other. Checking five tokens that are already written takes one pass, only slightly longer than one write, because the long part of a pass (moving the model's weights from memory) is the same whatever the pass holds. When the machine also serves many other requests that check five tokens each, the arithmetic grows until checking five costs as much as writing five, from about twelve other requests.

Five passes to write five tokens, one pass to check the same five. Reading five tokens costs about one pass; writing them takes five. A model can read tokens that already exist because it works out its odds for the next token at every position of a text in parallel, which is also how it reads your prompt.

So the fast chip is mostly waiting for memory? For one stream of text, often yes, and that waiting is the gap this trick uses. The arithmetic has room to spare, as long as nothing else is using it.

Now drag the slider. Each other request is another user whose pass also checks five tokens, so the arithmetic grows: with a few of them the check no longer hides behind the weights, and from about twelve it costs as much as writing five. A busy machine has no spare arithmetic; the vLLM documentation rates the technique’s benefit lower at high request rates.

If checking is cheap, what should we check? A second, much smaller model can supply candidates: it has far fewer weights to move, so each of its passes is cheap, and it often guesses the word the large model would. The plan:

1. The small model guesses a few tokens ahead, one after another.
2. The large model reads all of the guesses in a single pass and works out, at every position, what it would have written.
3. Guesses are kept until the first one that is rejected. There the large model writes its own token, and the guesses after it are thrown away.

But guess three depends on guess two. How can the large model read all of them in one go? It reads them as if they were all true. If guess two turns out wrong, everything after it is thrown away anyway, so nothing built on a wrong guess survives.

This figure runs the exact rule (next section) on ten words. Press “Small model guesses”, then “Large model checks”, and watch the counters: large-model passes, tokens written and small-model runs, since every guess costs one. Then try the other text, the other small model, more or fewer guesses and “New dice”. The line under the counters averages thousands of runs, so one lucky run proves nothing. One warning: here the odds at a position ignore the words chosen before; a real model’s odds depend on everything before them.

This figure needs JavaScript. It steps through speculative decoding on a toy text of about ten words with invented probabilities. In each round a small model guesses several words ahead, and one pass of the large model keeps the guesses up to the first rejected one, writes its own word there and discards the rest. With a good small model on a predictable reply, ten tokens take about 2.6 large-model passes on average over many dice rolls; with a poor small model on an open-ended sentence, nine tokens take about 5.3. Plain decoding needs one pass per token.

At four guesses per round, the good guesser on the predictable reply wrote its ten words in about 2.6 large-model passes on average; the poor guesser on the open-ended sentence needed about 5.3 for nine. A rejected guess costs you the guesses after it and nothing else, because that pass still wrote one token, as plain decoding would have; but each thrown-away guess was a small-model run.

In one round the small model guesses four words. The large model keeps the first two, rejects the third and writes its own word there. How many tokens does that round write, using one large-model pass?

1. Two, because only the kept guesses are written down.
2. Three: the two kept guesses and one word from the large model.
3. Four, because the large model reads all four guesses and writes every one of them.

Show answer

Three. Every round ends with a word from the large model: the replacement at the first rejected guess, or one extra word when every guess was kept.

Not A: the large model’s own word is written too.

Not C: the rejected guess and the one after it are thrown away.

Wait. If the small model’s guesses are kept, won’t the text end up sounding like the small model?

The rule is built so that it does not. The promise is about odds: the words that come out follow exactly the large model’s odds, as if it had written them alone. It is not the same text from the same dice (plain decoding differs each time too). It is the same distribution. For one position, call the large model’s odds for a word p and the small model’s q:

1. The small model draws a guess from its own odds, q.
2. The large model keeps the guess with chance min(1, p/q): always, if it likes the word at least as much as the small model did, and otherwise in proportion.
3. If the guess is not kept, the large model draws a replacement from what is left over: the words it likes more than the small model did, in proportion to how much more.

(When every guess in a round is kept, the same pass also yields one more word from the large model’s own odds.)

Why draw the replacement from what is left over? Why not from the large model’s own odds? Because the guesses already took their share. Drawing from the full odds again would count some words twice; the leftover is exactly the part that is missing, as the sums in the figure show.

One position, five words. In the top panel each word has two bars: the amber bar is the small model’s odds (drag it, or focus it and use the arrow keys), and the blue bar is the large model’s odds in two parts, the guesses that get kept (solid) and the replacements that top it up (hatched). The numbers under a word add them up. Then press “Run 10,000 times” for real draws.

This figure needs JavaScript. It shows one position with five possible words. The large model's odds are 42, 26, 16, 10 and 6 percent; the small model starts at 20, 45, 10, 5 and 20 percent, so a guess is kept 67 percent of the time. For the first word the large model's 42 is the 20 that kept guesses supply plus the 22 that replacements supply. After thousands of draws with the keep-or-replace rule the words that come out sit on the large model's odds, within about a point, while the small model on its own stays at its own odds, up to 22 points away on one word. Changing the small model's odds changes how often guesses are kept, not what comes out.

Take waves: the large model gives it 42 percent, the small model 20. That 20 percent of guesses is always kept, since the large model likes the word more; replacements supply the other 22. Guesses are rejected 33 percent of the time, and those replacements go to the words the large model likes more than the small one: waves by 22 points, gulls by 6, clouds by 5, which makes 33. For boats the small model guesses too often (45 against 26), so only 26 of its guesses survive and it is never a replacement: 26 + 0 = 26. Over-guessed words are trimmed, under-guessed ones topped up, and the two always cancel. The first paper proves this for any pair of models; the second proves the same for its version.

After 10,000 draws the blue bars sit on the ticks within about a point, split as the top panel says; the hatched amber bars follow the small model and miss them. Move a bar and the blue ones stay; what changes is the “kept” line, and with it how much time you save.

So a terrible small model can’t make the answers worse? Not the answers. It can make things slower, because more of its guesses get rejected. That is the next figure.

You swap in a much worse small model. According to the rule, what changes in the text the large model produces?

1. It gets worse, because more of the small model’s mistakes are kept.
2. It becomes more like the small model’s style, because its guesses come first.
3. Nothing: more guesses are rejected, so the only cost is time.

Show answer

The words still follow the large model’s odds, whatever the small model is; a worse one is kept less often, so more guesses are wasted. Drag the amber bars far away in the figure above and watch the blue bars stay put while the “kept” line falls.

Not A: a guess is only kept as often as the large model’s own odds allow.

Not B: after many draws the style is the large model’s, not a blend.

Two limits. The proof is about exact arithmetic; the papers and the vLLM documentation say real software holds it only up to numerical precision. And “same distribution” says nothing about whether the large model is right.

Two numbers decide the speed-up: how often a guess is kept, which the papers call alpha (α), and the cost of one small-model run compared with one large-model pass, which they call c (at 5 percent, twenty small runs cost one large pass). A round with γ (gamma) guesses costs 1 + γc passes. It writes at least one token, a second if the first guess was kept (chance α), a third if the first two were (α²), and so on up to γ + 1. So the expected tokens per large-model pass are 1 + α + α² + … + α^γ, which equals (1 − α^γ+1) / (1 − α), and the speed compared with plain decoding is that divided by 1 + γc.

These are the first paper’s formulas and assumptions: every guess is kept with the same chance, independently, and checking γ + 1 positions costs one pass at any γ. Real acceptance varies by position, and real checks stop being cheap well before twelve tokens (the first figure), so real curves can peak earlier.

Every guess kept with the same chance? That sounds convenient. It is what makes the formula tidy. The toy figure showed the real picture: easy words are kept far more often than hard ones. Use the curve for its shape, and measure for the number.

Drag the first slider, “Guesses per round”: the blue dot slides along the curve, and the dashed ring marks the best number of guesses for the other two settings. Then change those two, and press “Try a weak, costly small model”.

This figure needs JavaScript. It plots the first paper's formulas. With a 70 percent chance that a guess is kept and a small-model run that costs 10 percent of a large-model pass, two guesses per round give about 1.8 times the speed of plain decoding, four give the best result, about 2.0 times, and twelve fall back to about 1.5 times. With a 30 percent chance and a cost of 50 percent, four guesses give about 0.48 times: slower than plain decoding. Gains need the chance of keeping a guess to be higher than the cost.

At 70 percent and 10 percent, one guess already gives 1.5 times the speed of plain decoding and four give the best, a little under 2 times. Then the curve falls: an extra guess only matters if every guess before it was kept, which gets ever less likely, yet it costs the same every time.

The last button is the warning: a small model kept 30 percent of the time that costs half a large pass runs at about half the speed of plain decoding. No number of guesses helps unless the chance of keeping a guess is higher than its cost. For scale, the authors report two to three times faster for T5-XXL, a translation and summarisation model, with the same output distribution, on one accelerator with one request at a time (first paper), and two to two-and-a-half times for a very large language model on sixteen accelerators (second paper). Those numbers belong to those models, tasks and machines, not to any other.

A small model is kept 30 percent of the time and costs half a large-model pass to run. With four guesses per round, what do the formulas say?

1. Slower than plain decoding: about half the speed.
2. About 1.4 times faster, because each pass writes about 1.4 tokens.
3. About 3 times faster, because up to five tokens can be written in a pass.

Show answer

Slower. A round writes 1 + 0.3 + 0.09 + 0.027 + 0.008 ≈ 1.43 tokens on average, but it costs four small runs at 0.5 plus one large pass: 3 passes of time. 1.43 divided by 3 is about 0.48.

Not B: 1.4 is the tokens per pass. It forgets that the round also pays for four small-model runs.

Not C: five is the most a round can write, and it needs every guess kept.

Speculative decoding spends spare arithmetic and a second, cheaper model to save passes of the large one. It helps most with one stream of text or a few, a small model that agrees with the large one often (similar training text, the same vocabulary), easy-to-predict text (boilerplate, code, answers that repeat the question) and a cheap small model. It helps little, or hurts, when the machine is busy, when the small model is a poor match (acceptance varies between domains, the second paper reports), when there is no room for a second model, and when a guess is kept less often than it costs.

Variants guess in other ways and trade different things; whether the text is provably unchanged differs from one to the next, so read each paper’s claim:

• Prompt lookup decoding: no second model; it copies what followed the last few tokens in the prompt. Its README measured summarising and answering questions about a document, expects code editing to help too, and finds little where there is nothing to copy. A RAG answer that quotes its sources repeats its prompt (How RAG finds answers).
• Self-speculative decoding: the large model drafts with some of its own layers skipped, so there is no second model or extra training. Its authors report up to about 2 times on one model family.
• Medusa: extra prediction heads on the large model. Its authors report over 2.2 times with only the heads trained, and 2.3 to 3.6 times when heads and model are trained together, which changes the model.
• EAGLE: guesses from the large model’s internal features. Its authors report 2.7 to 3.5 times on one chat model and an unchanged text distribution.

Could I switch this on for a model I run at home?

A person chatting with a local model is the one-stream case of the first figure. Whether a given program supports it, which small model it would pair with yours and how much it would speed up your text are things to measure, not assume. This post makes no claim about any RuneLab model: it does not say that Hati or Sköll use speculative decoding, and it quotes no speed for either.

The source of every outside number and claim, as it stood on 2026-10-14 (the vLLM page is a living document).

1. Yaniv Leviathan, Matan Kalman and Yossi Matias, “Fast Inference from Transformers via Speculative Decoding” (2022): the rule, the proof (appendix A.1), the formulas, T5-XXL.
2. Charlie Chen, Sebastian Borgeaud, Geoffrey Irving, Jean-Baptiste Lespiau, Laurent Sifre and John Jumper, “Accelerating Large Language Model Decoding with Speculative Sampling” (2023): Theorem 1, sixteen accelerators.
3. The GitHub repository prompt-lookup-decoding and its README.
4. Jun Zhang and colleagues, “Draft & Verify: Lossless Large Language Model Acceleration via Self-Speculative Decoding” (2023).
5. Tianle Cai and colleagues, “Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads” (2024).
6. Yuhui Li, Fangyun Wei, Chao Zhang and Hongyang Zhang, “EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty” (2024).
7. The vLLM project’s documentation, Speculative Decoding.

Reading several tokens costs about as much as writing one, because for one stream of text a pass mostly moves weights. A small model guesses, the large model checks the guesses in one pass, and a keep-or-replace rule makes the words that come out follow the large model’s own odds. A bad small model costs time, not quality, and the gain falls away when the machine is busy.

• Ask for the speed-up on your own text and hardware, not a headline number.
• Check that the small model fits the work: same vocabulary, similar text, cheap to run.
• Read “the text does not change” as a property of the rule, up to rounding.