Speculative decoding
A small model guesses a few tokens ahead and the big model checks them all in one pass: faster, with the same output distribution.
Read first:Prefill and decodeSampling and decoding
Step 1 of 6· Illustrative tokens, guesses and probabilities; the acceptance rule and speed-up formula are real
A large model writes one token per full pass, and each pass is slow because it must read all of its weights. Speculative decoding adds a small, fast draft model that guesses the next few tokens. The large model then checks all the guesses together in a single pass, which takes about as long as producing one token. It keeps the guesses it agrees with, replaces the first one it disagrees with, and carries on from there. A careful accept-or-reject rule means the text follows exactly the distribution the large model would have produced alone. When the draft guesses well, several tokens arrive for the price of one pass.
Why it matters for your product
Speculative decoding cuts the time per output token without retraining the large model or changing its output distribution: the original paper measured 2 to 3 times faster generation on T5-XXL, and a DeepMind team reported 2 to 2.5 times on Chinchilla 70B. It suits latency-sensitive traffic such as chat and code completion. Gains depend on how predictable the text is, so measure acceptance rates on your own prompts, and expect less from busy servers where large batches already keep the arithmetic units occupied.
For engineersShow the maths
E[tokens per target pass] = (1 − α^(γ+1)) / (1 − α)
Each guess survives with probability α, the round stops at the first rejection, and the target always contributes one token of its own.
Worked example: α = 0.8, γ = 4: (1 − 0.8⁵) / 0.2 = (1 − 0.328) / 0.2 ≈ 3.36 tokens for every big pass.
Speed-up = (1 − α^(γ+1)) / ((1 − α)(γc + 1))
Tokens per round divided by the cost of a round: one target pass plus γ draft passes, each costing c of a target pass.
Worked example: α = 0.8, γ = 4, c = 0.05: 3.36 / 1.2 ≈ 2.8 times faster than plain decoding.
Keep x with probability min(1, p(x) / q(x)); otherwise sample from norm(max(0, p − q))
Guesses the target likes at least as much as the draft are always kept; over-confident guesses are thinned out, and the leftover probability is handed back through the replacement.
Worked example: In the visual's example, the target gives “1887” 5% and the draft 45%, so the guess survives 0.05 / 0.45 ≈ 11% of the time; otherwise the leftover mass sits entirely on “1889”, which is chosen.
It only helps when the draft often agrees with the target and the hardware has spare compute during decode: with low acceptance it is slower than plain decoding, large batches leave little idle compute to use, and it does more total arithmetic. In the standard method the draft and target share a vocabulary, and the draft is one more model to serve.
The words you will hear
Draft model
Primary sources
- 01Fast Inference from Transformers via Speculative DecodingLeviathan et al. · 2022
- 02Accelerating Large Language Model Decoding with Speculative SamplingChen et al. · 2023
- 03Blockwise Parallel Decoding for Deep Autoregressive ModelsStern et al. · 2018
- 04Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding HeadsCai et al. · 2024
Connected ideas
Generating text
KV cache
Why a model keeps the keys and values of earlier tokens instead of recomputing them, and what that memory costs.
ExploreGenerating text
Next-token prediction
A language model writes by guessing one token at a time, then feeding each guess back in.
ExploreTraining and alignment
Distillation
Training a small model to imitate a large one, including what the large one nearly said.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.