Sampling and decoding
How a list of probabilities becomes one word, and the dials that trade steadiness for variety.
Read first:Next-token prediction
Step 1 of 6· Illustrative scores and beam probabilities; the temperature, top-k, top-p, entropy and sampling maths is real
At every step a language model hands over a probability for each possible next token. Decoding is the rule for picking one. Greedy decoding always takes the favourite: predictable, but it can turn dull and repetitive. Sampling rolls a weighted die instead. Temperature reshapes the die before the roll: low values make the favourite even more likely, high values give rare words a better chance. Top-k and top-p trim the unlikely tail so a bad roll is less likely to land on nonsense. Model libraries expose these dials as temperature, top-k and top-p settings, and because sampling is random, the same prompt can give different answers.
Why it matters for your product
Decoding settings are among the cheapest levers on a system's behaviour. For extraction, classification or code, where there is one right answer, a low temperature or greedy decoding makes results steadier and easier to test. For drafting and brainstorming, a moderate temperature with top-p gives variety without nonsense. Randomness matters for evaluation too: if a system samples, run each test case several times, because a single pass can hide how often it fails.
For engineersShow the maths
pᵢ = exp(zᵢ / T) / Σⱼ exp(zⱼ / T)
Divide every score z by the temperature T, then apply softmax as usual. T = 1 leaves the model's distribution unchanged.
Worked example: Two tokens scoring 2 and 1 get 73% and 27% at T = 1. At T = 0.5 the gap doubles, giving 88% and 12%; at T = 2 it halves, giving 62% and 38%.
V(p) = the smallest set of top tokens with Σ P(x) ≥ p; P′(x) = P(x) / Σ_{y ∈ V(p)} P(y)
Top-p: walk down the sorted list until the running total reaches p, cut the rest, and rescale what is left to sum to one.
Worked example: Probabilities 0.5, 0.3, 0.15 and 0.05 with p = 0.9: the top three reach 0.95, so the last token is cut and the rest become 0.53, 0.32 and 0.16.
H = −Σᵢ pᵢ · log₂ pᵢ
Entropy is the probability-weighted average of each outcome's surprise, −log₂ p. 2 to the power H is the number of equally likely options it is like.
Worked example: A fair coin is 1 bit. Probabilities 0.5, 0.25 and 0.25 give 0.5 × 1 + 0.25 × 2 + 0.25 × 2 = 1.5 bits, like choosing among about 2.8 options.
Temperature and truncation only reshape what the model already believes: if the right answer has low probability, no setting will reliably find it. Beam search finds more probable text, not better text; on open-ended writing it tends to be bland and repetitive, and every extra beam adds computation.
The words you will hear
Greedy decoding
Primary sources
- 01The Curious Case of Neural Text DegenerationHoltzman et al. · 2019
- 02Hierarchical Neural Story GenerationFan et al. · 2018
- 03Distilling the Knowledge in a Neural NetworkHinton et al. · 2015
- 04Beam Search Strategies for Neural Machine TranslationFreitag, Al-Onaizan · 2017
- 05A Mathematical Theory of CommunicationShannon · 1948
- 06Generation (Transformers documentation)Hugging Face · 2026
Connected ideas
Generating text
Next-token prediction
A language model writes by guessing one token at a time, then feeding each guess back in.
ExploreGenerating text
Speculative decoding
A small model guesses a few tokens ahead and the big model checks them all in one pass: faster, with the same output distribution.
ExploreObservability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreInterpretability and safety
Hallucination
Why models state wrong answers with confidence, and how to trade a few answers for a lot of trust.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.