Next-token prediction
A language model writes by guessing one token at a time, then feeding each guess back in.
Read first:TokenisationTransformer architecture
Step 1 of 6· Toy model with hand-set scores; the softmax, log-probability and perplexity maths is real
A language model does one thing, over and over: given the text so far, it predicts what comes next. It gives every token it knows a score, softmax turns the scores into probabilities, one token is chosen and added to the end, and the longer text goes back in to predict the token after that. A whole reply is this loop run hundreds of times. Training is the same game played on existing text: at every position the model is pushed to give high probability to the token that really came next. Perplexity measures how well it plays: roughly, how many options it was torn between.
Why it matters for your product
Because a reply is produced one token at a time, output length drives latency and cost: a 500-token answer is 500 steps through the model, one after another. It also explains habits teams meet in production: once a token is chosen it stays, so an early wrong turn shapes everything after it, and the model writes fluent text whether or not it is true. Asking for short, structured outputs and checking them is usually cheaper and safer than asking for long free text.
For engineersShow the maths
P(x₁, …, xₙ) = P(x₁) · P(x₂ | x₁) · … · P(xₙ | x₁, …, xₙ₋₁)
The probability of a whole text is the product of each token's probability given everything before it. Generation samples from these conditionals one at a time; training makes the log of this product as large as possible.
Worked example: If the model gives “mat” 0.49, then “.” 0.47, then the end token 0.76, the whole continuation gets 0.49 × 0.47 × 0.76 ≈ 0.18.
P(token i) = e^(zᵢ) / Σⱼ e^(zⱼ)
Softmax: exponentiate each logit z and divide by the sum over all candidates. Only differences between scores matter.
Worked example: “mat” scores 3.2 and “sofa” 2.4. The gap of 0.8 means “mat” is e^0.8 ≈ 2.2 times as likely as “sofa”, whatever the other scores are.
Perplexity = exp( −(1/N) · Σₜ log P(xₜ | x₁, …, xₜ₋₁) )
Average the negative log-probabilities of the N tokens that were actually there, then undo the log. It is the geometric average of 1/P.
Worked example: For 0.49, 0.47 and 0.76 the negative logs are 0.71, 0.76 and 0.27, averaging 0.58. e^0.58 ≈ 1.8: about as unsure as choosing between two options.
Next-token training learns what text is likely, not what is true, so fluent output can still be wrong. Perplexity depends on how the test text is tokenised and prepared, so it compares models fairly only on the same text processed the same way.
The words you will hear
Token
Primary sources
- 01A Neural Probabilistic Language ModelBengio et al. · 2003
- 02Attention Is All You NeedVaswani et al. · 2017
- 03Language Models are Unsupervised Multitask LearnersRadford et al. · 2019
- 04Generation (Transformers documentation)Hugging Face · 2026
Connected ideas
Generating text
Sampling and decoding
How a list of probabilities becomes one word, and the dials that trade steadiness for variety.
ExploreGenerating text
Prefill and decode
Why the first word of an answer takes a moment and the rest stream out at a steady pace.
ExploreGenerating text
KV cache
Why a model keeps the keys and values of earlier tokens instead of recomputing them, and what that memory costs.
ExploreTraining and alignment
Pretraining and scaling laws
Why loss falls predictably with compute, and how to split a budget between model size and data.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.