Prefill and decode
Why the first word of an answer takes a moment and the rest stream out at a steady pace.
Read first:Next-token predictionKV cache
Step 1 of 6· Idealised timings for 7B parameters at 16-bit on A100 datasheet peaks, with an illustrative 128 KiB cache per token; real systems are slower
When you send a prompt, the model works in two phases. First comes prefill: it reads the whole prompt in a single pass, processing every token in parallel and storing their keys and values. That sets the time to the first token. Then comes decode: it writes the answer one token at a time, and every token needs another full pass over the model's weights. Prefill is usually limited by how fast the chip can do arithmetic; decode, by how fast it can move weights out of memory. That difference explains long-prompt waits, streaming speed and why servers batch many users together.
Why it matters for your product
The two phases map onto what users feel and what you pay for. Time to first token decides whether an app feels responsive, and it grows with prompt length, so bloated prompts and oversized retrieved context cost latency. Time per output token sets streaming speed and is limited by memory bandwidth, so smaller or quantised weights help more than faster arithmetic. Serving teams batch decodes, split long prefills into chunks and sometimes run the two phases on separate GPUs to hit both targets.
For engineersShow the maths
Latency ≈ TTFT + (n − 1) × TPOT
The wait for the first token, plus one time-per-token for each of the remaining n − 1 tokens.
Worked example: With TTFT = 45 ms and TPOT = 7 ms, a 200-token answer takes 45 + 199 × 7 ≈ 1,438 ms: about 1.4 seconds, 97% of it spent decoding.
Ridge point = peak FLOP/s ÷ memory bandwidth
A pass is compute-bound only if it does more operations per byte moved than this. A 16-bit model does 2 operations per 2-byte weight for each token in the pass, so a pass of T tokens does about T operations per byte.
Worked example: A100 80GB SXM: 312 TFLOP/s ÷ 2.039 TB/s ≈ 153. Prefill of 1,000 tokens runs at about 1,000 (compute-bound); decode at batch 1 runs at about 1 (memory-bound).
Decode step time ≈ (weight bytes + cache bytes read) ÷ memory bandwidth
When decode is memory-bound, each step costs the time to stream the weights, plus every active user's cache, from memory once.
Worked example: 14 GB of weights ÷ 2.039 TB/s ≈ 6.9 ms per token at batch 1, about 145 tokens per second at best.
These are idealised numbers: real kernels rarely reach datasheet peaks, attention adds work that grows with context, and under load queueing can dominate the wait. The split between compute and memory limits shifts with batch size and prompt length, so measure your own TTFT and TPOT rather than trusting a formula.
The words you will hear
Prefill
Primary sources
- 01Efficiently Scaling Transformer InferencePope et al. · 2022
- 02Orca: A Distributed Serving System for Transformer-Based Generative ModelsYu et al. · 2022
- 03SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked PrefillsAgrawal et al. · 2023
- 04DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model ServingZhong et al. · 2024
- 05Roofline: An Insightful Visual Performance Model for Multicore ArchitecturesWilliams et al. · 2009
- 06NVIDIA A100 Tensor Core GPU DatasheetNVIDIA · 2021
Connected ideas
Speed and cost
Batching and PagedAttention
Keeping a GPU busy with many conversations at once, and storing their memory in pages.
ExploreGenerating text
Speculative decoding
A small model guesses a few tokens ahead and the big model checks them all in one pass: faster, with the same output distribution.
ExploreSpeed and cost
FlashAttention
The same attention maths, done in tiles so the big score table never touches slow memory.
ExploreSpeed and cost
Quantisation
Storing a model's numbers with fewer bits, and keeping the answers nearly the same.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.