Batching and PagedAttention
Keeping a GPU busy with many conversations at once, and storing their memory in pages.
Read first:KV cachePrefill and decode
Step 1 of 6· Illustrative requests and a toy 64-slot memory; the OPT-13B, A100, Orca and vLLM figures are published
Writing text one token at a time leaves much of a GPU's arithmetic idle, so servers handle many conversations at once. The simple approach starts a batch together and waits until its longest answer is finished, leaving slots empty. Continuous batching instead lets a finished request leave, and a waiting one join, after every single token. The other limit is memory: each conversation keeps a growing cache of past keys and values. PagedAttention stores that cache in small fixed-size pages, the way an operating system manages memory, so almost nothing is wasted and more conversations fit on the same GPU.
Why it matters for your product
For a team running its own model, these two ideas largely decide how many users one GPU can serve, and so the cost of each request. Modern serving engines build them in, so the practical work is setting batch and memory limits and measuring throughput at the latency your users will accept, as the papers do, rather than quoting peak tokens per second.
For engineersShow the maths
utilisation = busy slot-steps ÷ (slots × total steps)
Count how many slot-steps did real work and divide by all the slot-steps the GPU spent.
Worked example: The eight requests need 40 slot-steps. Static batching takes 17 steps on 4 slots: 40 ÷ 68 = 59%. Continuous batching takes 12: 40 ÷ 48 = 83%.
KV bytes per token = 2 × hidden size × layers × bytes per number
Every layer stores a key and a value vector for every token, each as long as the hidden state.
Worked example: OPT-13B: 2 × 5,120 × 40 × 2 bytes = 819,200 bytes, about 800 KB per token, so a 2,048-token request needs about 1.6 GB.
paged waste per request < block size
Blocks fill in order and a new one is allocated only when the last is full, so at most one block per request is partly empty.
Worked example: With blocks of 16, a 100-token request uses 7 blocks (112 slots) and wastes 12. Reserving 2,048 contiguous slots would leave 1,948 empty.
Batching only helps when many requests are in flight, and a fuller batch makes each step do more work. Paging removes wasted memory but not the cache itself: long contexts still need a lot of memory, which caps how many requests fit at once.
The words you will hear
Static batching
Primary sources
Connected ideas
Generating text
KV cache
Why a model keeps the keys and values of earlier tokens instead of recomputing them, and what that memory costs.
ExploreGenerating text
Prefill and decode
Why the first word of an answer takes a moment and the rest stream out at a steady pace.
ExploreSpeed and cost
FlashAttention
The same attention maths, done in tiles so the big score table never touches slow memory.
ExploreGenerating text
Speculative decoding
A small model guesses a few tokens ahead and the big model checks them all in one pass: faster, with the same output distribution.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.