KV cache
Why a model keeps the keys and values of earlier tokens instead of recomputing them, and what that memory costs.
Read first:Self-attentionMulti-head attention
Step 1 of 6· Model shapes from the LLaMA and Mistral 7B papers and the byte maths are real; the MQA preset is a hypothetical variant
A language model writes one token at a time, and each new token looks back at every earlier token through attention. For that it needs each earlier token's key and value vectors, in every layer. Those never change once computed, so instead of recomputing them for the whole text at every step, the model stores them in a KV cache and adds one new entry per token. Repeated work becomes memory. The price is size: the cache grows with every token and with every conversation served at once, and with long contexts it takes a large share of GPU memory.
Why it matters for your product
The cache is what keeps token-by-token generation fast, and why long contexts cost more to serve. Every conversation in flight carries its own cache: in the PagedAttention paper's example, a 13B model on a 40 GB GPU uses about 65% of memory for weights and close to 30% for the cache. When choosing or sizing a model, check its KV-head count and your real context lengths, not just its parameter count. Grouped-query attention, sliding-window caches and paged allocation are among the main levers.
For engineersShow the maths
KV bytes = 2 × layers × KV heads × head size × bytes per number × tokens × sequences
Two tensors (keys and values), for every layer, every KV head and every position, at the chosen precision, for every token held in every conversation in flight.
Worked example: LLaMA-7B at 16-bit: 2 × 32 × 32 × 128 × 2 = 524,288 bytes per token, so a 2,048-token conversation holds 1 GiB. Mistral 7B, with 8 KV heads, holds 256 MiB.
Token passes over n steps: without a cache 1 + 2 + … + n = n(n + 1) / 2; with a cache n
Without a cache, step t reprocesses all t tokens written so far; with one, it processes only the new token and reads the rest from memory.
Worked example: For 1,000 generated tokens: 500,500 token passes without a cache and 1,000 with one, about 500 times fewer.
The cache saves recomputation, not reading: every new token still reads all cached keys and values, so each step slows as the context grows, and the cache is only valid while the earlier text stays unchanged. Sharing key-value heads trades a little quality for memory, which is why GQA sits between multi-head and multi-query attention.
The words you will hear
KV cache
Primary sources
- 01Efficiently Scaling Transformer InferencePope et al. · 2022
- 02Fast Transformer Decoding: One Write-Head is All You NeedShazeer · 2019
- 03GQA: Training Generalized Multi-Query Transformer Models from Multi-Head CheckpointsAinslie et al. · 2023
- 04Efficient Memory Management for Large Language Model Serving with PagedAttentionKwon et al. · 2023
- 05LLaMA: Open and Efficient Foundation Language ModelsTouvron et al. · 2023
- 06Mistral 7BJiang et al. · 2023
Connected ideas
Generating text
Prefill and decode
Why the first word of an answer takes a moment and the rest stream out at a steady pace.
ExploreSpeed and cost
Batching and PagedAttention
Keeping a GPU busy with many conversations at once, and storing their memory in pages.
ExploreSpeed and cost
FlashAttention
The same attention maths, done in tiles so the big score table never touches slow memory.
ExploreSpeed and cost
Quantisation
Storing a model's numbers with fewer bits, and keeping the answers nearly the same.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.