Inside the transformer

Self-attention

How every word looks at every other word to work out what it means here.

Intermediate · 6 steps

Read first:Embeddings Positional encoding

Step 1 of 6· Illustrative attention weights for one sentence; the softmax and weighted-sum maths is real

use the steps or arrow keys

Loading visual…

in one minute

Self-attention lets each token in a sentence look back at the others and decide which ones matter for understanding it. For every token the model makes three vectors: a query (what am I looking for?), a key (what do I contain?) and a value (what do I pass on?). Comparing a token's query with every key gives scores, softmax turns the scores into weights that add up to one, and the token's new representation is the weighted blend of the values. That is how “it” in “the cat sat because it was tired” ends up carrying the meaning of “cat”.

Why it matters for your product

Attention is why models can follow a reference across pages, use the passages a retrieval system hands them and keep long conversations coherent. Its cost grows with the square of the context length, which is why long prompts are slower and more expensive; techniques such as FlashAttention, KV caching and grouped-query attention exist to cut that compute and memory. It is also why we keep retrieved context tight in the systems we build: more text is not free.

For engineersShow the maths

Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V

Score every query against every key, divide by the square root of the key size, turn each row into weights with softmax, and use the weights to average the values.

Worked example: With dₖ = 64, raw scores are divided by 8. Scores of 4.0 and 2.0 become 0.5 and 0.25, so softmax gives weights of about 56% and 44% instead of 88% and 12%.

MultiHead(X) = Concat(head₁, …, headₕ) · Wᴼ, headᵢ = Attention(X·Wᵢᑫ, X·Wᵢᴷ, X·Wᵢⱽ)

Run h smaller attentions in parallel, each with its own projections, then join their outputs and mix them with one more matrix.

where it stops working

Full attention compares every token with every other, so compute and memory grow with the square of the context length. Attention weights show where a head looks, not a faithful explanation of why the model answered as it did.

Key terms