Inside the transformer

Positional encoding

How a model that reads every word at once still knows which came first.

Intermediate · 6 steps

Step 1 of 6· Illustrative word vectors and angles; the attention, sinusoid, rotation and dot-product maths is real

use the steps or arrow keys

Loading visual…

in one minute

Attention looks at all the words at once, so on its own it cannot tell “dog bites man” from “man bites dog”. Positional encoding fixes this by giving each token a signal for where it sits. The original Transformer added a pattern of waves to each word's vector, a kind of barcode for its position. Many newer models, including LLaMA, use rotary embeddings (RoPE) instead: they turn parts of each query and key by an angle that grows with position, so when two tokens are compared, what counts is how far apart they are. This choice also shapes how far past its training length a model can read.

Why it matters for your product

Position handling largely decides how long a context a model can use. A model trained on 2,048 tokens does not simply work on 8,000: its position signals leave the range it learned, and attention can break down badly. Methods such as position interpolation and ALiBi exist to stretch that limit, with brief fine-tuning or none. For long documents, test a model at the lengths you actually plan to use rather than relying on the maximum it accepts.

For engineersShow the maths

PE(pos, 2i) = sin(pos / 10000^(2i/d)), PE(pos, 2i+1) = cos(pos / 10000^(2i/d))

Even dimensions carry a sine and odd ones a cosine. Each pair has its own wavelength, growing geometrically from 2π positions for the first pair to 10,000 × 2π for the last.

Worked example: The first pair turns one radian per position: at position 1 it reads (sin 1, cos 1) ≈ (0.84, 0.54), at position 2 ≈ (0.91, −0.42). It repeats every 2π ≈ 6.3 positions.

RoPE: rotate each pair (x₁, x₂) of a query or key by m·θᵢ, θᵢ = 10000^(−2(i−1)/d)

A token at position m has pair i of its query and key turned by m times that pair's speed θᵢ. Early pairs turn fast; late pairs turn very slowly.

q_m · k_n = |q| |k| cos(φ + (m − n)·θ)

For one rotated pair, where φ is the angle between the query and key before rotation, the score depends on the positions only through the gap m − n. The full score adds up one such term per pair.

Worked example: With θ = 30°, a query at 5 and a key at 3 are turned 60° further apart than the unrotated pair, exactly as at positions 105 and 103, so the scores are identical.

where it stops working

Models still degrade on inputs much longer than they were trained on, and learned absolute positions cannot go past the end of their table at all. One controlled study of small decoder-only models found that RoPE, ALiBi and learned absolute positions all generalised poorly to longer inputs on reasoning tasks.

Key terms