Inside the transformer

Residual stream and normalisation

Each layer adds to a running total instead of replacing it, and a norm keeps what it reads in range.

Intermediate · 6 steps

Step 1 of 6· Illustrative vectors; the LayerNorm and RMSNorm maths and the depth bounds are real

use the steps or arrow keys

Loading visual…

in one minute

Inside a transformer, each token carries one list of numbers from the bottom of the model to the top. Every layer reads that list, works out a small change and adds it on, instead of replacing what was there. This running total is called the residual stream, and because the original signal always passes straight through, very deep models stay trainable. The total tends to grow as layers add to it, so before each layer reads it, a normalisation step rescales a copy to a steady size. LayerNorm centres and scales the numbers; RMSNorm, used in models such as LLaMA, only scales them, which is cheaper.

Why it matters for your product

These quiet parts decide whether a deep model trains at all. Residual connections make depth usable, pre-norm placement lets training skip a fragile learning-rate warm-up, and RMSNorm saves a little compute on every token. For teams, the residual stream is also the best mental model when reading interpretability work or judging a fine-tune: each layer is a small edit to one shared vector, so a change in one layer can surface in any layer above it.

For engineersShow the maths

x ← x + Attention(Norm(x)); x ← x + FFN(Norm(x))

A pre-norm block: each sub-layer reads a normalised copy of the stream and adds its result back. The stream itself is never overwritten.

Worked example: If x = [1.0, 2.0] and a layer's update is [0.25, −0.5], the stream becomes [1.25, 1.5]. Nothing was replaced; the update simply sits on top.

LayerNorm(x) = γ ⊙ (x − μ) / √(σ² + ε) + β

Subtract the mean μ of the features, divide by their standard deviation σ (the tiny ε avoids dividing by zero), then apply the learned gain γ and bias β.

Worked example: x = [2, 4, 6]: μ = 4 and σ = 1.63, so with γ = 1 and β = 0 the output is [−1.22, 0, 1.22]. Add 10 to every input and the output is still [−1.22, 0, 1.22].

RMSNorm(x) = γ ⊙ x / √(mean(x²) + ε)

Divide by the root mean square of the features and apply the gain. There is no mean to subtract and no bias.

Worked example: x = [2, 4, 6]: mean(x²) = 18.67, so the RMS is 4.32 and the output is [0.46, 0.93, 1.39]. The middle value lands at 0.93, not 0, because nothing was centred.

where it stops working

Normalisation controls scale, not meaning: it cannot rescue a layer that has learned something unhelpful. In pre-norm models the stream keeps growing with depth, so each later layer's update is a smaller share of the total, and every layer competes for the same fixed number of dimensions.

Key terms