Inside the transformer

Feed-forward network

The per-token layer that widens each vector, filters it and writes back what it recalls.

Intermediate · 7 steps

Read first:Neural networks Residual stream and normalisation

Step 1 of 7· Illustrative weights and memory units, patterned on Geva et al.; the ReLU, Swish and parameter-count maths is real

use the steps or arrow keys

Loading visual…

in one minute

After attention has let tokens share information, every token passes through a small network of its own, the feed-forward network. It stretches the token's vector to about four times its width, lets an activation function switch some of those numbers on and silence others, then squeezes the result back and adds it to the running total. The same network is applied to every token separately. Models such as LLaMA use a gated version, SwiGLU, in which one branch decides how much of another gets through. Research suggests these layers act like a memory: parts respond to patterns in the text and push the prediction towards words that usually follow.

Why it matters for your product

Because feed-forward layers hold about two-thirds of a transformer layer's weights, they dominate model size, memory and much of the compute per token. That is why mixture-of-experts designs such as the Switch Transformer replace them, swapping one big feed-forward network for many and running only one per token. The memory view also explains why model-editing methods such as ROME change feed-forward weights when they rewrite a specific fact.

For engineersShow the maths

FFN(x) = max(0, x·W₁ + b₁)·W₂ + b₂

Widen with W₁, set negative hidden values to zero with ReLU, then narrow back with W₂.

Worked example: A hidden unit whose input is −0.7 outputs 0; one at 1.3 outputs 1.3. Only the positive units contribute anything to the result.

FFN_SwiGLU(x) = (Swish(x·W) ⊙ x·V)·W₂, Swish(z) = z·σ(z)

Two widening projections: one goes through Swish and gates the other element by element, then W₂ projects back down.

Worked example: A gate input of z = 2 gives Swish(2) = 2 × 0.88 = 1.76; with a value of 0.5 the unit outputs 0.88. At z = −2 the gate gives only −0.24: nearly shut.

Weights per layer: attention 4·d², FFN 2·d·4d = 8·d²

Query, key, value and output projections are each d × d. The FFN has two d × 4d matrices; SwiGLU uses three of d × (8d/3), the same total.

Worked example: At d = 4,096, attention holds 67.1 million weights per layer and the feed-forward network 134.2 million: two-thirds of the layer.

where it stops working

The key-value picture is an interpretation, not a wiring diagram: Geva et al. found each layer's output combines hundreds of active memories, so a fact rarely lives in one unit. The layer also sees one token at a time; moving information between tokens is attention's job.

Key terms