Transformer architecture
How a stack of identical blocks turns a prompt into a guess for the next word.
Read first:Neural networksTokenisationEmbeddings
Step 1 of 6· Illustrative vectors, weights and guesses; GPT-2 sizes and the parameter maths are real
A GPT-style language model is a tall stack of identical blocks. Your text is split into tokens, and each token becomes a list of numbers, a vector, with a signal for its position added. Each vector then travels down its own lane, called the residual stream. Every block reads the lanes and adds a small update: first attention, where each token gathers information from the tokens before it, then a feed-forward step that works on each token alone. After dozens of blocks, the last token's vector is turned into a score for every word in the vocabulary, and the highest-scoring words are the model's guesses for what comes next.
Why it matters for your product
Almost every modern language model is this same stack, varied in width, depth and details, so the parts tell you where cost comes from. Weights grow with layers × width², so doubling the width roughly quadruples the weights in the blocks. Every generated token runs the whole stack once, which is why output length drives latency, and attention grows with context length. When we size a system, these three numbers are where we start.
For engineersShow the maths
x ← x + Attention(Norm(x)); x ← x + FFN(Norm(x))
Each block reads a normalised copy of the lane and adds its result back. The lane itself is never replaced; this is the pre-norm arrangement GPT-2 used.
Worked example: If one number in a lane is 0.80 and attention's update is −0.10, the lane holds 0.70, and the feed-forward step starts from 0.70, not from scratch.
weights in the blocks ≈ 12 × L × d²
Attention holds about 4d² weights per block (query, key, value and output matrices) and the feed-forward network about 8d² (d → 4d → d). Multiply by the number of layers L; embeddings come on top.
Worked example: GPT-2's largest model: 12 × 48 × 1,600² ≈ 1.47 billion, plus about 0.08 billion in embeddings, close to the 1.54 billion the paper reports.
p(next token) = softmax(Norm(x_last) · Wᵤ)
The last lane, normalised, is multiplied by the unembedding matrix to give one logit per vocabulary token; softmax turns the logits into probabilities.
Worked example: In GPT-2's largest model, a 1,600-number vector times a 1,600 × 50,257 matrix gives 50,257 logits, one per token.
This is the decoder-only design behind GPT-style models; encoder-only and encoder–decoder models differ mainly in the attention mask and an extra cross-attention step. The labels say what each part is built to do, but what individual layers actually learn is still an open research question.
The words you will hear
Embedding table
Primary sources
- 01Attention Is All You NeedVaswani et al. · 2017
- 02Deep Residual Learning for Image RecognitionHe et al. · 2015
- 03Language Models are Unsupervised Multitask LearnersRadford et al. · 2019
- 04On Layer Normalization in the Transformer ArchitectureXiong et al. · 2020
- 05Scaling Laws for Neural Language ModelsKaplan et al. · 2020
- 06A Mathematical Framework for Transformer CircuitsElhage et al. · 2021
Connected ideas
Inside the transformer
Self-attention
How every word looks at every other word to work out what it means here.
ExploreInside the transformer
Residual stream and normalisation
Each layer adds to a running total instead of replacing it, and a norm keeps what it reads in range.
ExploreInside the transformer
Feed-forward network
The per-token layer that widens each vector, filters it and writes back what it recalls.
ExploreGenerating text
Next-token prediction
A language model writes by guessing one token at a time, then feeding each guess back in.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.