Training and alignment

Pretraining and scaling laws

Why loss falls predictably with compute, and how to split a budget between model size and data.

Intermediate · 6 steps

Read first:Next-token prediction Loss and gradient descent

Step 1 of 6· Loss curves use the re-fitted Chinchilla formula (Besiroglu et al., 2024); model sizes and token counts are as published; stage 1 probabilities are illustrative

use the steps or arrow keys

Loading visual…

in one minute

Pretraining is the long first phase in which a model reads an enormous amount of text and learns by predicting the next token, again and again. Its error, called loss, falls smoothly and predictably as you add parameters, data and compute: on a chart with logarithmic axes the trend is close to a straight line. That lets labs forecast a large training run from small ones. The Chinchilla study added a second lesson: for a fixed compute budget, grow the model and its training data together, at roughly 20 tokens of text per parameter. Many earlier large models were too big for the amount of text they had read.

Why it matters for your product

Scaling laws turn a very expensive bet into a forecast: a lab can fit curves on small runs and predict what a large run will reach before paying for it. For a team choosing a model, the useful lesson is total cost. A model trained well past the compute-optimal ratio is smaller, so each request is cheaper and faster; for a model that answers many requests, that can outweigh its one-off training bill. It also means bigger models need proportionally more good training text, not only more GPUs.

For engineersShow the maths

C ≈ 6 · N · D

Training compute in floating-point operations is about six times the number of parameters N times the number of training tokens D: two for the forward pass and four for the backward pass.

Worked example: Chinchilla: 6 × 70 billion × 1.4 trillion ≈ 5.9 × 10²³ FLOPs, about the same budget as Gopher's 280 billion parameters on 300 billion tokens.

L(N, D) = E + A / N^α + B / D^β

Loss is a floor set by the text itself, plus a penalty for having too few parameters, plus a penalty for reading too few tokens. Both penalties shrink as power laws.

Worked example: With the re-fitted values E = 1.82, A = 482, α = 0.348, B = 2085, β = 0.366, a 70B model on 1.4T tokens gets 1.82 + 0.08 + 0.08 ≈ 1.97 nats per token.

N_opt ≈ √(C / 120), D_opt ≈ 20 · N_opt

Put D = 20N into C = 6ND and solve: the compute-optimal model size grows with the square root of the budget, and so does the data.

Worked example: A budget of 1.2 × 10²² FLOPs gives √(10²⁰) = 10 billion parameters trained on 200 billion tokens.

where it stops working

Scaling laws predict average next-token loss for the data, architecture and training recipe they were fitted on, not when a specific skill will appear. The 20-to-1 rule ignores the cost of serving the model and assumes there is enough good, fresh text to read without repeating it.

Key terms