Backpropagation
Working backwards from the error to find how much each weight is to blame.
Read first:Neural networksLoss and gradient descent
Step 1 of 7· A one-neuron toy network; every forward value and gradient is computed live with the chain rule
To improve, a network needs to know how each weight affects the loss. Backpropagation answers that for every weight in one backward sweep. The forward pass computes the prediction and remembers every intermediate value. The backward pass starts at the loss and walks back through the same calculation. Each step knows its own local slope, and the chain rule says the overall slope is the product of the local slopes along the path. The result is the gradient for every weight at a cost of only a few forward passes, typically two to three, which is what makes training networks with billions of weights practical.
Why it matters for your product
Backpropagation is what makes training affordable: the gradient for every weight comes from one backward pass, not one experiment per weight. It also explains practical limits. The backward pass needs values saved during the forward pass, so training uses far more memory than running a model, which is one reason fine-tuning needs bigger GPUs than inference. And when gradients shrink or explode across many layers, training stalls or blows up, so teams track gradient sizes alongside the loss.
For engineersShow the maths
∂L/∂w = ∂L/∂a · ∂a/∂z · ∂z/∂w
The loss's sensitivity to w is the product of the local slopes along the path from w to the loss.
Worked example: (−0.755) × 0.235 × 2 = −0.355, so raising w by 0.01 should lower the loss by about 0.0035.
σ′(z) = σ(z) · (1 − σ(z)) ≤ 0.25
The sigmoid's slope is largest at z = 0, where it equals 0.25, and smaller everywhere else.
Worked example: Back through ten sigmoids with weights of 1, the signal is multiplied by at most 0.25¹⁰ ≈ 0.00000095: under one millionth.
Backpropagation gives exact gradients for the current batch, but says nothing about where the lowest loss is; choosing the step is the optimiser's job. It must keep the forward pass's intermediate values, and through very deep or recurrent chains its gradients can vanish or explode.
The words you will hear
Forward pass
Primary sources
- 01Learning representations by back-propagating errorsRumelhart et al. · 1986
- 02Automatic differentiation in machine learning: a surveyBaydin et al. · 2015
- 03Understanding the difficulty of training deep feedforward neural networksGlorot, Bengio · 2010
- 04Deep Residual Learning for Image RecognitionHe et al. · 2015
Connected ideas
Neural network basics
Optimisers: SGD, momentum and Adam
The same slope, a different stride: how training turns a gradient into a step.
ExploreInside the transformer
Residual stream and normalisation
Each layer adds to a running total instead of replacing it, and a norm keeps what it reads in range.
ExploreTraining and alignment
Fine-tuning and LoRA
Teaching a pretrained model a new job by training a small add-on instead of every weight.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.