Neural network basics

Loss and gradient descent

Measure how wrong the model is, then step downhill, a little at a time.

Basics · 6 steps

Step 1 of 6· A one-weight toy loss, L = (w − 3)²; every step and number is computed exactly

use the steps or arrow keys

Loading visual…

in one minute

Training needs one number that says how wrong the model is: the loss. Here the model makes a single guess, the correct answer is 3, and the loss is the squared miss. Plot the loss for every possible setting of the weight and you get a bowl. The gradient is the slope of the bowl where you stand. It points uphill, so gradient descent steps the other way. The learning rate sets the size of each step. Too small and training crawls; well chosen and it settles at the bottom; too large and every step overshoots, and it can climb right out of the bowl.

Why it matters for your product

The learning rate is one of the most sensitive settings in training. Too high and a run that costs days of GPU time can spike and waste the budget; too low and it burns compute while barely improving. That is why teams watch loss curves as runs progress, start with small warm-up steps and stop runs that go wrong early. The same applies when fine-tuning a model on your own data: choose the learning rate carefully and track the loss on held-out examples, not only on the training set.

For engineersShow the maths

w ← w − η · ∂L/∂w

The new weight is the old weight minus the learning rate η times the slope of the loss.

Worked example: For L = (w − 3)² the slope is 2(w − 3). From w = 0 with η = 0.2 the slope is −6, so w moves to 0 + 0.2 × 6 = 1.2 and the loss falls from 9 to 3.24.

(w − 3) ← (1 − 2η) · (w − 3)

On this bowl every step multiplies the distance to the bottom by (1 − 2η). The distance shrinks only while that factor is between −1 and 1, which means 0 < η < 1.

Worked example: η = 0.2 gives a factor of 0.6; η = 0.85 gives −0.7, a zig-zag that still shrinks; η = 1.1 gives −1.2, so the distance grows by 20% a step.

w ← w − (η / |B|) · Σᵢ∈B ∇Lᵢ(w)

In practice the gradient is averaged over a random mini-batch B of examples instead of the whole dataset, which makes each step cheap but noisy.

where it stops working

Real losses are not neat bowls: they have flat stretches, narrow valleys and many dips, and mini-batch gradients are noisy. Gradient descent only ever uses the local slope, so it finds a good low point, not a guaranteed best one.

Key terms