Neural network basics

Neural networks

Weighted sums and a simple bend, stacked in layers, turn numbers into a prediction.

Basics · 7 steps

Step 1 of 7· Illustrative weights for a toy spam scorer; the sums, ReLU and softmax are computed live

use the steps or arrow keys

Loading visual…

in one minute

A neural network is a stack of very simple calculators called neurons. Each neuron multiplies every input by a weight, adds the results plus a constant called the bias, and passes the total through a simple bend such as ReLU, which keeps positive numbers and turns negative ones into zero. A layer is many neurons reading the same inputs; the next layer reads their outputs, so it can combine simple patterns into richer ones. The last layer produces scores, which softmax turns into probabilities. Nobody writes the weights by hand: training finds them from examples. Without the bends, any stack of layers would collapse into a single straight-line rule.

Why it matters for your product

Neural models, from a small spam filter to a large language model, are built from this pattern at scale: weighted sums and simple bends, repeated, plus extra parts such as attention in transformers. A figure such as “7B” counts these weights and biases, which is why model size drives memory, cost and speed. It also sets expectations. A network is a fitted function, not a list of rules, so you cannot read its behaviour from the code: it has to be measured, with tests and evaluations on realistic inputs.

For engineersShow the maths

h = ReLU(W·x + b)

Multiply the input vector by the layer's weight matrix, add the bias vector, then set every negative entry to zero.

Worked example: Inputs (0.8, 0.6, 0.3), weights (1.2, 0.8, −0.4) and bias −0.3 give 0.96 + 0.48 − 0.12 − 0.3 = 1.02. It is positive, so ReLU passes 1.02 on.

W₂·(W₁·x + b₁) + b₂ = (W₂·W₁)·x + (W₂·b₁ + b₂)

Without an activation between them, two layers are exactly one layer whose weights are the product W₂·W₁, so stacking adds nothing.

Worked example: A layer that doubles its input followed by one that triples it is simply a single layer that multiplies by 6.

pᵢ = exp(zᵢ) / Σⱼ exp(zⱼ)

Softmax: exponentiate every score and divide by the total, so the outputs are positive and add up to one.

Worked example: Scores of 1.16 for spam and −0.54 for not spam differ by 1.7, which softmax turns into about 85% and 15%.

where it stops working

With enough neurons a network can approximate almost any continuous function, but that guarantee says nothing about finding the right weights or behaving well on inputs unlike its training data. A network is only as good as the examples and the objective it was trained on.

Key terms