Neural network basics

Convolution and vision

A small window of weights slides across an image, looking for the same pattern everywhere.

Basics · 6 steps

Step 1 of 6· A toy 8 × 8 image and textbook kernels; convolution, pooling and patch counts are computed live, patch-embedding weights are illustrative

use the steps or arrow keys

Loading visual…

in one minute

To a computer an image is a grid of numbers. A convolutional network looks at it through a small window of weights called a kernel, for example 3 by 3 pixels. At each position it multiplies the pixels under the window by the weights, adds them up and writes one number into a new grid, the feature map. Because the same small kernel is reused at every position, a pattern such as an edge is found wherever it appears. Stacking layers lets later ones combine edges into shapes and objects. Vision transformers take another route: they cut the image into patches and treat each patch as a token.

Why it matters for your product

Reusing one small kernel everywhere means far fewer weights than wiring every pixel to every neuron, plus a built-in assumption that a pattern means the same thing wherever it appears. That makes convolutional networks efficient and effective with modest amounts of data. Vision transformers drop that assumption and learn it from data instead; in the original work they fell short of similar-sized convolutional networks on ImageNet alone and pulled ahead only with much larger pre-training sets. How much data you have is a key input when choosing between them.

For engineersShow the maths

out[i, j] = Σₘ Σₙ K[m, n] · I[i + m, j + n] + b

Each output cell is the sum of the kernel weights times the pixels beneath them, plus a bias. Strictly this is cross-correlation, because the kernel is not flipped; when the kernel is learned, the two are interchangeable.

Worked example: A vertical-edge kernel with columns (1, 0, −1) over a patch whose left column is (1, 1, 1) and right column (0, 0, 0) gives 1 + 1 + 1 = 3, its strongest response.

o = ⌊(i + 2p − k) / s⌋ + 1

The output size from the input size i, kernel size k, padding p and stride s.

Worked example: An 8-pixel image, a 3-pixel kernel, no padding and stride 1: (8 − 3) / 1 + 1 = 6.

N = H·W / P²

A vision transformer cuts an H × W image into N square patches of P × P pixels, one token each.

Worked example: A 224 × 224 image in 16 × 16 patches: 50,176 ÷ 256 = 196 tokens.

where it stops working

A convolution sees only a small neighbourhood, so relating distant parts of an image takes many layers, and its built-in assumptions help less once data is plentiful. Vision transformers remove those assumptions but need more data or pre-training to learn them, and attention's cost grows with the square of the number of patches.

Key terms