Mechanistic interpretability
Opening the model up to find the concepts it uses, and turning them up or down to test them.
Read first:Neural networksResidual stream and normalisation
Step 1 of 6· Illustrative neurons, features and replies; the toy-model ReLU and SAE loss maths is real, and the dictionary sizes, feature IDs and the 10× Golden Gate clamp are from the papers
Mechanistic interpretability tries to reverse-engineer what a trained model computes. Individual neurons turn out to be a poor place to look: in one small language model, a single neuron fires on academic citations, English dialogue, web requests and Korean text. The leading explanation is superposition. A model stores many more concepts than it has neurons by giving each concept its own direction across many neurons, which works because most concepts are rarely active at the same time. Researchers train a sparse autoencoder to unpack those directions into separate features, many of which a person can name. Turning a feature up or down then shows whether the name is right.
Why it matters for your product
Most teams judge a model only by its outputs. Interpretability adds a way to ask why: to look for internal features linked to deception, bias or unsafe code, to monitor them, and to check whether a fix removes a cause or only a symptom. Anthropic found features of these kinds in Claude 3 Sonnet and showed that clamping some of them changed behaviour. The tools are still research-grade, so treat them as a complement to evals and red-teaming, and be wary of anyone claiming full visibility into a model.
For engineersShow the maths
x′ = ReLU(Wᵀ·W·x + b)
The toy model from Toy Models of Superposition: W squeezes many features x into a few dimensions, Wᵀ reads them back out, and a negative bias b with a ReLU filters out the small overlaps between features.
Worked example: Five arrows in a pentagon sit 72° apart, so each leaks cos 72° ≈ 0.31 onto its neighbours. With arrows of squared length 1.4 and b = −0.45, a lone feature reads back 1.4 − 0.45 = 0.95, and a neighbour gets 1.4 × 0.31 − 0.45 ≈ −0.02, which the ReLU turns into exactly 0.
f = ReLU(W_enc·x + b_enc); x̂ = W_dec·f + b_dec; loss = ‖x − x̂‖² + λ·Σᵢ fᵢ·‖W_dec,i‖
The encoder turns an activation x into feature strengths f, the decoder rebuilds it as x̂, and training minimises the rebuild error plus a penalty λ on how much feature activity is used, which pushes most features to zero.
Worked example: In Anthropic's Claude 3 Sonnet run, fewer than 300 of about 34 million features were active on a token on average: under 0.001%.
x′ = x + (c − fᵢ(x)) · dᵢ
Steering: set feature i to a chosen value c by adding the difference along its decoder direction dᵢ, then let the rest of the forward pass run as normal.
Worked example: The Golden Gate Bridge demonstration set c to 10 × the feature's largest activation seen in data.
A sparse autoencoder is an approximation: on Claude 3 Sonnet it explained at least 65% of the variance in the activations, and its authors think they are probably orders of magnitude short of finding every feature. Feature names come from people or models reading examples, so they can be wrong or incomplete until a causal test such as steering backs them up.
The words you will hear
Polysemantic neuron
Primary sources
- 01Toy Models of SuperpositionElhage et al. · 2022
- 02Towards Monosemanticity: Decomposing Language Models With Dictionary LearningBricken et al. · 2023
- 03Sparse Autoencoders Find Highly Interpretable Features in Language ModelsCunningham et al. · 2023
- 04Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 SonnetTempleton et al. · 2024
- 05Scaling and evaluating sparse autoencodersGao et al. · 2024
Connected ideas
Inside the transformer
Transformer architecture
How a stack of identical blocks turns a prompt into a guess for the next word.
ExploreInside the transformer
Feed-forward network
The per-token layer that widens each vector, filters it and writes back what it recalls.
ExploreObservability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreInterpretability and safety
Hallucination
Why models state wrong answers with confidence, and how to trade a few answers for a lot of trust.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.