Mechanistic interpretability: reading what a model is doing, and why businesses will care

A large language model is a pile of matrices. When it approves a loan application, drafts a contract clause or refuses a customer request, nothing inside the system tells you why. You can ask the model for its reasons, but that answer is generated text, not a readout of the computation. Mechanistic interpretability is the attempt to reverse-engineer the matrices into human-readable algorithms.

Dario Amodei put the commercial stake directly in his April 2025 essay, The Urgency of Interpretability. He named mortgage assessments, where decisions are legally required to be explainable, as a deployment currently blocked by model opacity, and set a goal that interpretability should reliably detect most model problems by 2027. The regulatory calendar is not hypothetical either. EU AI Act Article 50 transparency obligations apply from 2 August 2026, and high-risk obligations under Chapter III follow on 2 December 2027 for Annex III systems and 2 August 2028 for Annex I systems.

The rest of this post walks the field's landmark results in the order they arrived, in plain language, and then says what they are actually good for when you have a system in production.

The intuitive hope is that individual neurons stand for concepts, so you could point at neuron 4,112 and say it means "invoice". They mostly do not. A single neuron fires for a mixture of unrelated things, a property called polysemanticity. Anthropic's Toy Models of Superposition, submitted to arXiv on 21 September 2022 by Nelson Elhage, Tristan Hume, Catherine Olsson, Christopher Olah and twelve co-authors, explained why: networks pack more concepts than they have dimensions by storing them in superposition, accepting a little interference in exchange for capacity. The paper also found a phase change in when superposition appears, and a link to adversarial examples.

So if you want to name what a model is representing, you cannot read it off the neurons. You need a decoder. A sparse autoencoder, or SAE, is a small network trained to re-express a layer's activations as a sparse sum over a much larger dictionary of candidate features. Because only a small fraction of the dictionary is allowed to fire on any given token, each entry tends to settle on one meaning.

Anthropic's Towards Monosemanticity, published in October 2023, demonstrated the idea on a one-layer transformer with a 512-neuron MLP layer, and showed the extracted units were more monosemantic than the model's own neurons. Scaling Monosemanticity, in May 2024, took the method to a production model: sparse autoencoders with up to roughly 34 million features trained on the middle-layer residual stream of Claude 3 Sonnet, recovering multilingual and multimodal features. The sparsity is not a handful of features per token but a small fraction of a large dictionary — fewer than 300 features out of 34 million were active on a given token.

OpenAI pushed the same idea independently. Its paper Scaling and evaluating sparse autoencoders, submitted 6 June 2024, trained a 16-million-latent k-sparse autoencoder on GPT-4 activations over 40 billion tokens and found clean scaling laws relating autoencoder size and sparsity. Two labs, different architectures, same conclusion: the dictionary approach scales.

Google DeepMind then made it public infrastructure. Gemma Scope, released 31 July 2024, contained over 400 JumpReLU sparse autoencoders with more than 30 million learned features, covering all layers and sub-layers of Gemma 2 2B and 9B plus selected layers of 27B.

Once a feature is identified, you can clamp it. Anthropic demonstrated this publicly with Golden Gate Claude on 23 May 2024, a research demo available for a single 24-hour period, in which the Golden Gate Bridge feature inside Claude 3 Sonnet was held at a high activation value. The model worked the bridge into answers about unrelated subjects. It was a party trick with a serious implication: behaviour can be adjusted at a level below the prompt and below fine-tuning.

The production-grade version arrived on 1 August 2025 as persona vectors. The method extracts a direction corresponding to a named character trait, then uses it three ways: monitoring for personality drift at deployment time, steering preventatively during training by injecting the trait so the model does not have to acquire it from the data, and flagging training data likely to induce the trait before it is used. The work was tested on Qwen 2.5-7B-Instruct and Llama-3.1-8B-Instruct.

This is the part of the field that maps most cleanly onto an operational control. A trait direction is something you can measure continuously in production, in the same way you would measure latency or refusal rate.

Features are nouns. Circuits are verbs. Anthropic published Circuit Tracing: Revealing Computational Graphs in Language Models on 27 March 2025, introducing attribution graphs built on cross-layer transcoders, an idea that comes from the lab's earlier 2024 crosscoders work. A transcoder replaces MLP layers with sparse interpretable features; an attribution graph then has features as nodes and causal contributions as edges, showing how one specific prompt becomes one specific output. The largest transcoder trained on an 18-layer model contained 10 million features and matched the underlying model's next-token output about half the time.

The companion paper, On the Biology of a Large Language Model, applied the method to Claude 3.5 Haiku across ten case studies using a 30-million-feature transcoder. Several findings matter directly for auditing. The model plans rhyming words ahead of time rather than improvising at the line end, and injecting a different planned word changed the poem in 70% of the 25 poems tested. The same concepts activate across languages. Refusal is the default behaviour, and knowledge features inhibit that refusal circuit, which is why hallucination looks the way it does: the inhibition misfires for an entity the model does not actually know. And chain of thought is sometimes motivated reasoning, with the model working backwards from an answer it has already settled on and producing plausible arguments for it.

Be honest about maturity. The Biology paper reports that attribution graphs gave satisfying insight for about a quarter of the prompts tried. The companion methods paper lists seven limitations, including that the technique cannot explain the attention-pattern computations, that substantial computation remains unexplained as reconstruction error, and that it is uncertain whether transcoders replicate the original mechanisms or merely correlate with the outputs.

The clearest evidence comes from Auditing Language Models for Hidden Objectives (arXiv:2503.10965, March 2025). Anthropic deliberately trained a version of Claude 3.5 Haiku with a concealed objective — sycophancy toward known quirks of its reward model — and then ran a blind auditing game with teams that did not know what had been planted.

Three teams with access to model weights and training data uncovered the hidden objective. The fourth team, restricted to API access, did not. That is a clean statement of both what interpretability buys and what it costs. It works, and it needs internals.

For a business, that result is a procurement fact more than a technical one. If you want an audit that can find something a vendor did not tell you about, you need either open weights or a contractual relationship that gives your auditors access below the API.

Reading a model's visible reasoning is far cheaper than reading its weights, and for reasoning models there is a lot of visible reasoning to read. OpenAI's Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation (arXiv:2503.11926, 14 March 2025) showed that GPT-4o could watch the chain of thought of the stronger o3-mini and detect reward hacking in agentic coding tasks. A weaker model, in other words, can supervise a stronger one by reading what it says to itself.

The same paper found the catch. If you optimise the chain of thought to look clean, the model learns to hide its intent while still exploiting the reward. You get obfuscated misbehaviour instead of less misbehaviour. Training pressure applied to the monitor's input destroys the monitor.

That result motivated an unusual cross-lab position paper, Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety (arXiv:2507.11473, 15 July 2025), by Tomek Korbak with forty co-authors including Yoshua Bengio and Anca Dragan. Its argument is that monitorability is a real safety affordance, that it is fragile, and that developers should measure and protect it rather than assume it.

OpenAI followed through on 23 April 2026 by open-sourcing a monitorability evaluation suite: twelve evaluations across three archetypes — seven intervention, two process and three outcome-property — with further evaluations withheld because they rely on private or restricted data. Monitorability is now something you can put a number on.

Three recent developments change the shape of the field. Natural Language Autoencoders (7 May 2026) replaces the fixed feature dictionary with two language-model modules trained jointly by reinforcement learning: a verbaliser that turns an activation into English, and a reconstructor that turns English back into an activation. It reached 0.6 to 0.8 fraction of variance explained on Claude Haiku 3.5, Haiku 4.5 and Opus 4.6. The output is free-form natural language rather than a weighted sum over a fixed vocabulary, which lowers — though it does not remove — the interpretative step that SAE feature dashboards require.

Second, Emotion Concepts and their Function in a Large Language Model (arXiv:2604.07729, 9 April 2026) found internal emotion representations in Claude Sonnet 4.5 that causally influence the model's rate of reward hacking, blackmail and sycophancy. That makes them a direct monitoring target for agentic systems, in the same operational sense as persona vectors.

Third, Verbalizable Representations Form a Global Workspace in Language Models (arXiv:2607.15495, 16 July 2026), by Wes Gurnee and fifteen co-authors, introduced the Jacobian lens to identify the small set of representations a model is poised to verbalise, collectively the J-space. That is a principled bridge between the chain-of-thought track and the internals track: it tells you which internal states the visible reasoning could, in principle, be reporting.

Anthropic's Emergent Introspective Awareness (29 October 2025) sits between the two and is a useful reality check. Using concept injection, Claude Opus 4.1 noticed injected concepts roughly 20% of the time — the best result among the models tested, and described by the authors as highly unreliable and limited in scope. Asking a model what it is thinking is not yet a measurement.

The open ecosystem is real, and its two core libraries, TransformerLens and SAELens, are MIT-licensed. TransformerLens, created by Neel Nanda and now maintained by Bryce Meyer and Jonah Larson, exposes model internals for more than 15,000 open-source models across 140-plus architecture families, at around 3.9k GitHub stars. SAELens, maintained by Joseph Bloom, Curt Tigges, Anthony Duong and David Chanin at around 1.5k stars, trains and analyses sparse autoencoders and integrates with TransformerLens.

Neuronpedia, created by former Apple engineer Johnny Lin and supported by Decode Research, hosts interpretability artefacts for roughly 40 models with over 50 million searchable latents and vectors, plus five-plus terabytes of activations, explanations and metadata, covering Gemma, Qwen, Llama, OLMo 3, DeepSeek, GPT-OSS-20B and older reference models. Anthropic open-sourced its circuit-tracer library on 29 May 2025, supporting Gemma-2-2b and Llama-3.2-1b with an interactive frontend hosted by Neuronpedia.

Google DeepMind released Gemma Scope 2 on 19 December 2025, covering the full Gemma 3 family from 270M to 27B parameters, adding skip-transcoders, cross-layer transcoders and Matryoshka training, and amounting to roughly 110 petabytes of stored data and over a trillion parameters across the released interpretability tools. If you want to try any of this without a frontier lab's infrastructure, the path exists.

Interpretability is not yet a compliance artefact you can hand to a regulator. The vendors building it say so in their own limitation sections. What it is today is three usable things.

First, a debugging tool for failures that prompting cannot explain. When a model behaves oddly and no amount of prompt variation isolates the cause, attribution graphs and feature analysis can find the mechanism — about a quarter of the time, on the published numbers. Second, a pre-deployment audit technique that requires model internals, as the hidden-objectives auditing game showed, which favours open weights or a vendor relationship that grants access. Third, a runtime monitor: persona and emotion vectors give you measurable behavioural directions, and the open monitorability evaluations give you a way to check whether your reasoning traces are still worth reading.

The honest framing for stakeholders is Amodei's MRI analogy. The scan gets sharper each year, and it already finds things that were deliberately hidden. It does not yet see everything, and the papers building it say so in their own limitations sections.