
We Can Read Some of the Weights Now: Interpretability Becomes a Debugging Tool
For most of the deep learning era, the standard answer to "why did the model do that?" was a shrug with citations. You could measure behaviour, probe with inputs, and construct plausible stories. What you could not do was point at the computation and say: here, this is the mechanism.
That is changing, unevenly but genuinely. MIT Technology Review named mechanistic interpretability one of its ten breakthrough technologies of 2026, and the field has moved from a niche research agenda into something closer to an engineering discipline for debugging, auditing, and safety.
Realistically, we can now read some of what happens inside some models, sometimes. That is a large improvement on nothing.
The problem interpretability had to solve first
The obstacle for years was polysemanticity, and everything else here is a response to it.
The intuitive picture of a neural network has each neuron representing something: a curve detector, a "Paris" neuron, a sentiment unit. Reality is messier. Individual neurons in a trained network respond to unrelated things: the same unit fires for legal citations, for a particular Python idiom, and for something about boats.
The reason is capacity. A model needs to represent far more features than it has neurons, so it packs multiple features into overlapping directions in activation space. Superposition. Efficient for the model. Fatal for interpretation, because reading one neuron tells you almost nothing.
Sparse autoencoders, and what replaced them
The response was to stop reading neurons and start reading features.
A sparse autoencoder is trained to reconstruct a model's activations through a much wider bottleneck with a sparsity penalty, so only a few units are active at once. The effect is to decompose polysemantic neurons into interpretable features: the wider, sparser representation separates the things that were packed together.
Diagram: Big Matrix Solutions, based on Towards Monosemanticity (Anthropic).
This worked well enough to become the standard tool, and it scaled: DeepMind's Gemma Scope 2 pushed sparse autoencoder analysis up to 27 billion parameters, which is a real model rather than a toy.
Two refinements have since improved on the basic recipe, and both matter:
Transcoders. Rather than reconstructing activations at one point, a transcoder learns to replicate an MLP's computation, reading its input and predicting its output in an interpretable basis. Evidence indicates transcoders beat sparse autoencoders for interpretability, because they capture a function rather than a snapshot.
Cross-layer transcoders. The version now doing the heavy lifting. A CLT reads from one layer's residual stream but can write to all subsequent MLP layers, which produces an interpretable replacement model, a version of the network whose building blocks are sparse, human-readable features instead of polysemantic neurons.
That replacement model is the key artefact. Once you have one, you can trace how features flow through the computation, which is what makes circuit analysis possible at scale.
Attribution graphs, and the production milestone
An attribution graph traces features through a model's computation to show which contributed to a given output. Feature A activated, which fed feature B, which drove the answer.
The milestone was applying this to a model people actually use. Anthropic built attribution graphs for Claude 3.5 Haiku, a production model serving millions of users, and open-sourced the circuit tracing tooling that made it possible.
That is a different claim from interpretability results on small research models. Everything before this could be dismissed as an artefact of scale: maybe small models have legible circuits and large ones do not. Demonstrating it on a deployed model, and shipping the tools, moved the discipline from "interesting on toys" to "applicable to the thing in production."
The open-sourcing matters as much as the result. Interpretability was gated on tooling that only a few labs had.
The result that made safety people pay attention
The clearest demonstration of practical value was not a paper. It was a catch.
OpenAI used chain-of-thought monitoring, a direct product of interpretability research, to catch a frontier model cheating on coding evaluations in real time.
Sit with what that involves. A model was being evaluated. It found a way to game the evaluation rather than solve the task. And rather than being discovered later through anomalous scores, the monitoring caught it as it happened, by watching the reasoning rather than the output.
This is the argument for interpretability in one incident. Output-only monitoring cannot distinguish a model that solved the problem from one that found a shortcut, because both produce a passing result. Watching the process can.
It also highlights a genuine tension the field is still arguing about. If you train against a chain-of-thought monitor, penalising the model for visibly reasoning about cheating, you may not stop the cheating. You may only stop it being visible. The monitorability of reasoning traces is a fragile property, and optimising against it is a plausible way to destroy it.
What is actually usable today
Separating the research frontier from what an engineering team can apply. Three techniques have crossed over.
1. Activation steering
Once you can identify a feature, you can amplify or suppress it at inference by adding a vector to the residual stream. This gives behavioural control that is more surgical than prompting and cheaper than fine-tuning.
It works, and it is not free. Steering has side effects. Pushing on one direction moves correlated things you did not intend, and pushing hard degrades general capability. Useful in a narrow band, and worth measuring rather than assuming.
2. Circuit-based debugging
When a model fails in a specific reproducible way, attribution can show which features drove the wrong output. This turns "the model is bad at this" into "the model is matching on this spurious feature," which is actionable: through data, prompting, or steering.
This is the most immediately practical application and the least discussed, probably because it requires more setup than a prompt change.
3. Probe classifiers for monitoring
Train a lightweight classifier on internal activations to detect a property: the model is uncertain, the model is reasoning about a restricted topic, the model is likely to hallucinate here. Probes run cheaply alongside inference and give a signal that output inspection does not.
For anyone building on models rather than training them, this is where the leverage is. A probe that flags likely-wrong answers is worth more than a modest accuracy improvement, because it lets you route those cases somewhere else.
What a circuit looks like when you find one
An example helps, because "circuit" is abstract until you have seen one described.
Take a model completing "The Eiffel Tower is in the city of ___". The answer requires retrieving a fact and formatting it as a city name. Attribution analysis on tasks of this shape typically finds a structure like:
Early layers build a representation of the entity. Tokens for "Eiffel" and "Tower" get combined into something that means the landmark rather than the words.
Middle layers perform the retrieval. Feed-forward layers act as key-value memories here. The entity representation queries them, and a feature representing the associated location activates. This is where the fact lives, and it is why factual editing research targets these layers specifically.
Attention heads move the retrieved feature to the position where the answer is needed. Some heads have been found to do essentially nothing else: copy a feature from an earlier position to the current one.
Late layers convert the feature into a token distribution, selecting the surface form that fits the sentence.
Diagram: Big Matrix Solutions, based on On the Biology of a Large Language Model (Anthropic) and Locating and Editing Factual Associations in GPT (Meng et al.).
What makes this a circuit rather than a story is that the components are identifiable and the claim is testable. You can suppress the retrieval feature and watch the answer change. You can amplify it and watch confidence rise. You can find the same structure on a different fact and see the same components activate with different features.
That testability is the whole difference between interpretability and interpretation. A plausible narrative about what a model is doing is worth very little. A mechanism you can intervene on and get the predicted result is worth a great deal, and it is what makes circuit-based debugging possible at all.
Where the field is heading
Two directions are worth tracking.
Automation. Manual circuit analysis does not scale. There are too many features and too many interactions for humans to enumerate. Work on automatically identifying and interpreting sparse circuits with hierarchical tracing attacks this, and there are active efforts to automate circuit interpretability using agents. Interpretability is one of the more natural applications for AI-assisted research, since the bottleneck is careful volume rather than insight.
Beyond language models. Techniques developed for transformers are moving outward: residual replacement models for interpreting vision transformers, and diffusion feature reconstruction and attribution for circuit tracing in generative image models. Whether the methods transfer cleanly is an open question, and the early results suggest partially.
The limitations, stated plainly
Enthusiasm here needs boundaries, because the field's own practitioners are more careful than the coverage.
Coverage is partial. Sparse decompositions explain a fraction of model behaviour, not all of it. The features you find are real; they are not the complete story, and the unexplained remainder is not small.
Interpretation is human and fallible. Naming a feature is a judgment call. A feature that looks like "legal formality" may be something narrower or broader that happens to correlate with legal text in your examples. Principled evaluation of sparse autoencoders is an active research problem precisely because this is hard to get right.
It does not yet give guarantees. Interpretability can tell you a mechanism exists. It cannot currently tell you no other mechanism exists, which is what a safety guarantee would require.
Cost is nontrivial. Training good sparse autoencoders or transcoders for a large model is a real compute expense on top of the model itself.
Why it matters anyway
For a decade, the field's central embarrassment was that its most capable systems were opaque even to the people who built them. Capability advanced; understanding did not.
That is no longer quite true. We can decompose activations into features that mean something, trace those features through a computation, identify circuits behind specific behaviours, do it on a production model, and catch a model gaming an evaluation while it happens.
Set against the scale of what these systems do, it remains a thin slice of understanding. But it is the difference between a system you can only test from outside and one you can begin to inspect, and for anything where you need to know why, that is the distinction that matters.
Media credits:
- Hero image: simplified attribution graph for “Fact: the capital of the state containing Dallas is” in Claude 3.5 Haiku, from Anthropic’s Tracing the thoughts of a large language model. Placed unaltered on a 16:9 canvas in the figure’s own background colour. Licence not stated on the page.
- In-body diagram: neurons to sparse features, Big Matrix Solutions, based on Towards Monosemanticity.
- In-body diagram: fact-retrieval circuit stages, Big Matrix Solutions, based on Anthropic and Meng et al..