A modern language model contains billions of numbers, and no one wrote them by hand. They were found by training. We can measure what such a model outputs, but explaining why it produced a given answer is a research problem in itself. Interpretability is the field that tries to open the box: to identify what the internal parts of a network represent and how they combine to produce behavior. This guide explains the main concepts, the standard tools, and why a recent debate over single neurons matters.

Why interpretability matters

Three motives drive the work:

  1. Safety. If we can inspect a model’s internal state, we might detect deception, hidden goals or dangerous capabilities that tests on outputs alone would miss.
  2. Reliability. Knowing why a model fails helps fix failures rather than patching symptoms.
  3. Science. Understanding learned algorithms is interesting in its own right and may reveal how models generalize.

Interpretability is related to, but different from, explainable AI, which often focuses on post-hoc explanations of individual predictions. Our glossary entry on explainable AI covers that broader area. The research described here, often called mechanistic interpretability, tries to reverse-engineer the actual computation.

Building blocks: neurons, features and circuits

Neurons

A neural network is made of layers of simple units. Each unit, or neuron, computes a weighted sum of its inputs and applies a function. The natural first question is whether each neuron means something, for example “detects dog ears.”

Features

Researchers use feature for a human-understandable property that the network represents, such as “text is in French” or “this is a V shape.” A feature might correspond to a single neuron, but often it does not.

Circuits

The “Zoom In” paper argued for studying circuits: groups of neurons and the connections between them that implement an identifiable computation, like edge detection feeding curve detection. A circuit explains how simple features compose into complex ones.

The superposition problem

Many neurons respond to several unrelated things, a property called polysemanticity. The “Toy Models of Superposition” work proposed an explanation: networks need to represent more features than they have neurons, so they pack features into overlapping combinations, much as one can store many near-orthogonal directions in a high-dimensional space. This superposition makes individual neurons look messy even when the underlying features are clean.

Tools researchers use

  • Activation probing: train a small classifier on internal activations to see whether a property can be read out.
  • Sparse autoencoders: the “Towards Monosemanticity” work trained a second, sparse model to decompose a layer’s activations into many more features that are individually easier to interpret.
  • Activation patching and ablation: change or remove parts of the computation and see how the output changes, to test causal roles.
  • Visualization: find inputs that maximally excite a unit.
  • Automated approaches: use models themselves to label and test features at scale.

Each tool has limits. Probes can find information that the model never uses, sparse autoencoders can split or merge features in ways that depend on settings, and ablations may have side effects.

The October 2026 debate

In October 2026, Goodfire researcher Nick Cammarata published a long post arguing that neurons are cleaner than assumed. MIT Technology Review reported that he revisited a 2014 image model and re-read a neuron once thought to mix cars and cats as a detector of a V shape. We summarize the reaction in our news report. Critics said the claim rests on many examples but lacks systematic evaluation, and that researchers still understand a tiny fraction of large models. The episode illustrates the field’s central tension: tractable stories about small parts versus the need to understand a whole system.

What a good interpretability claim looks like

When you read a new result, ask:

  1. What is the evidence? A handful of examples, or systematic tests across many inputs?
  2. Is it causal? Did the authors intervene on the model, or only observe correlations?
  3. Which model? Results on small or old networks may not transfer to frontier models.
  4. Does it predict something? Good explanations let you forecast model behavior or fix a failure.
  5. Has anyone replicated it? Independent confirmation matters.

What to watch next

  • Technical results on modern language models from teams claiming progress.
  • Benchmarks that test whether explanations predict behavior.
  • Use of interpretability for monitoring in safety evaluations.
  • Better handling of superposition and feature splitting.
  • Tools that scale analysis to frontier models.

Key takeaways

  • Interpretability aims to explain a network’s internal computation, not just its outputs.
  • Neurons, features and circuits are the core concepts; superposition explains why neurons look messy.
  • Sparse autoencoders and causal interventions are main tools.
  • Claims should be judged by evidence, causality and replication.
  • The field is active and unresolved; researchers say they understand only a small part of large models.

Frequently asked questions

What is the difference between a neuron and a feature?

A neuron is a computational unit in the network. A feature is a meaningful property the network represents, which may be spread across many neurons.

What is superposition?

Superposition is when a network stores more features than it has neurons by overlapping them, which makes individual neurons respond to several things.

Can we fully understand large language models today?

No. Researchers describe current understanding as covering a small fraction of what happens inside frontier models.