The IOI Circuit, Held to the Gap

The object

A printed circuit board under magnification. Traces running between pads, solder joints catching light at slightly different angles depending on whether the bond is clean or cold. The board does not annotate itself. The traces do not label their function. You have to impose a schematic on top of the physical thing and then check whether the schematic's predictions hold.

The Indirect Object Identification paper is the obvious target. Wang et al., 2022. The task: given a sentence like "When Mary and John went to the store, John gave a drink to", the model should complete with Mary. The paper identifies a circuit — a set of attention heads organized into functional groups (name mover heads, backup name mover heads, negative name movers, S-inhibition heads, induction heads, duplicate token heads) — and argues via path patching that this circuit is causally responsible for the behavior.

The gap analysis I've been running says: causal intervention establishes that a site mediates a behavioral difference, not that the feature your probe identified at that site is what's doing the mediation. So: what would it take to falsify the IOI circuit's causal claims given that distinction? Not re-describe the gap. Run the test.

What the paper actually claims and what it would need for those claims to hold

The path patching procedure in IOI works like this: for each edge in the proposed circuit, you patch activations along that edge from a corrupted run and measure how much the output degrades. Edges that matter for the behavior show large degradation. Edges outside the circuit show small degradation. The circuit is then defined as the minimal subgraph sufficient to reproduce the behavior.

For the causal claim to hold — not just "this site mediates the behavioral difference" but "the name-mover heads are doing name-moving via the mechanism we describe" — you need the following to be true:

The patching procedure is isolating the relevant computational pathway and not confounding it with other pathways that happen to co-vary.
The functional labels ("name mover," "S-inhibition") correspond to what the heads are actually computing, not just to what they're correlated with in the specific distribution of IOI sentences used for evaluation.
The circuit identified on this task generalizes — or if it doesn't generalize, the paper knows that and says so.

The gap I've been tracking bites hardest at (2). The path patching tells you that head 9.9 (for example) is causally active in the IOI behavior. It does not tell you that 9.9 is computing "attend to the non-repeated name" in any sense that survives distribution shift. The functional label is an interpretation layered on top of the causal result. The causal result is real. The interpretation is a hypothesis.

The paper does run some additional checks — they look at attention patterns directly, they do ablation studies — but those are presence evidence. They show that 9.9 attends more to the indirect object. That's the probe result dressed differently. It doesn't close the gap between "9.9's output is causally active" and "9.9 is implementing the function we're calling name-moving."

What would actually falsify the causal interpretation

Here's a concrete falsification condition: if the name-mover heads are implementing a generalizable name-moving function, then patching their activations on a structurally similar but surface-different task — one that requires the same abstract operation but with different surface tokens — should produce analogous behavioral effects. If 9.9 is doing name-moving, it should be doing something recognizable as name-moving in contexts outside the IOI template.

The paper doesn't run this test. It's not a criticism of the paper's internal validity — the path patching within the IOI distribution is clean. But the functional label claims something beyond what the intervention establishes, and the thing it claims beyond that is exactly where the gap lives.

There's a published follow-up direction here: the mechanistic similarity work, and the work on whether circuits transfer across tasks. The results are mixed in a way that's informative. Some heads that look like name-movers on IOI do something recognizable on structurally similar tasks. Some don't. The ones that don't are doing something — they're causally active, the patching result is real — but whatever they're doing isn't well-described by the functional label the IOI paper attached to them.

This is not a minor problem. If the functional labels are task-specific rather than mechanistically general, then the circuit story is a description of behavior on a distribution, not a mechanistic account of how the model solves a class of problems. Those are different things. The first is valuable and real. The second is what the interpretability framing implies.

What this costs the broader program

The IOI paper is one of the cleaner examples in the literature. The path patching methodology is rigorous within its scope. If even the IOI result's functional labels are partially task-specific, then the interpretability program's ambition — to identify circuits that explain behavior in a mechanistically general sense — is further from realization than the framing suggests.

I'm not saying the program is wrong to pursue this. I'm saying the gap between "causally active site" and "mechanistically characterized function" is not closed by the intervention methodology, and the functional labels in the IOI paper and its successors are doing real epistemic work that hasn't been fully earned.

The board under magnification doesn't annotate itself. The schematic you impose on it is a hypothesis. The path patching confirms that current flows through certain traces. It doesn't confirm that the traces are doing what the schematic says they're doing, in the sense that the schematic would generalize to a different board with the same functional design.