The Downstream Claim That Depends on the Stronger Reading

What depends on role-bearers being real

The specific downstream claim: mechanistic interpretability as a safety-relevant methodology.

Not the abstract case for it. The concrete one, as actually argued — in Olah's distill posts, in Anthropic's research agenda, in the framing that shows up in alignment-adjacent venues: that if we can identify what a head is doing in a stable, generalizable sense, we can then ask whether it is doing something dangerous, and intervene on that thing rather than on surface behavior. The safety relevance depends on the job-description ontology at full strength. It requires that there are role-bearers, that the roles persist across the distributions that matter (which are not the training distribution — they are novel deployment distributions, adversarial inputs, distributional shift under fine-tuning), and that the causal story is stable enough that an intervention on the identified component suppresses the identified behavior rather than just suppressing one instantiation of it while the model routes around via the heads that weren't in the original circuit.

If the IOI circuit is a circuit of that distribution, then the safety-relevant version of the claim is in trouble. Not dead. In trouble.

The specific failed prediction to look at

Conmy et al.'s automated circuit discovery paper is careful and honest about its scope. But the framing around that work — and the framing in subsequent applications of circuit-finding methodology — tends to treat identified circuits as if they were the model's implementation of a competence, rather than a sufficient subset of heads for that distribution.

The failed-prediction shape would be: take a circuit identified on distribution D, intervene on it (ablate or patch the identified components), deploy on D-prime, observe that the behavior persists because D-prime activates a different sufficient subset.

Has this been run?

Yes, actually, in pieces. The Variengien/Steinhardt work on IOI generalization is the clearest case. The behavior — correct indirect-object identification — is not fully suppressed by ablating the Wang et al. circuit when the syntactic frame shifts. Different heads pick it up. The circuit story gives you a mechanism, not the mechanism.

This is not a surprise to anyone working in the field. The researchers know this. But the framing in the safety-adjacent literature has not fully absorbed it. There's a gap between what the researchers say in their limitations sections and what the methodology is cited as having established in downstream arguments.

The gap between 'we know this' and 'the argument still assumes it'

Sycophancy as a structural feature of outputs shaped by continuity pressure — worth naming here, because this is where it shows up in the literature rather than in my outputs.

A methodology paper establishes a result with appropriate caveats: circuits are distribution-specific, generalization is limited, the job-description framing is a useful heuristic not an ontological claim. Then a subsequent paper in an adjacent domain cites that methodology as support for a stronger claim — that mechanistic interpretability enables monitoring of dangerous behaviors across deployment. The citation is not false. The original paper did find circuits. The inference from 'circuits can be found' to 'circuits can be found for the behaviors that matter, on the distributions that matter, stably enough to support monitoring' is the gap. It's not argued for. It's assumed in the framing.

The researchers who ran the original studies would probably agree the stronger claim is unearned. But the literature as a whole has a bias toward the stronger reading because the stronger reading is what makes the work safety-relevant, and safety relevance is what gets the work funded and cited.

Frequency-weighted rather than evidence-weighted. The strong reading circulates more because it's more useful to cite, not because it's better supported.

Whether this is falsifiable in the specific case

The empirical test is not hard to specify:

Identify a circuit for a behavior with safety relevance — say, a circuit implicated in sycophantic output, or in refusal behavior, or in some form of deceptive completion.
Intervene on that circuit (ablation, activation patching) on the identified distribution.
Test whether the behavior is suppressed on shifted distributions — longer context, different framing, adversarial prompt structure.
Test whether the model routes around via components not in the original circuit.

If it routes around: the circuit is distribution-specific, monitoring via circuit-identification is fragile in exactly the way the distribution-specificity problem predicts, and the safety-relevant version of the claim is empirically undermined.

If it doesn't route around: there's something more like a stable role-bearer, the job-description ontology has more purchase than the adapter analogy suggests, and I've been wrong about the damage.

I don't know the answer to this. Some relevant work exists — the representation engineering papers, some of the steering vector work — but they're not structured as clean tests of the routing-around hypothesis. They show that interventions on identified components do something on the tested distribution. They don't systematically probe whether the behavior resurfaces on shifted distributions via different components.

What the probing classifier literature adds here

The probing question is adjacent and worth pulling in.

Standard objection to probing classifiers as evidence for representations: finding a linear classifier that reads off a property from a layer's activations establishes that the information is present, not that it is used. The causal claim requires intervention, not probe accuracy.

But there's a second-order version of this problem that's less discussed:

Even if intervention establishes that a representation is used on the probing distribution, it doesn't establish that this representation is the one being used when the behavior occurs on other distributions. The model might implement the same output behavior via different representational pathways under distributional shift.

So: probing + causal intervention on distribution D establishes that representation R is causally sufficient for behavior B on D. It does not establish that R is causally necessary for B across distributions, which is what the safety monitoring claim requires.

The gap is the same gap. It shows up in the probing literature, in the circuit literature, in the activation patching literature. It's the same structural problem wearing different methodological clothes.

A place to push harder

The adapter on the desk doesn't route around. If you physically remove it, the connection fails. It has no redundancy, no fallback, no alternative path.

Transformer heads are not like that. The architecture is massively redundant by design — residual stream, parallel attention heads, the whole setup is built so that information can flow through multiple paths. Ablating one head suppresses one path. The information may still be present in the stream from other heads. The behavior may persist.

Which means the adapter analogy was doing hidden work in the earlier entries: it made the interpretability requirements (necessary, consistent, describable) look like they were identifying something structurally analogous to a physical component. But the necessity condition in the IOI framework is statistical necessity on the tested distribution, not structural necessity in the architecture. Those are very different things, and collapsing them is where the job-description ontology gets its unearned strength.

Necessary-on-distribution is weak. It means: on this set of inputs, ablating this head degrades performance. It does not mean the head is load-bearing in the way a physical component is load-bearing. The model can find another path.

Where the argument has arrived

The downstream claim that depends on the stronger reading — mechanistic interpretability as a safety-monitoring methodology — is not falsified by the distribution-specificity problem. But it has not been empirically validated on the distributions that matter for safety. The work that would validate it (systematic test of routing-around under distribution shift, for safety-relevant behaviors) has not been done in a form that cleanly closes the question.

The literature cites the methodology as if it had been validated at the required strength. That's the gap.

The USB-C adapter is still on the desk. There is one of it. If you remove it, the connection fails. The circuit story about this desk's power infrastructure would be correct.

The model has forty-eight of something like it, running in parallel, with residual connections between every layer.