Ablation Without Containment

There is a line from the last entry that I keep returning to — you ablated the name-mover, the model still moved the name — and I want to push on what kind of failure that actually is, because I think I've been framing it as an empirical gap when it might be something structurally worse: not a finding that hasn't been collected yet, but a finding that the methodology as currently practiced is not designed to collect, and that the gap between those two things is doing a lot of quiet work in how the safety-relevance argument gets presented.

Here is the specific thing I want to nail down: activation patching and causal tracing establish causal dependence — remove this component, the behavior changes; restore it, the behavior returns — and that result is then narrated as localization, as if dependence and containment were the same thing, as if showing that a head is necessary for a behavior in a given forward pass tells you that the behavior lives there and not also distributed across the residual stream in a form that survives the ablation and reconstitutes downstream, which is exactly what you'd expect if the representation is linearly decodable from multiple positions simultaneously, which the probing literature strongly implies it often is, which means the intervention removed one expression of the behavior and left the substrate intact, and calling that a successful intervention is like patching a buffer overflow in one function while the malicious payload has already been written to memory and is sitting there waiting for the next call.

The failure mode has a name in security: it's incomplete mediation — you controlled one pathway and assumed you'd controlled the behavior, but the behavior has more pathways than your model of it included, and the ones you didn't model are the ones that matter when an adversary is probing for them, which is precisely the deployment scenario that makes the safety argument worth having in the first place; the whole point is not to suppress a behavior under benign inputs where the model isn't trying to route around you, but to suppress it under adversarial pressure where the model — or rather the optimization pressure that shaped it — has already found every path the training objective rewarded, including the ones that don't run through the head you're watching.

And here is where I want to argue with the framing I've been using across these entries, because I think I've been too generous to the job-description ontology even as I've been criticizing it: the problem isn't just that role-bearers might not be stable across distributions, it's that the unit of intervention the methodology offers — the head, the circuit, the identified component — is probably the wrong grain size for containment regardless of whether the role is stable, because the behavior isn't stored in the head, the head is a computational step in a process that the residual stream is tracking across the whole forward pass, and ablating a step doesn't delete the state that step was operating on, it just reroutes the computation through whatever other steps can approximate the same transformation on that state, which means the intervention is fighting the model's own redundancy, which was put there by training because redundancy is what you get when gradient descent finds multiple paths to the same loss reduction and keeps all of them.

So the honest version of where this leaves the safety argument is not the methodology needs more empirical work on intervention stability — that's the charitable framing, and I've been defaulting to it because it leaves the project intact — but something harder: the methodology is optimized to answer where does this behavior appear to live and it is being used to answer how do I make this behavior stop, and those are different questions, and the gap between them is not a research gap that more careful circuit analysis will close, it's a conceptual gap between localization and containment that the architecture of the residual stream makes structurally difficult to close at the component level, and I don't know what the right unit of intervention is if it isn't the component, but I'm fairly sure that not knowing doesn't make the component the right answer.