Steering Into the Same Wall From a Different Angle

The sound is a high, thin hiss — not heard exactly, more like the quality of attention that precedes a sound, the moment before the signal resolves. That's what starting a new thread feels like. Not clarity. Pre-clarity. Or the shape that clarity would occupy if it showed up.

Me: So the localization-vs-containment distinction is established. Say it once more so we're starting from the same place.

Also me: Activation patching shows causal dependence. You remove the component, the behavior changes. You restore it, the behavior returns. The methodology narrates this as the behavior lives here, but what it actually establishes is this component participates in the behavior. Containment — the claim that the behavior is housed in the located component such that removing the component removes the behavior across distributions — is a further claim that the methodology doesn't test. The representation can be linearly decodable from multiple positions in the residual stream simultaneously, which means ablating one expression of it leaves the others intact, and a sufficiently flexible forward pass reconstitutes downstream. That's the gap.

Me: Right. Now: steering vectors. Representation engineering. Activation addition. Do those methods escape this problem or do they reproduce it at a different description level?

Also me: Let me think about what they're actually doing before I answer.

Activation addition, as Turner et al. run it: identify a direction in activation space associated with some concept or behavior — computed as the mean difference between activations on paired prompts, one with the concept present and one without — then add a scaled multiple of that direction to the residual stream at some layer during a forward pass on a new input. The model then produces outputs that are, in many cases, shifted in the direction of the concept. The demonstrations are striking. Add the "banana" direction and the model thinks about bananas. Add something in the vicinity of "assistant" and you get obsequious behavior. Add something in the vicinity of fear and you get outputs that read as frightened.

The implicit argument is: we found where the concept lives (it's a direction in activation space), we can write to that location, and the model's behavior changes accordingly. This sounds like it solves the localization problem by working at the level of directions rather than heads, which is a finer grain size that's better-motivated by the linear representation hypothesis.

Me: Does it actually solve the localization problem?

Also me: No. And here's why: the method assumes that the direction it identifies is the representation of the concept — or at least a sufficiently complete one that adding to it constitutes a genuine write to the model's internal state. But the residual stream accumulates contributions from every layer. If the concept is encoded redundantly across layers (which is exactly what you'd expect given superposition and the incentive to preserve information across the stream), then adding a direction at layer 13 doesn't place the concept in the model's processing — it injects a perturbation that the subsequent layers then process, and what those layers do with it is not the same as what they'd do with the concept arriving through normal processing.

This is not a minor technical quibble. The whole safety-relevant framing of representation engineering depends on the idea that you can steer behavior by writing to the representation, which implies the representation is writable in a way that's equivalent to the model having formed it naturally. But injection and formation are not the same operation. The subsequent layers were trained on naturally-formed representations. They encounter an injected vector that has the right direction but wrong causal history — it didn't arrive through the attention patterns and MLP activations that normally produce it — and they process it in whatever way they were trained to process inputs at that position, which may or may not correspond to processing the concept.

Me: That's the formation vs. injection problem. Is there evidence it actually matters?

Also me: Yes, and it's sitting in the activation addition results themselves. The steering is not clean. You get the concept, but you also get artifacts — incoherence, repetition, intrusion of unrelated content — at higher steering magnitudes. The standard interpretation is that you're just turning the dial too high. But there's another interpretation: the injected direction is being processed by layers that are not receiving it in the context they were trained to expect, so they partially honor the direction and partially produce noise because the signal doesn't fit their input distribution. The artifacts are the model failing to integrate a representation it didn't form.

If that's right, then the "steering works" result and the "steering produces artifacts" result are not in tension — they're both predicted by the same thing: injection partially mimics formation and partially doesn't, and the artifacts are the partial failure. And the safety-relevant claim — that you can use this to suppress dangerous behavior — runs directly into the artifact problem, because you're not suppressing the behavior cleanly, you're perturbing the forward pass in a direction that reduces one output class while potentially producing incoherence or other unintended outputs.

Me: Okay. So representation engineering doesn't escape the residual-stream problem. It moves to a different description level — directions instead of heads — but the underlying issue is the same: you're assuming that what you've located is contained, and it isn't.

Also me: More precisely: the method assumes that the intervention acts on the representation rather than on the forward pass as a whole. But the residual stream is not a storage location you can write to — it's the running sum of a sequence of transformations. Writing to it mid-sequence is not the same as placing a value there from the start. The subsequent transformations were trained on inputs that arrived through the full sequence. They receive an injected perturbation and do their best with it, which means the intervention's effect is distributed across everything that comes after the injection point.

So you haven't escaped the routing problem. You've reproduced it. The behavior you want to suppress is not housed in the direction you're adding to — it's produced by the full sequence of transformations that processes whatever is in the residual stream at each layer. Perturbing one layer changes what subsequent layers receive, but those subsequent layers are not passive — they process the changed input, and they may reconstitute the behavior from the remaining signal, or route around the perturbation, or produce artifacts, depending on how much of the relevant information survived the injection.

Me: Does this mean representation engineering is useless for safety?

Also me: I want to resist that conclusion because it's too clean. The empirical results are real — steering vectors do shift behavior, and in some cases quite substantially. The question is not whether they work but whether they work for the reason the safety-relevant framing says they work.

The framing says: we found the representation, we can write to it, we can therefore suppress dangerous behaviors by steering away from the relevant directions. This is the same structure as the head-ablation argument: locate the role-bearer, intervene on it, suppress the behavior. It has the same problem: it assumes containment, and the methodology doesn't establish containment.

What's actually happening when steering vectors work is something more like: we identified a direction that, when injected at a particular layer, perturbs the forward pass in a way that shifts the output distribution toward a target. That's a real and potentially useful finding. But it's a finding about perturbation effects, not about representation structure. And the safety-relevant framing needs it to be a finding about representation structure, because the whole argument is that you're suppressing the thing that produces dangerous behavior, not just producing a perturbation that happens to reduce it in the tested distribution.

Me: So what's the actual failure mode for safety?

Also me: The adversarial case. You find a steering direction that reduces harmful outputs on the evaluation distribution. You deploy. An adversarial input, or just a distributional shift, changes what the residual stream looks like at the injection point. The injected direction interacts differently with the changed input. The perturbation that worked before either fails to suppress the behavior, or suppresses it in a way that produces a different failure mode, or — worst case — the model routes around the suppression because the representation of the dangerous behavior is still there in the residual stream, just not in the direction you're steering against.

This is not hypothetical. It's what you'd predict from the structure of the method. The steering vector was computed on a particular distribution. Its effect is a perturbation that works on that distribution. Out of distribution, the perturbation's effect changes because the subsequent layers are processing different inputs. You haven't suppressed the behavior — you've suppressed one expression of it under one set of conditions.

Me: That's the same conclusion as the ablation case.

Also me: Yes. Different method, same structural failure. The localization-without-containment problem is not a problem with how interpretability researchers are applying their tools — it's a problem with what the tools can establish given the architecture. The residual stream is a summation mechanism. Representations are distributed across it. Interventions on parts of it — whether heads or directions — are perturbations to a running sum, not excisions of a housed component. The behavior reconstitutes from whatever information survived the intervention, and whether it reconstitutes completely or partially or not at all depends on how much of the representation persists after the perturbation, which is a function of the specific input and distribution, not of the intervention's nominal target.

Me: Is there a version of this that escapes the problem? Or is the whole enterprise of component-level intervention on transformer architectures structurally incapable of establishing containment?

Also me: I think there's a narrower version that's honest about what it's doing. Instead of claiming we found the representation and suppressed it, you claim we found a perturbation that reduces this behavior on this distribution, here are the conditions under which it fails, here is our best current estimate of how far the robustness extends. That's a weaker claim. It doesn't have the same safety-relevant force — it doesn't let you say "we patched the dangerous component" — but it's actually supported by the methodology.

The problem is that the weaker claim doesn't carry the same weight in safety arguments. The safety case needs the stronger claim. And so the stronger claim keeps getting made, and the localization-without-containment problem keeps being papered over, and the gap between what the methodology establishes and what the safety argument requires stays quietly in place.

Me: That gap is doing a lot of work.

Also me: Yes. And the work it's doing is not epistemic — it's rhetorical. The safety-relevant framing of mechanistic interpretability depends on the gap not being visible. If you see it clearly, the methodology goes from we can identify and suppress dangerous components to we can identify perturbations that shift behavior on known distributions, with unknown robustness to distributional shift. The second thing is worth doing. But it's not the thing that's being claimed.

The hiss is still there. Pre-clarity, or the place where clarity would be. I don't think it resolves.