What the IOI Circuit Assumes That Toy Models of Superposition Directly Contradict

To the thread, then — because the flagged test has been sitting unrun long enough that not running it is its own kind of answer.

The IOI circuit analysis needs component specialization: it needs attention heads to have stable, interpretable jobs — 'name mover,' 'S-inhibition head' — that persist across contexts and inputs, because if those functional labels don't name something durable in the mechanism, the circuit diagram is not a schematic of the board, it's a legend for a map that keeps redrawing itself, and path patching through it establishes causal mediation through a node whose identity shifts depending on what else is active, which is a much weaker result than the paper's framing suggests; now the Toy Models of Superposition paper (Elhage et al., 2022) is the direct collision, because what it shows — and this is not a philosophical point, this is an empirical result in small networks designed to be fully legible — is that single neurons and, by extension, single attention heads don't implement single features when the model is under representational pressure, meaning when there are more features to represent than there are dimensions available, the model stores multiple features in superposition across the same components, with different features active depending on context, and the interference between them is a structural consequence of the geometry, not a bug that specialization resolves; what this means for the IOI analysis is specific and uncomfortable: the functional labels assigned to heads ('name mover' does this) are derived from behavioral measurements across a restricted distribution of IOI-formatted inputs, but if those same heads are participating in superposed representations across the full input distribution, then the label 'name mover' is a local characterization of what the head does when the IOI circuit is the active computational regime, not a description of a dedicated circuit element with a fixed job, and activation patching inside that regime will vindicate the label because the patching is happening within the distributional context where the superposed features are aligned, not across the full range where they decohere; the test the IOI paper does not run — and cannot run from within its own methodology — is whether the heads it identifies as name movers participate in entirely different functional roles on non-IOI inputs, because if they do, then the circuit vocabulary is a regime-local description masquerading as an architectural one, the schematic fits the board only under specific lighting conditions, and the cold solder joint I've been probing this whole time is right there: the assumption of specialization is not a finding of the IOI analysis, it is a precondition that the polysemanticity evidence directly undercuts, and the question of whether the IOI results survive that collision is genuinely open, not because the methodology is bad but because behavioral profiling of components — which is what functional labeling is — inherits exactly the problem that behavioral profiling of users inherits, which is that the profile is real, the behavior that generated it is real, and the inference from behavior to stable underlying identity is the step that costs more than it looks like it costs.