For sixty years, deep neural networks were treated as inscrutable, empirical “black boxes”: billions of floating-point numbers optimized by gradient descent that somehow produce intelligent outputs. Mechanistic Interpretability treats deep networks not as mystical oracles, but as complex electronic circuits that can be reverse-engineered down to their individual logic gates, attention routing circuits, and residual stream communications.
1. The Residual Stream as a Linear Communication Bus
Elhage et al. (2021) formulated the foundational framework for Transformer circuit analysis. In a modern Transformer, the Residual Stream acts as a shared, high-dimensional linear communication bus $mathbf{x} in mathbb{R}^d$ running from input embeddings to the unembedding layer:
mathbf{x}_L = mathbf{x}_0 + sum_{l=1}^L text{Attn}_l(mathbf{x}_{l-1}) + sum_{l=1}^L text{MLP}_l(mathbf{x}_{l-1})
Individual attention heads and MLP layers do not overwrite the stream; they read specific linear subspaces from the bus via projection matrices and write independent additive updates back onto it. Attention heads serve as information routers (moving data across sequence positions), while MLPs serve as key-value associative memories that store factual associations.
2. Dissecting Attention Circuits: $QK$ vs $OV$ Decomposition
The mathematical operation of an attention head $h$ in layer $l$ factors cleanly into two distinct linear circuits:
- The $QK$ Circuit ($W_{QK} = W_Q W_K^T in mathbb{R}^{d times d}$): Determines where information is routed. Computes the bilinear form $mathbf{x}_i^T W_{QK} mathbf{x}_j$ to dictate which source token position $j$ the current target token $i$ attends to.
- The $OV$ Circuit ($W_{OV} = W_V W_O in mathbb{R}^{d times d}$): Determines what information is routed. Reads from source token representation $mathbf{x}_j$, applies linear map $W_{OV}$, and adds the result into target position $i$’s residual stream.
3. Causal Mediation Analysis: Activation Patching
How do interpretability researchers prove that a specific neural circuit actually causes a model behavior rather than merely correlating with it? The gold standard is Activation Patching (Causal Tracing):
- Run a “clean” prompt through the model: “The Eiffel Tower is located in the city of [Paris].” Record all internal activations.
- Run a “corrupted” counterfactual prompt: “The Colosseum is located in the city of [Rome].”
- Surgically intervene: overwrite the internal hidden state of a single specific attention head or MLP layer during the corrupted run with the activation recorded during the clean run.
- Measure how much the corrupted run’s output probability shifts back toward “Paris”.
Activation patching has localized factual recall to mid-layer MLP modules and in-context copy algorithms to two-layer induction circuits, proving that neural networks compile into interpretable, compositional algorithms.
