In June 2017, Ashish Vaswani et al. published “Attention Is All You Need”, unveiling the Transformer architecture. By completely banishing recurrence and convolutions in favor of self-attention, the Transformer removed the sequential temporal bottleneck of RNNs, unlocking fully parallel GPU hardware saturation and $O(1)$ maximum path lengths between arbitrary tokens. The Transformer is now the universal computational engine spanning natural language processing, computer vision, structural biology, and frontier AI.
1. Multi-Head Attention (MHA): Orthogonal Representation Subspaces
A single self-attention head averages over all semantic relationships simultaneously. However, language demands that a word attend concurrently to multiple disparate aspects: syntactic dependency, coreference resolution, and lexical semantic context. Multi-Head Attention projects queries, keys, and values into $h$ distinct lower-dimensional subspaces of dimension $d_k = d_{text{model}} / h$:
$$text{head}_i = text{Attention}(Q W_i^Q, K W_i^K, V W_i^V)$$
$$text{MultiHead}(Q, K, V) = text{Concat}(text{head}_1, dots, text{head}_h) W^O$$
where $W_i^Q, W_i^K in mathbb{R}^{d_{text{model}} times d_k}$, $W_i^V in mathbb{R}^{d_{text{model}} times d_v}$, and $W^O in mathbb{R}^{h d_v times d_{text{model}}}$.
By computing attention independently across orthogonal linear projections, heads specialize into functional modules: some heads track previous tokens, others track syntactic verbs, and others attend across long distances.
2. Sinusoidal Positional Encoding: Breaking Permutation Invariance
Self-attention is mathematically Permutation Equivariant: if the rows of input sequence $X$ are permuted by permutation matrix $P$, the output is identically permuted: $text{Attention}(PX) = P text{Attention}(X)$. To make the model cognizant of sequential word order, Vaswani et al. injected static geometric vectors:
$$text{PE}_{(pos, 2i)} = sinleft( frac{pos}{10000^{2i/d_{text{model}}}} right), quad text{PE}_{(pos, 2i+1)} = cosleft( frac{pos}{10000^{2i/d_{text{model}}}} right)$$
By the trigonometric angle addition formulas, any relative displacement $pos + k$ can be expressed as a linear transformation of $text{PE}_{pos}$, allowing the network to easily learn relative position offsets.
3. Position-Wise Feed-Forward Networks (FFN)
Attention only performs linear combinations of value vectors retrieved across tokens; it cannot synthesize new non-linear features within a token. The Feed-Forward Network (FFN) applied to each token independently supplies non-linear capacity:
$$text{FFN}(x) = sigma(x W_1 + b_1) W_2 + b_2$$
Typically, hidden dimension expands by a factor of 4: $d_{text{ffn}} = 4 times d_{text{model}}$. Geva et al. (2021) showed that the first layer $W_1$ acts as a key-value memory retrieval mechanism matching semantic concepts, while $W_2$ writes new factual representations into the token’s latent state.
4. The Residual Stream as a Shared Communication Bus
In modern mechanistic interpretability (Elhage et al., 2021), a Transformer layer stack is formalized not as a traditional sequential pipeline, but as a persistent Residual Stream $mathbf{x} in mathbb{R}^{d_{text{model}}}$.
The residual stream acts as a central communication bus. Attention heads and MLP layers read from the stream, perform their specialized computations, and write their delta updates additively back onto the stream:
$$mathbf{x}_{l+1} = mathbf{x}_l + sum_i text{head}_{l,i}(mathbf{x}_l) + text{MLP}_l(mathbf{x}_l)$$
This additive architecture allows earlier layers to transmit unmodified representations directly to the final layer, while later layers read and enrich features in parallel, providing the theoretical basis for modern transformer scaling.
