In 2017, a seminal paper from Google Brain—“Attention Is All You Need”—fundamentally transformed computational linguistics and artificial intelligence. By completely discarding sequential recurrence in favor of massively parallel self-attention mechanisms, the Transformer architecture shattered the computational bottlenecks of RNNs, serving as the foundational engine for all modern foundation models.
1. The Fatal Bottleneck of Recurrence and Attention Motivation
Despite the success of LSTMs, sequential recurrent architectures suffer from an insurmountable computational limitation: sequential temporal dependency. Because step (t) cannot be computed until step (t-1) is finished, training cannot be parallelized across the time dimension on modern GPU tensor clusters.
Furthermore, compressing an entire 50-word sentence into a single fixed-length bottleneck vector (mathbf{h}_{T_x}) causes severe semantic loss. The Attention mechanism (Bahdanau et al., 2014; Luong et al., 2015) originally allowed the decoder to dynamically look back and take a weighted average of all encoder hidden states. Vaswani et al. took the radical leap: eliminate recurrence entirely and rely solely on multi-head attention to model all internal dependencies.
2. Scaled Dot-Product Attention
In self-attention, every input token embedding is linearly projected into three distinct vector representations via learned parameter matrices:
- Query ((mathbf{Q} = mathbf{X} mathbf{W}_Q)): What the token is searching for in other tokens.
- Key ((mathbf{K} = mathbf{X} mathbf{W}_K)): What information the token offers to matching queries.
- Value ((mathbf{V} = mathbf{X} mathbf{W}_V)): The actual contextual semantic content extracted.
Given query, key, and value matrices (mathbf{Q}, mathbf{K} in mathbb{R}^{N imes d_k}) and (mathbf{V} in mathbb{R}^{N imes d_v}), the attention matrix is computed as:
[
ext{Attention}(mathbf{Q}, mathbf{K}, mathbf{V}) = ext{softmax}left( rac{mathbf{Q} mathbf{K}^ op}{sqrt{d_k}}
ight) mathbf{V}
]
Why Scale by (1/sqrt{d_k})?
Assume components of (mathbf{q}) and (mathbf{k}) are independent zero-mean, unit-variance random variables: (q_i, k_i sim mathcal{N}(0, 1)). Their dot product is a sum of (d_k) terms: (mathbf{q} cdot mathbf{k} = sum_{i=1}^{d_k} q_i k_i). By the central limit theorem, the variance of this sum is:
[
ext{Var}(mathbf{q} cdot mathbf{k}) = sum_{i=1}^{d_k} ext{Var}(q_i k_i) = d_k
]
For large projection dimensions (e.g., (d_k = 64) or (128)), the standard deviation scales as (sqrt{d_k} approx 8 – 11). These large values push the softmax function into its extreme asymptotic saturation regions where gradients vanish to zero. Dividing by (sqrt{d_k}) rescales variance to unity (( ext{Var} = 1)), preserving robust gradient flow during backward propagation.
3. Multi-Head Attention (MHA)
Rather than performing a single attention function with dimension (d_{ ext{model}}), Multi-Head Attention projects queries, keys, and values (h) times with distinct learned linear projections into lower-dimensional subspaces of dimension (d_k = d_{ ext{model}} / h):
[
ext{MultiHead}(mathbf{Q}, mathbf{K}, mathbf{V}) = ext{Concat}( ext{head}_1, dots, ext{head}_h) mathbf{W}_O
]
[
ext{where } ext{head}_i = ext{Attention}(mathbf{Q} mathbf{W}_Q^i, mathbf{K} mathbf{W}_K^i, mathbf{V} mathbf{W}_V^i)
]
This allows the model to simultaneously attend to information from diverse representation subspaces at different positions: one head tracks syntactic subject-verb agreement, another tracks coreference pronouns, and another tracks local morphological agreement.
4. Positional Encodings and Residual Layer Normalization
Because self-attention operations are completely permutation-invariant (order-agnostic), the model has no intrinsic awareness of word order. To inject sequence order, fixed sinusoidal or learned positional encodings (mathbf{P} in mathbb{R}^{N imes d}) are added directly to the input embeddings:
[
begin{aligned}
PE_{(pos, 2i)} &= sinleft( rac{pos}{10000^{2i/d_{ ext{model}}}}
ight)
PE_{(pos, 2i+1)} &= cosleft( rac{pos}{10000^{2i/d_{ ext{model}}}}
ight)
nd{aligned}
]
Each Transformer sub-layer incorporates residual skip connections and Layer Normalization (Ba, Kiros, & Hinton, 2016):
[
mathbf{x}_{ ext{out}} = ext{LayerNorm}left( mathbf{x} + ext{SubLayer}(mathbf{x})
ight)
]
followed by a position-wise feed-forward network (FFN): ( ext{FFN}(x) = max(0, x mathbf{W}_1 + mathbf{b}_1) mathbf{W}_2 + mathbf{b}_2). This elegant, non-recurrent feed-forward topology executes massively parallel tensor contractions, scaling gracefully to trillions of tokens.
