Between the publication of the original Transformer in 2017 and the release of LLaMA and Mistral in 2023, the foundational architectural blueprint of deep autoregressive language models underwent a silent, systematic Darwinian evolution. While early models (GPT-1 through GPT-3) adhered closely to Vaswani’s initial formulations, modern trillion-parameter foundation models have converged onto an ultra-optimized, unified architectural standard: Pre-RMSNorm, SwiGLU non-linearities, Rotary Position Embeddings (RoPE), and untied vocabulary projections.
1. The Evolution of Autoregressive Decoder Blocks
A causal autoregressive language model decomposes the joint probability of a sequence $X = (x_1, dots, x_T)$ via the chain rule of probability:
$$P(X) = prod_{t=1}^T P(x_t mid x_1, dots, x_{t-1})$$
To enforce causality, self-attention applies a lower-triangular causal attention mask $M in mathbb{R}^{T times T}$ where $M_{ij} = -infty$ for $j > i$, preventing queries at position $i$ from accessing future keys $j > i$.
Over six years of rigorous industrial ablation, four architectural choices were definitively discarded:
- Post-LN $to$ Pre-LN: Moving LayerNorm from the outer residual sum directly into the sublayer input branches eliminates gradient vanishing across depth.
- LayerNorm $to$ RMSNorm: Removing mean-centering accelerates forward and backward passes by $10%-30%$ without degradation.
- ReLU/GELU $to$ SwiGLU: Replacing standard feed-forwards with gated bilinear projections substantially increases parameter expressivity.
- Absolute Positional Encodings $to$ RoPE: Replacing static positional vectors with multiplicative rotary embeddings unlocks context extrapolation.
2. SwiGLU: Gated Linear Units and Expressive Volume
In 2020, Noam Shazeer at Google introduced GLU (Gated Linear Unit) Variants for Transformers. In standard architectures, the FFN applies a single non-linear projection: $text{FFN}(x) = text{GELU}(x W_1 + b_1) W_2 + b_2$.
Shazeer’s SwiGLU (Swish Gated Linear Unit) splits the input through two parallel linear projections, gating one with the continuous Swish/SiLU activation function $text{swish}(z) = z cdot sigma(beta z)$ before element-wise multiplication:
$$text{SwiGLU}(x) = left( text{swish}(x W_1) odot (x W_2) right) W_3$$
To maintain identical parameter parity with standard $4 d_{text{model}}$ expansion, SwiGLU sets the intermediate hidden dimension to $d_{text{ffn}} = frac{8}{3} d_{text{model}}$. Empirically, SwiGLU delivers lower perplexity at identical compute budgets, becoming the universal standard in PaLM, LLaMA, Mistral, and Gemma.
3. Parameter Allocation and Layer Sizing
Given total parameter budget $N$, how should layers, hidden width, and attention heads be sized? Let $L$ be depth, $d$ be hidden dimension, and $V$ be vocabulary size. Non-embedding parameters are dominated by:
$$N_{text{non-emb}} approx 12 L d^2$$
Across the historical frontier, empirical balance favored wide, deep networks with key dimension $d_k = d / h = 128$:
- GPT-3 (175B): $L = 96, d = 12288, h = 96, d_k = 128$
- PaLM (540B): $L = 118, d = 18432, h = 48, d_k = 256$
- LLaMA-3 (70B): $L = 80, d = 8192, h = 64, d_k = 128, V = 128256$
4. Untied Output Embeddings: Discarding Weight Tying
In early language models (Press & Wolf, 2017), the input token embedding matrix $W_e in mathbb{R}^{V times d}$ was shared with the final unembedding projection head: $W_{text{head}} = W_e^T$. While weight tying saves $V times d$ parameters when $|V|$ is small, modern models employ massive token vocabularies ($V = 128text{k}$ to $256text{k}$).
Chung et al. (2020) demonstrated that tying input and output representations forces an artificial symmetry: input embeddings seek semantic clustering, whereas output logits require sharp geometric separation to assign near-zero probabilities to irrelevant words. Decoupling (untying) $W_e$ and $W_{text{head}}$ improves downstream reasoning accuracy across scale.
