The History of Computing & AI (Part 18): The Architecture of Universal Attention — ‘Attention Is All You Need’, Transformers, and Scaling Laws

Throughout the history of computer science, architectures were designed to process sequence step-by-step: word after word, cycle after cycle, token after token. In June 2017, a team of eight researchers at Google Brain published an eight-page paper with an audaciously simple title: “Attention Is All You Need.” By casting aside recurrence and convolution entirely in favor of parallel self-attention, they invented the Transformer—the universal cognitive engine that powers every modern large language model on Earth.

1. The Sequential Bottleneck of Recurrent Nets

Before 2017, natural language processing was dominated by Recurrent Neural Networks (RNNs), LSTMs, and GRUs. An RNN processes a sentence sequentially from left to right, maintaining a hidden state vector $h_t$ that updates at each step:

$$h_t = f(h_{t-1}, x_t)$$

This sequential dependency introduced two fatal limitations:

• No Training Parallelism: Because step $t$ strictly requires the hidden state from step $t-1$, an RNN cannot parallelize training across sequence length $N$. Modern GPU clusters packed with thousands of parallel tensor cores sat idle, waiting for the sequential recurrence loop.
• Information Compression & Catastrophic Forgetting: As sequences grew past 100 words, passing meaning through an information bottleneck caused long-range semantic dependencies (such as the agreement between a subject in paragraph 1 and a pronoun in paragraph 3) to blur and vanish.

2. ‘Attention Is All You Need’ (2017)

At Google Brain in Mountain View, researchers Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Łukasz Kaiser, and Illia Polosukhin proposed a revolutionary paradigm: completely discard recurrence. Process the entire sequence simultaneously in parallel, allowing every token to look directly at every other token through Self-Attention.

The Mathematics of Scaled Dot-Product Attention

The Transformer projects every input token embedding into three separate learned vector representations: a Query ($Q$), a Key ($K$), and a Value ($V$), each of dimension $d_k$:

$$Q = X W_Q, \quad K = X W_K, \quad V = X W_V$$

The attention weight between any two tokens is computed via the dot product of the Query with all Keys, scaled by $\frac{1}{\sqrt{d_k}}$ to prevent gradient vanishing in large dimensions, normalized through a softmax function, and multiplied by the Values:

$$\text{Attention}(Q, K, V) = \text{softmax}\left( \frac{Q K^T}{\sqrt{d_k}} \right) V$$ “` Transformer Scaled Dot-Product Attention Flow: Queries (Q) —+ |—> [ MatMul: Q * K^T ] —> [ Scale: 1/sqrt(d_k) ] —> [ Softmax ] —+ Keys (K) ——+ |—> [ MatMul: Attention * V ] —> Output Values (V) —-+ “`

Because attention computes every pairwise relationship simultaneously across an $N \times N$ matrix, the path length between any two words is $O(1)$, completely eliminating vanishing gradients over distance. To capture different semantic relationships concurrently (such as syntactic agreement, coreference, and semantic roles), the Transformer employs Multi-Head Attention (MHA):

$$\text{MultiHead}(Q, K, V) = \text{Concat}(\text{head}_1, \dots, \text{head}_h) W^O \quad \text{where } \text{head}_i = \text{Attention}(Q W_i^Q, K W_i^K, V W_i^V)$$

3. The Pre-Training Revolution: BERT and GPT

The Transformer’s massive parallel throughput unlocked unprecedented scalability. In 2018, natural language processing split into two foundational branches:

• BERT (Bidirectional Encoder Representations from Transformers, Jacob Devlin, Google, 2018): Utilized the Transformer Encoder to train bidirectional representations using Masked Language Modeling (filling in randomly blanked tokens: The [MASK] sat on the mat). BERT set state-of-the-art records across 11 NLP benchmarks overnight.
• GPT (Generative Pre-trained Transformer, Alec Radford et al., OpenAI, 2018–2020): Utilized the Transformer Causal Decoder with masked self-attention to predict the next token in a sequence: $P(w_t \mid w_1, \dots, w_{t-1})$. OpenAI demonstrated that unsupervised next-token prediction over vast internet corpora produces emergent multi-task reasoning capabilities.

4. Neural Scaling Laws: The Empirical Physics of AI

In 2020, OpenAI theoretical physicist Jared Kaplan and his collaborators published a transformative empirical study: Scaling Laws for Neural Language Models. They discovered that language model performance does not depend on clever architectural tweaks, but follows precise, continuous power-law relationships spanning more than six orders of magnitude.

Cross-entropy loss ($L$) scales predictably as a power law of model parameters ($N$), dataset tokens ($D$), and floating-point operations of compute ($C$):

$$L(N) \approx \left(\frac{N_c}{N}\right)^{\alpha_N}, \quad L(D) \approx \left(\frac{D_c}{D}\right)^{\alpha_D}, \quad L(C) \approx \left(\frac{C_c}{C}\right)^{\alpha_C}$$

The Chinchilla Optimal Correction (2022)

In 2022, Jordan Hoffmann and the DeepMind team revised Kaplan’s initial findings in their landmark paper Training Compute-Optimal Large Language Models (Chinchilla). DeepMind proved that earlier models (like GPT-3 with 175B parameters trained on 300B tokens) were severely undertrained.

For compute-optimal training, model size $N$ and dataset size $D$ must be scaled in equal proportion ($N_{opt} \propto C^{0.5}, D_{opt} \propto C^{0.5}$): for every doubling of model parameters, the training dataset must also double. The Transformer was not merely another neural network; it was a scalable mathematical engine whose performance scaled reliably with raw thermodynamic compute.

---