Between 2020 and 2022, artificial intelligence transitioned from an artisanal empirical science to an empirical branch of computational physics. With the discovery of neural scaling laws, AI research recognized that loss, downstream task generalization, and emergent reasoning were not erratic stochastic phenomena, but predictable power-law functions of compute, parameter count, and training dataset volume.
1. Kaplan et al. (2020) & The Power-Law Frontier
In 2020, Jared Kaplan and his colleagues at OpenAI published “Scaling Laws for Neural Language Models”. They demonstrated that the cross-entropy test loss $L$ of autoregressive language models scales as a precise power-law with compute ($C$), dataset size ($D$), and parameter count ($N$), holding performance independent of network depth or width when properly tuned:
$$L(N) = left(frac{N_c}{N}right)^{alpha_N}, quad L(D) = left(frac{D_c}{D}right)^{alpha_D}, quad L(C) = left(frac{C_c}{C}right)^{alpha_C}$$
Critically, the scaling exponents showed that performance improved smoothly over orders of magnitude without plateauing. Kaplan’s original formulation suggested that to optimize a fixed compute budget $C approx 6 N D$, model parameters ($N$) should scale rapidly ($N propto C^{0.73}$) while dataset tokens ($D$) scaled modestly ($D propto C^{0.27}$). This led directly to massive parameter expansions, culminating in the 175-billion parameter GPT-3.
2. The Chinchilla Correction: Optimal Compute Allocation
In March 2022, Jordan Hoffmann and the DeepMind team re-evaluated these power laws in their landmark paper “Training Compute-Optimal Large Language Models” (introducing the 70B parameter model Chinchilla). DeepMind revealed that Kaplan’s training runs had used a constant learning rate schedule that systematically underestimated the value of token volume.
Fitting a parametric loss function modeling both parameters and tokens:
$$L(N, D) = E + frac{A}{N^alpha} + frac{B}{D^beta}$$
DeepMind established that the scaling exponents are nearly identical: $alpha approx 0.34$ and $beta approx 0.28$. Under optimal compute allocation:
$$N propto C^{a}, quad D propto C^{b}, quad text{where} quad a = frac{beta}{alpha + beta} approx 0.50, quad b = frac{alpha}{alpha + beta} approx 0.50$$
Thus, model parameters and training tokens must scale in equal 1:1 proportion. For every doubling of parameters, the training dataset size must also double ($D approx 20N$ tokens). Models like GPT-3 (175B parameters trained on 300B tokens) were severely undertrained; Chinchilla (70B parameters trained on 1.4 trillion tokens) drastically outperformed GPT-3 and Gopher while requiring less memory and inference latency.
3. The Emergence of In-Context Learning
The profound historical consequence of scaling was the emergence of In-Context Learning (ICL). Prior to GPT-3, applying a model to a novel task required supervised fine-tuning—backpropagating gradients through the entire network using labeled task data.
At 175 billion parameters, GPT-3 demonstrated that sufficiently scaled autoregressive models could perform zero-shot and few-shot tasks purely via prompting, without a single parameter update:
$$theta^* = theta, quad hat{y} = argmax_y P_theta(y mid x_{text{prompt}}, x_{text{context}})$$
The model acts as an algorithmic meta-learner, dynamically simulating task representations within its attention activations during the forward pass. This realization marked the birth of Foundation Models: single generalist systems trained self-supervised on internet-scale corpora that serve as the base substrate for infinite specialized downstream tasks.
4. Emergent Abilities vs. Metric Discontinuities
As models crossed scale thresholds (typically $>10^{22}$ FLOPs), they began exhibiting abilities not present in smaller models: multi-step mathematical reasoning, symbolic translation, code synthesis, and Theory of Mind benchmarks. While debates emerged regarding whether emergent abilities were sudden phase transitions or artifacts of nonlinear scoring metrics, the practical reality was undeniable: scaling compute and data transformed passive text completion into capable cognitive engines.
