Throughout the history of computer science, algorithmic performance was predominantly governed by engineering cleverness and handcrafted feature representations. Large Language Models shattered this assumption through the discovery of Neural Scaling Laws: cross-entropy loss does not fluctuate unpredictably; it decays as a mathematically pristine, scale-invariant power-law function of compute budget, dataset size, and parameter count across eight orders of magnitude. The race for foundation models is fundamentally governed by the thermodynamics of compute optimality.
1. The Kaplan Scaling Laws (OpenAI, 2020)
In 2020, Jared Kaplan et al. at OpenAI published the first comprehensive empirical study of Transformer scaling dynamics. When performance is not bottlenecked by dataset size $D$ or parameter count $N$, test cross-entropy loss $L$ adheres strictly to continuous power-law relations:
$$L(N) = left( frac{N_c}{N} right)^{alpha_N}, quad L(D) = left( frac{D_c}{D} right)^{alpha_D}, quad L(C) = left( frac{C_c}{C} right)^{alpha_C}$$
where $C approx 6 N D$ estimates total floating-point operations (FLOPs) required to train a dense model with $N$ non-embedding parameters across $D$ tokens ($2 text{ FLOPs/param}$ on the forward pass, $4 text{ FLOPs/param}$ on the backward pass).
Kaplan et al. fitted the empirical exponents as $alpha_N approx 0.076$, $alpha_D approx 0.096$, and concluded that models should scale parameters far more aggressively than training tokens:
$$N_{text{opt}} propto C^{0.73}, quad D_{text{opt}} propto C^{0.27}$$
This empirical law led to the design of first-generation massive foundation models: GPT-3 (175 billion parameters trained on only 300 billion tokens), Gopher (280B on 300B tokens), and MT-NLG (530B on 270B tokens).
2. The Chinchilla Paradigm Shift (DeepMind, 2022)
In 2022, Jordan Hoffmann et al. at Google DeepMind revisited Kaplan’s scaling formulations, identifying a fatal methodological flaw: Kaplan had maintained a fixed learning rate cosine decay schedule while varying compute budgets, severely disadvantaging larger token runs.
Hoffmann et al. fitted an explicit parametric loss envelope over 400 experimental training runs spanning $10^{18}$ to $10^{24}$ FLOPs:
$$L(N, D) = E + frac{A}{N^alpha} + frac{B}{D^beta}$$
where $E = 1.69$ is the irreducible entropy of the English language, $A = 406.4$, $B = 410.7$, $alpha = 0.34$, and $beta = 0.28$.
Minimizing $L(N, D)$ subject to total compute constraint $C = 6 N D$ via the method of Lagrange multipliers:
$$mathcal{L}(N, D, lambda) = E + A N^{-alpha} + B D^{-beta} + lambda (6 N D – C)$$
Evaluating partial derivatives $frac{partial mathcal{L}}{partial N} = -alpha A N^{-alpha-1} + 6 lambda D = 0$ and $frac{partial mathcal{L}}{partial D} = -beta B D^{-beta-1} + 6 lambda N = 0$ yields the profound Chinchilla Scaling Law:
$$N_{text{opt}} propto C^{frac{beta}{alpha + beta}} approx C^{0.5}, quad D_{text{opt}} propto C^{frac{alpha}{alpha + beta}} approx C^{0.5}$$
3. The Chinchilla Rule of Thumb: 1:1 Parity
Because $alpha approx beta approx 0.3$, parameters and tokens should scale in exact 1:1 parity:
$$frac{D_{text{opt}}}{N_{text{opt}}} approx frac{G}{alpha / beta} approx 20$$
For compute-optimal pretraining, a model must be trained on approximately 20 tokens per parameter.
DeepMind demonstrated this by training Chinchilla (70 billion parameters on 1.4 trillion tokens). Despite being less than half the size of GPT-3 (175B), Chinchilla crushed GPT-3, Gopher, and MT-NLG across every downstream linguistic, mathematical, and reasoning benchmark while consuming half the inference memory.
4. Inference-Optimal Over-Training: LLaMA and Modern Scale
Chinchilla compute-optimality minimizes total FLOPs consumed during pretraining. In real-world enterprise deployments, however, models serve billions of inference requests over months. A smaller model trained far beyond the Chinchilla frontier achieves lower per-token serving latency and lower HBM memory footprints.
In 2023, Meta introduced LLaMA (Touvron et al.), pioneering inference-optimal over-training: training a 7B model on 1.0T tokens ($140text{ tokens/param}$) and a 70B model on 2.0T tokens ($30text{ tokens/param}$), proving that loss continues to decline smoothly far past Chinchilla boundaries without saturation.
