By late 2022, raw pretrained foundation models could mimic virtually any distribution of human text and imagery. However, raw models were frequently unaligned—prone to catastrophic hallucinations, adversarial jailbreaks, toxic completions, and sycophantic mimicry. The breakthrough that ignited the global consumer AI explosion was Alignment: engineering methods to steer base probability distributions toward helpfulness, truthfulness, and rigorous multi-step deliberation.
1. Aligning Pretrained Intelligence: RLHF
To transform an unconstrained next-token completion engine into a cooperative assistant, researchers turned to Reinforcement Learning from Human Feedback (RLHF). The canonical RLHF pipeline consists of three sequential phases:
- Supervised Fine-Tuning (SFT): Training the pretrained base model on high-quality demonstration dialogues.
- Reward Model Training: Training a scalar reward function $r_phi(x, y)$ on human pairwise preference judgments using the Bradley-Terry comparison model:
$$P(y_1 succ y_2 mid x) = sigma(r_phi(x, y_1) – r_phi(x, y_2))$$
- Policy Optimization via PPO: Optimizing the generative policy $pi_theta$ using Proximal Policy Optimization against the learned reward model while penalizing drift via a Kullback-Leibler (KL) divergence penalty against the reference model $pi_{text{ref}}$:
$$max_theta mathbb{E}_{(x, y) sim mathcal{D}}left[r_phi(x, y)right] – beta D_{KL}left(pi_theta(y mid x) parallel pi_{text{ref}}(y mid x)right)$$
The KL divergence penalty $beta D_{KL}$ ensures the policy does not exploit imperfections in the reward model (preventing reward hacking). This exact alignment architecture powered ChatGPT in November 2022, transforming the research landscape into a global societal inflection point.
2. Direct Preference Optimization (DPO)
While PPO yielded robust models, training a separate reward model and managing multiple concurrent networks in GPU memory introduced substantial instability. In 2023, Rafailov et al. introduced Direct Preference Optimization (DPO), mathematically proving that the constrained RL objective can be solved analytically. Under the optimal policy $pi^*$, the implicit reward can be expressed directly in terms of the language model probabilities:
$$r(x, y) = beta log frac{pi_theta(y mid x)}{pi_{text{ref}}(y mid x)} + beta log Z(x)$$
Substituting this directly into the Bradley-Terry objective yields a simple, stable binary cross-entropy loss over preferred ($y_w$) and dispreferred ($y_l$) completions without requiring a separate reward model or dynamic reinforcement learning loop:
$$mathcal{L}_{text{DPO}}(theta; pi_{text{ref}}) = -mathbb{E}_{(x, y_w, y_l)}left[log sigmaleft(beta log frac{pi_theta(y_w mid x)}{pi_{text{ref}}(y_w mid x)} – beta log frac{pi_theta(y_l mid x)}{pi_{text{ref}}(y_l mid x)}right)right]$$
3. Diffusion Models & The Visual Creative Explosion
Parallel to language breakthroughs, visual synthesis experienced a profound revolution through Denoising Diffusion Probabilistic Models (DDPM) and Score-Based Generative Networks. Instead of adversarial zero-sum games (GANs), diffusion models formulate image generation as reversing a forward thermodynamic stochastic differential equation:
$$dx = f(x, t)dt + g(t)dw$$
By training neural networks to predict score functions $nabla_x log p_t(x)$ that iteratively strip Gaussian noise across continuous time, architectures like Stable Diffusion, Midjourney, and Flux achieved photorealistic generative mastery across high-dimensional artistic domains.
4. System 1 vs. System 2 Reasoning & Test-Time Compute Scaling
Traditional autoregressive generation operates analogously to Daniel Kahneman’s System 1 cognitive processing: fast, intuitive, and constant compute per token. For intricate logic, mathematics, and software architecture, System 1 repeatedly fails due to unrecoverable early token choices.
The paradigm shifted dramatically with Chain-of-Thought (CoT) prompting and subsequent algorithmic deliberation frameworks. By dedicating compute at inference time—generating extensive hidden chains of reasoning, backtracking, exploring search trees (Monte Carlo Tree Search), and verifying candidate paths using automated verifiers—models enter System 2 deliberate reflection.
With the release of models like OpenAI o1 and o3, AI established a new fundamental scaling dimension: Test-Time Compute Scaling. Reinforcement learning applied directly to search and reasoning chains enables models to achieve superhuman performance on graduate-level mathematics, competitive programming, and formal scientific verification.
