While Supervised Fine-Tuning teaches a model how to format answers, it cannot easily adjudicate between subtly nuanced, creative, or factual explanations. Expert annotators cannot reliably write millions of optimal responses, but they can effortlessly compare two candidate outputs and state which is superior. Reinforcement Learning from Human Feedback (RLHF) leverages pairwise human preferences to optimize language models against human values, balancing helpfulness, harmlessness, and honesty via Proximal Policy Optimization (PPO).
1. The Bradley-Terry Preference Model
Given prompt $x$ and two candidate model completions $(y_w, y_l)$, where human evaluators prefer $y_w$ (winner) over $y_l$ (loser), the probability of preference is modeled under the classical Bradley-Terry logistic formulation (1952):
P(y_w succ y_l mid x) = sigma(r_psi(x, y_w) – r_psi(x, y_l)) = frac{1}{1 + e^{-(r_psi(x, y_w) – r_psi(x, y_l))}}
Where $r_psi(x, y) in mathbb{R}$ is a scalar Reward Model parameterized by neural network weights $psi$. The reward model is trained via binary cross-entropy on pairwise preference datasets $mathcal{D}$:
mathcal{L}_{text{RM}}(psi) = – mathbb{E}_{(x, y_w, y_l) sim mathcal{D}} left[ log sigma(r_psi(x, y_w) – r_psi(x, y_l)) right]
2. Policy Optimization with KL Divergence Penalties
Equipped with a trained reward model $r_psi$, the language model policy $pi_theta$ is fine-tuned to maximize expected reward. However, unconstrained reinforcement learning quickly leads to Reward Hacking: the policy exploits quirks or blind spots in the reward model to generate nonsensical, sycophantic, or repetitive text that scores artificially high.
To tether the model to sensible language distributions, an explicit Kullback-Leibler (KL) divergence penalty is enforced between the policy $pi_theta$ and the frozen initial SFT reference policy $pi_{text{ref}}$:
max_theta mathbb{E}_{x sim mathcal{D}, y sim pi_theta(cdot mid x)} left[ r_psi(x, y) – beta mathbb{D}_{text{KL}}(pi_theta(y mid x) parallel pi_{text{ref}}(y mid x)) right]
The per-token composite reward signal delivered to the RL agent is:
R(x, y) = r_psi(x, y) – beta log frac{pi_theta(y mid x)}{pi_{text{ref}}(y mid x)}
Parameter $beta$ controls the elasticity of the anchor: a higher $beta$ prevents policy collapse and preserves output diversity.
3. Proximal Policy Optimization (PPO) in Text Generation
Language model generation is treated as an episodic Markov Decision Process (MDP) where:
- State $s_t$: The prompt concatenated with all generated tokens up to step $t-1$: $[x, y_1, dots, y_{t-1}]$.
- Action $a_t$: Selecting token $y_t$ from vocabulary $mathcal{V}$.
- Transition: Deterministic appending of token $y_t$ to the sequence.
To avoid destructive gradient steps, PPO clips the probability ratio $rho_t(theta) = frac{pi_theta(a_t mid s_t)}{pi_{theta_{text{old}}}(a_t mid s_t)}$:
mathcal{L}_{text{PPO}}(theta) = mathbb{E}_t left[ minleft( rho_t(theta) hat{A}_t, , text{clip}(rho_t(theta), 1-epsilon, 1+epsilon) hat{A}_t right) right]
Where $hat{A}_t$ is the Generalized Advantage Estimator (GAE) computed from a learned value function $V_phi(s)$. This complex orchestra of four concurrent neural networks (Actor $pi_theta$, Critic $V_phi$, Reference $pi_{text{ref}}$, and Reward $r_psi$) forms the foundational bedrock that brought ChatGPT and modern aligned models into existence.
