In complex multistep deduction—such as formal mathematics, cryptography, and complex system engineering—a single invalid assumption at step 3 corrupts the entirety of a thirty-step proof. Yet traditional reinforcement learning evaluates models purely on the final output: Outcome-Supervised Reward Models (ORMs). Process Supervision and Process Reward Models (PRMs) revolutionize cognitive alignment by delivering dense, step-level supervision, transforming reasoning from an unguided lottery into an exact search over mathematical thoughts.
1. The Pathologies of Outcome Supervision (ORMs)
An Outcome Reward Model evaluates only the terminal state $y_T$:
r_{text{ORM}} = mathcal{R}(x, y_T) in {0, 1}
While simple to collect (checking whether a math answer matches “42” or a code script passes a test), ORMs suffer from fatal failure modes:
- Credit Assignment Dilution: In a 1,000-token derivation, which specific step was brilliant, and which was flawed? The gradient is blurred across hundreds of intermediate tokens.
- Rewarding Flawed Heuristics: If an agent makes two consecutive arithmetic errors that cancel each other out ($+5$ followed by $-5$), the final answer happens to be correct. An ORM provides positive reinforcement, actively rewarding hallucinatory and mathematically invalid reasoning circuits.
2. Process Reward Models (PRMs): Dense Step-Wise Feedback
Introduced by Lightman et al. (2023) at OpenAI, Process Supervision scores every discrete step $k$ in a reasoning trajectory:
mathcal{L}_{text{PRM}}(phi) = – sum_{k=1}^K left[ y_k^* log r_phi(s_k) + (1 – y_k^*) log(1 – r_phi(s_k)) right]
Where $s_k = [x, text{step}_1, dots, text{step}_k]$ and $y_k^* in {+1, -1}$ denotes whether human mathematicians or automated formal verifiers certify step $k$ as mathematically sound.
Dense step-level feedback eliminates credit assignment ambiguity: if step 4 contains a logical fallacy, $r_phi(s_4) to 0$, providing an immediate, localized error signal that penalizes the specific attention projection that produced the error.
3. Search Over Thoughts: Guided Reasoning Trees
Equipped with a trained PRM $r_phi$, the model no longer samples reasoning chains linearly; it performs Search Over Thoughts:
- Step-Level Best-of-$N$: At each step $k$, generate $N$ candidate next-step thoughts. Evaluate all $N$ branches with the PRM and select the highest-scoring step before continuing.
- Beam Search & Monte Carlo Tree Search: Maintaining a frontier of promising reasoning branches. If a branch score drops below threshold $tau_{text{prune}}$, the entire branch is pruned immediately, conserving compute and redirecting search toward verified sub-proofs.
Process supervision triples search efficiency over raw outcome voting, unlocking the computational engine behind modern Olympian-level AI theorem provers.
