Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
FreeScaling automated process verifiers for LLM reasoning via progress-aware rewards
FreeFree tier
About Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning
This paper introduces process advantage verifiers (PAVs) for improving reasoning in large language models. PAVs are trained to predict progress under a prover policy, measuring the change in likelihood of producing a correct response after each step. The method uses a prover policy distinct from the base policy, enabling weak prover policies to improve a stronger base policy. Empirical results show that test-time search against PAVs is 8% more accurate and 1.5-5x more compute-efficient than outcome reward models (ORMs), and online reinforcement learning with PAVs yields 5-6x gains in sample efficiency and 6% improvement in accuracy over ORMs.
Key Features
Process advantage verifiers (PAVs) that predict step-level progress under a prover policy
Theoretical characterization of good prover policies for process rewards
Demonstration that weak provers can improve strong base policies
8% accuracy improvement and 1.5-5x compute efficiency in test-time search vs. ORMs
5-6x sample efficiency gain and 6% accuracy gain in online RL with dense PAV rewards
Pros & Cons
Pros
- Better credit assignment than outcome reward models by giving step-level feedback
- Compute-efficient test-time search (1.5-5x improvement)
- Significant gains in sample efficiency for online RL (5-6x)
- Can improve a stronger base policy even when the prover is weaker
Best For
Improving multi-step reasoning in large language modelsCredit assignment for reinforcement learning from process rewardsTest-time search for reasoning tasksSample-efficient fine-tuning of LLMs with dense rewards