Generative Verifiers: Reward Modeling as Next-Token Prediction

Generative Verifiers (GenRM) proposes a novel approach to reward modeling for large language models (LLMs) by framing verification as a next-token prediction task. Unlike traditional discriminative verifiers that score candidate solutions, GenRM trains verifiers jointly on verification and solution generation using the standard next-token prediction objective. This method seamlessly integrates with instruction tuning, enables chain-of-thought reasoning, and leverages additional test-time compute via majority voting for improved verification accuracy. Experimental results demonstrate significant performance gains over discriminative verifiers, DPO verifiers, and LLM-as-a-Judge across multiple benchmarks, including algorithmic tasks (5% to 45.3%), GSM8K (73% to 93.4%), MATH (28% to 44.6% in easy-to-hard generalization), and MMLU abstract algebra (37.9% to 53.5%). The approach also scales favorably with model size and test-time compute, and training with synthetic verification rationales is shown to be effective at detecting subtle errors in math problems.