Process Reward Models That Think
FreeData-efficient process reward models with verification chain-of-thought
FreeFree tier
About Process Reward Models That Think
ThinkPRM is a generative process reward model (PRM) that verifies each step in a solution by generating a verification chain-of-thought (CoT). It is fine-tuned from long CoT models using orders of magnitude fewer process labels than discriminative PRMs, achieving state-of-the-art results on math reasoning benchmarks like ProcessBench, MATH-500, and AIME 2024, and showing strong out-of-domain generalization on GPQA-Diamond and LiveCodeBench. The work highlights the value of scalable test-time computation for verification with minimal supervision. Code, data, and models are open-sourced.
Key Features
Generates a verification chain-of-thought (CoT) for each step in a solution
Fine-tuned on only 1% of process labels required by discriminative PRMs (orders of magnitude fewer)
Outperforms discriminative verifiers on ProcessBench, MATH-500, and AIME '24 under best-of-N and reward-guided search
Scales verification compute more effectively than LLM-as-a-Judge, with 7.2% improvement on ProcessBench subset
Strong out-of-domain generalization on GPQA-Diamond and LiveCodeBench (+8% and +4.5% versus full PRM800K)
Open-source release of code, data, and trained models
Pros & Cons
Pros
- Extremely data-efficient: uses only 1% of process labels from PRM800K
- Outperforms both LLM-as-a-Judge and discriminative PRMs on multiple benchmarks
- Leverages inherent reasoning abilities of long CoT models
- Open-source and permissively licensed (MIT-like via arXiv licensing)
- Demonstrates effective test-time compute scaling for verification
Cons
- Requires fine-tuning of a long CoT model, which may be computationally expensive
- Evaluation focused primarily on math and coding tasks; general domain performance not thoroughly tested
- Generative verification CoT may introduce additional latency compared to discriminative classifiers
- Dependency on base long CoT model (not specified if standalone)
Best For
Step-by-step verification for math reasoning problems (e.g., MATH-500, AIME competitions)Process-level reward modeling for test-time scaling of large language modelsImproving correctness of generated solutions via best-of-N selectionReward-guided search (e.g., beam search or tree search) for multi-step reasoning
FAQ
What is ThinkPRM?
ThinkPRM is a process reward model that verifies each step in a solution by generating a chain-of-thought (CoT) explanation. It is fine-tuned from long CoT models and requires far fewer process labels than traditional discriminative PRMs.
How does ThinkPRM compare to LLM-as-a-Judge?
Under the same token budget, ThinkPRM scales verification compute more effectively and outperforms LLM-as-a-Judge by 7.2% on a subset of ProcessBench.
Is ThinkPRM open source?
Yes, the code, data, and models are released. The paper is on arXiv and the repository provides access to all artifacts.
What benchmarks were used to evaluate ThinkPRM?
ThinkPRM was evaluated on ProcessBench, MATH-500, AIME '24, GPQA-Diamond, and LiveCodeBench, showing improvements over baselines.
How many process labels does ThinkPRM require?
It uses only 1% of the process labels in the PRM800K dataset, making it highly data-efficient.