Let's Verify Step by Step
FreeProcess supervision outperforms outcome supervision for math reasoning
FreeFree tier
About Let's Verify Step by Step
Let's Verify Step by Step presents a method for training large language models using process supervision, which provides feedback for each intermediate reasoning step, as opposed to outcome supervision that only gives feedback for the final result. The authors show that process supervision significantly outperforms outcome supervision on the challenging MATH dataset, achieving 78% accuracy on a representative subset of the test set. They also demonstrate that active learning improves the efficacy of process supervision. To support further research, the paper releases PRM800K, a dataset of 800,000 step-level human feedback labels used to train the best reward model.
Key Features
Process supervision provides feedback for each intermediate reasoning step
Outcome supervision only provides feedback for the final result
Process-supervised model achieves 78% accuracy on MATH dataset subset
Active learning improves the efficacy of process supervision
Release of PRM800K: 800,000 step-level human feedback labels
Pros & Cons
Pros
- Process supervision significantly improves performance over outcome supervision
- Active learning makes process supervision more efficient
- Publicly released dataset (PRM800K) enables further research
Cons
- Requires expensive human feedback for each intermediate step
- Evaluation limited to the MATH dataset, generalizability unclear
- Method may not scale to tasks with ambiguous or open-ended reasoning
Best For
Training more reliable large language models for complex multi-step reasoningImproving mathematical problem-solving capabilities of LLMsResearch on process vs outcome supervision in reinforcement learning from human feedback
FAQ
What is the main contribution of this paper?
The paper shows that process supervision (feedback per reasoning step) significantly outperforms outcome supervision (feedback only on final result) for training LLMs on math problems, and releases the PRM800K dataset of 800k step-level human labels.
What dataset was used for evaluation?
The authors evaluated on the MATH dataset, specifically a representative subset of the test set.
What is PRM800K?
PRM800K is a dataset of 800,000 step-level human feedback labels used to train a process reward model.
Does the paper release code or data?
The paper releases the PRM800K dataset to support related research.