Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
FreeAutomatically verify and reinforce LLMs step-by-step for math reasoning.
FreeFree tier
About Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations
Math-Shepherd is a process-oriented math process reward model that assigns a reward score to each step of math problem solutions. Its training leverages automatically constructed process-wise supervision data, eliminating the heavy reliance on manual annotations. The model is evaluated in two scenarios: verification (reranking multiple outputs from large language models) and reinforcement learning (step-by-step Proximal Policy Optimization). When applied to Mistral-7B, Math-Shepherd improved accuracy on GSM8K from 77.9% to 84.1% via step-by-step PPO and to 89.1% via verification, and on MATH from 28.6% to 33.0% via PPO and to 43.5% via verification.
Key Features
Automatically constructed process-wise supervision data
Step-by-step reward scoring for math solutions
Supports verification via reranking multiple LLM outputs
Supports reinforcement learning with step-by-step PPO
Improves accuracy of open-source LLMs like Mistral-7B
Eliminates need for human annotation in process reward training
Pros & Cons
Pros
- Trainable without human annotations, reducing cost and effort
- Demonstrates significant accuracy improvements on GSM8K and MATH benchmarks
- Applicable to both verification and reinforcement learning scenarios
- Works with open-source LLMs like Mistral-7B
Best For
Reranking multiple solutions generated by LLMs for math problemsReinforcing LLMs with process-level rewards for improved math reasoningAutomatically evaluating step-wise correctness in math problem solvingEnhancing performance of open-source language models on mathematical tasks
FAQ
What is Math-Shepherd?
Math-Shepherd is an innovative process-oriented math process reward model that automatically assigns a reward score to each step of math problem solutions, trained without human annotations.
How does Math-Shepherd improve LLM performance?
It can be used for verification (reranking outputs) or reinforcement learning (step-by-step PPO). For example, it boosted Mistral-7B accuracy on GSM8K from 77.9% to 84.1% with PPO and to 89.1% with verification.