V-STaR

Analysis

Why This Paper Matters

Large language models (LLMs) often struggle with multi-step reasoning tasks like math problem solving and code generation. While self-improvement methods like STaR (Self-Taught Reasoner) boost performance by training on correct self-generated solutions, they discard incorrect ones—missing a valuable signal. V-STaR (Verification for Self-Taught Reasoners) flips this by using both correct and incorrect solutions to train a verifier via Direct Preference Optimization (DPO). This verifier then selects the best answer among multiple candidates at inference time, leading to substantial gains on math and code benchmarks. The paper shows that learning from mistakes is as important as learning from successes, and that DPO-based verifiers outperform traditional reward models.

How It Works

The core idea is an iterative self-improvement loop. First, a pretrained LLM (e.g., LLaMA2) is fine-tuned on the original training data to create a generator (GSFT). Then, for each problem in the training set, the generator produces multiple candidate solutions (e.g., 16 per query). Each solution is labeled as correct or incorrect using ground truth answers (for math) or test cases (for code).

Correct solutions are added to the generator's training data (DGEN). Both correct and incorrect solutions are used to build a preference dataset (DVER) for the verifier: each pair consists of a correct (preferred) and an incorrect (not preferred) solution for the same problem. The verifier is trained using DPO, which unifies language modeling and binary classification objectives. The reference policy for DPO is the SFT model (GSFT). This process repeats for multiple iterations (up to T), with the generator being fine-tuned on the accumulated DGEN and the verifier on DVER.

At inference, the final generator produces multiple candidate solutions, and the verifier scores each one. The solution with the highest verifier score is selected (best-of-k).

Results

V-STaR consistently outperforms baselines like STaR (which only uses correct solutions) and verification-only methods. On GSM8K (math) and MBPP (code), 7B and 13B models achieve 6-17% absolute accuracy improvements on math and 4-12% on code. The iterative training (multiple rounds) yields better results than a single iteration. Out-of-domain transfer also improves: training on GSM8K boosts performance on a MATH subset, and training on MBPP helps on HumanEval.

DPO-based verifiers outperform ORM-style verifiers, especially when considering more candidate solutions (best-of-k). Best-of-k accuracy saturates around k=16, meaning additional candidates provide diminishing returns.

V-STaR also beats majority voting (self-consistency) when searching over many candidates. The verifier's generation ability degrades after a few updates, so it is not used as a generator. Including the verifier in the training loop to filter solutions did not help on MBPP.

Significance

V-STaR offers a practical, data-efficient way to improve LLM reasoning without external supervision beyond initial labels. By leveraging both correct and incorrect self-generated solutions, it turns model failures into training signals. The use of DPO for verifier training is a key innovation, unifying language modeling and reward modeling in a single objective. This approach could reduce the need for human annotation in fine-tuning and make LLMs more reliable in reasoning tasks. The code and models are open-source, enabling further research and application.