V-STaR: Training Verifiers for Self-Taught Reasoners

Free

Train verifiers from self-generated correct and incorrect solutions to boost LLM reasoning.

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About V-STaR: Training Verifiers for Self-Taught Reasoners

V-STaR is a method for improving large language model (LLM) reasoning through iterative self-improvement and verifier training. It builds on the STaR (Self-Taught Reasoner) approach by not only fine-tuning on self-generated correct solutions but also leveraging incorrect solutions to train a verifier using Direct Preference Optimization (DPO). This verifier evaluates the correctness of candidate solutions at inference time, selecting the best among many. The method runs multiple iterations, progressively improving both the reasoner and the verifier. Evaluated on code generation and math reasoning benchmarks with LLaMA2 models, V-STaR achieves 4% to 17% test accuracy improvements over existing self-improvement and verification approaches.

Key Features

Iterative self-improvement for LLMs

Trains a verifier using Direct Preference Optimization (DPO) on both correct and incorrect self-generated solutions

Selects best solution among many candidates at inference time

Progressively improves reasoner and verifier over multiple iterations

Supports code generation and math reasoning tasks

Pros & Cons

Pros

Utilizes both correct and incorrect solutions, extracting more value from the self-improvement process
Simple to integrate into existing STaR-like pipelines
Demonstrated significant accuracy improvements (4-17%) on standard benchmarks
Verifier helps filter out poor solutions at inference time
Iterative nature leads to progressively better performance

Cons

Requires multiple training iterations, which may be computationally expensive
Performance gains depend on the quality and diversity of self-generated solutions
Currently validated only on LLaMA2 models, generalization to other LLMs not confirmed
May need careful hyperparameter tuning for DPO training

Best For

Improving reasoning abilities of large language modelsCode generation tasks requiring multiple candidate solutionsMathematical reasoning and problem solving with LLMsSelf-improvement frameworks for LLMsResearch in verification methods for LLM outputs

FAQ

What is V-STaR?

V-STaR is a method for training a verifier using both correct and incorrect solutions generated during LLM self-improvement, improving reasoning performance over standard STaR.

How does V-STaR differ from STaR?

STaR only fine-tunes on correct self-generated solutions, discarding incorrect ones. V-STaR also leverages incorrect solutions to train a verifier via DPO, which selects the best solution at inference time.

What benchmarks were used to evaluate V-STaR?

V-STaR was evaluated on common code generation and math reasoning benchmarks using LLaMA2 models.

What is the reported accuracy improvement?

V-STaR delivers 4% to 17% test accuracy improvement over existing self-improvement and verification approaches.

Is V-STaR related to reinforcement learning?

Yes, it uses Direct Preference Optimization (DPO), a reinforcement learning from human feedback (RLHF) method, to train the verifier.