WizardMath

Analysis

Why This Paper Matters

Mathematical reasoning remains a critical benchmark for large language models, yet open-source models have historically lagged behind proprietary systems like GPT-4 and PaLM. WizardMath tackles this gap head-on by introducing Reinforcement Learning from Evol-Instruct Feedback (RLEIF), a method that combines instruction evolution with process-supervised reinforcement learning. The paper shows that a 13B model can outperform PaLM 540B on GSM8k, and a 70B model rivals ChatGPT and Claude Instant. This is a significant step toward democratizing advanced math reasoning capabilities.

How It Works

WizardMath builds on the Evol-Instruct paradigm from WizardLM and the process reward model (PRM) concept. The RLEIF pipeline consists of three stages:

1. Supervised Fine-Tuning (SFT): The base Llama-2 model is fine-tuned on a curated dataset of 15k step-by-step solutions for GSM8k and MATH, re-generated using a WizardLM 70B alpha model to ensure correct answers. Additionally, 1.5k open-domain conversations from WizardLM are mixed in to improve instruction following.

2. Reward Model Training: Two separate reward models are trained:

   • Instruction Reward Model (IRM): Evaluates the quality of evolved instructions based on definition, precision, and integrity. Training data is generated by having ChatGPT and Wizard-E produce multiple evolved versions of each instruction, then ranked by Wizard-E.
   • Process-supervised Reward Model (PRM): Assesses the correctness of each step in the model's solutions. Since no powerful open-source math model existed, ChatGPT provides step-level supervision.

3. PPO Training with Evol-Instruct: The original math instructions are evolved through 8 turns of upward and downward evolution, expanding the dataset from 15k to 96k. During PPO, the final reward is the product of the instruction reward (rI) and the answer reward (rA). Wizard-E, a fine-tuned Llama model, performs the evolution without external APIs.

The evolution process includes:

• Downward evolution: Simplifies instructions by reducing difficulty or creating easier variants on different topics.
• Upward evolution: Generates harder questions by adding constraints, concretizing, or increasing reasoning steps.

Results

• WizardMath 13B achieves 63.9% on GSM8k, surpassing PaLM 540B (56.5%), Minerva 540B (58.8%), and GPT-3.5 (57.1%). On MATH, it scores 14.0%, beating PaLM 540B (8.8%) and GPT-3 175B (5.2%).
• WizardMath 70B reaches 81.6% on GSM8k, matching or exceeding Claude Instant (80.9%), ChatGPT (80.8%), and PaLM 2 (80.7%). On MATH, it scores 22.7%, outperforming Text-davinci-002 (19.1%).
• WizardMath 7B outperforms most open-source models in the 7B–40B range, including MPT, Falcon, Vicuna, and ChatGLM 2.
• Compared to Llama-2 70B, WizardMath 70B improves by 24.8% on GSM8k and 9.2% on MATH.

Significance

WizardMath demonstrates that careful integration of instruction evolution and process-supervised reinforcement learning can dramatically boost mathematical reasoning in open-source LLMs. The method is data-efficient (starting from only 15k examples) and does not require proprietary APIs for evolution after training Wizard-E. This work sets a new bar for open-source math models and provides a practical blueprint for applying RLEIF to other reasoning domains. The public release of code and weights further accelerates community progress.