Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
FreeEnhancing LLM reasoning via MCTS and iterative preference learning
About Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning
This paper introduces a method to enhance the reasoning capabilities of Large Language Models (LLMs) by combining Monte Carlo Tree Search (MCTS) with iterative preference learning, inspired by AlphaZero. The approach uses MCTS's look-ahead ability to collect step-level preference data, breaking down instance-level rewards into finer granularity. It combines outcome validation with stepwise self-evaluation to improve intermediate step consistency. The policy is updated via Direct Preference Optimization (DPO) using the generated step-level preference data, with a theoretical justification for using on-policy sampling. Extensive evaluations on arithmetic and commonsense reasoning tasks (GSM8K, MATH, ARC-C) show significant accuracy improvements over supervised fine-tuning baselines (e.g., +5.9% on GSM8K, +5.8% on MATH, +15.8% on ARC-C with Mistral-7B). The paper also analyzes the trade-off between training and inference compute. The code is publicly available.
Key Features
Pros & Cons
- Achieves significant accuracy improvements on multiple reasoning benchmarks
- Uses fine-grained step-level preference signals for better learning
- Combines outcome validation with self-evaluation for consistency
- Open-source code available
- May require substantial computational resources due to MCTS look-ahead
- Effectiveness depends on careful tuning of MCTS and preference learning parameters