CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
FreeBoosting LLM reasoning generalization with critical plan step learning
FreeFree tier
About CPL: Critical Plan Step Learning Boosts LLM Generalization in Reasoning Tasks
CPL (Critical Plan Step Learning) is a post-training method for large language models (LLMs) that enhances generalization in multi-step reasoning tasks. It combines Monte Carlo Tree Search (MCTS) over high-level abstract plans with Step-level Advantage Preference Optimization (Step-APO), which integrates advantage estimates from MCTS into Direct Preference Optimization (DPO). This approach helps LLMs learn critical plan steps, improving both in-domain and out-of-domain reasoning. When trained solely on GSM8K and MATH datasets, CPL yields significant gains on those benchmarks and on out-of-domain tasks such as HumanEval, GPQA, ARC-C, MMLU-STEM, and BBH.
Key Features
Monte Carlo Tree Search (MCTS) over high-level abstract plans to explore diverse reasoning strategies
Step-level Advantage Preference Optimization (Step-APO) combining advantage estimates with Direct Preference Optimization (DPO)
Focus on learning critical plan steps for multi-step reasoning tasks
Trained exclusively on GSM8K and MATH datasets for in-domain improvement and out-of-domain generalization
Achieved +10.5% on GSM8K, +6.5% on MATH, +12.2% on HumanEval, +8.6% on GPQA, +4.0% on ARC-C, +2.2% on MMLU-STEM, +1.8% on BBH
Pros & Cons
Pros
- Significant performance gains on both training datasets (GSM8K, MATH) and multiple out-of-domain benchmarks
- Generalizes beyond the training domains, improving reasoning on HumanEval, GPQA, ARC-C, MMLU-STEM, and BBH
- Novel combination of MCTS search on plan space and step-level preference optimization for effective learning
Best For
Improving reasoning capabilities and generalization of LLMs across math, coding, science, and other reasoning tasksPost-training refinement of LLMs using self-play generated data and reinforcement learningEnhancing performance on out-of-domain reasoning benchmarks without task-specific fine-tuning
FAQ
What does CPL stand for?
CPL stands for Critical Plan Step Learning.
How does CPL improve LLM reasoning?
CPL uses Monte Carlo Tree Search to explore diverse plan steps in multi-step reasoning tasks and applies Step-level Advantage Preference Optimization (Step-APO) to learn critical plan steps, enhancing both reasoning and generalization.
What datasets was CPL trained on?
CPL was trained exclusively on GSM8K and MATH datasets.
Does CPL only work on math reasoning?
No, despite being trained only on math datasets, CPL improves performance on out-of-domain reasoning tasks such as HumanEval (coding), GPQA (general Q&A), ARC-C (science), MMLU-STEM (STEM), and BBH (big bench hard).