ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search

Free

LLM self-training via process reward guided tree search.

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search

ReST-MCTS* is a reinforced self-training approach for LLMs that integrates process reward guidance with tree search (MCTS*) to collect high-quality reasoning traces and per-step value estimates. It circumvents the need for per-step manual annotation by using tree-search-based reinforcement learning: given oracle final correct answers, it infers process rewards by estimating the probability that a step leads to the correct answer. These inferred rewards serve dual purposes: as value targets to refine the process reward model and to select high-quality traces for policy model self-training. The method achieves higher accuracy compared to Best-of-N and Tree-of-Thought within the same search budget, and continuously enhances language models over multiple iterations, outperforming other self-training algorithms such as ReST^EM and Self-Rewarding LM. The paper was accepted to NeurIPS 2024 and the code is publicly released.

Key Features

Integrates process reward guidance with Monte Carlo Tree Search (MCTS*)

Collects higher-quality reasoning traces and per-step value estimates

Trains both policy model and process reward model without manual annotation

Infers process rewards by estimating probability a step leads to correct answer

Achieves higher accuracy than Best-of-N and Tree-of-Thought under same search budget

Supports iterative self-training to continuously enhance LLM reasoning

Open-source code released

Pros & Cons

Pros

Higher accuracy compared to existing reasoning baselines like Best-of-N and Tree-of-Thought
Eliminates need for per-step manual annotation in process reward training
Continuously improves model performance across multiple self-training iterations
Provides both reasoning traces and value estimates for dual-purpose training
Code publicly available for replication and extension

Best For

Improving reasoning capabilities of large language modelsSelf-training LLMs on tasks requiring multi-step reasoningGenerating high-quality training data for fine-tuning without human annotationEnhancing performance on math, logic, and other complex reasoning benchmarks

FAQ

What is the main contribution of ReST-MCTS*?

It integrates process reward guidance with tree search (MCTS*) to automatically collect high-quality reasoning traces and per-step value estimates, enabling self-training of LLMs without manual annotation.

How does ReST-MCTS* infer process rewards without annotations?

Given oracle final correct answers, it estimates the probability that a step leads to the correct answer using tree-search-based reinforcement learning, and uses these estimates as process rewards.

What results were achieved with ReST-MCTS*?

The tree-search policy outperformed Best-of-N and Tree-of-Thought, and using its traces for self-training continuously improved three language models, surpassing self-training algorithms like ReST^EM and Self-Rewarding LM.