When is Tree Search Useful for LLM Planning? It Depends on the Discriminator

Free

Tree search for LLM planning requires >90% discriminator accuracy to outperform simpler methods.

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About When is Tree Search Useful for LLM Planning? It Depends on the Discriminator

This research paper, accepted at ACL 2024, investigates the conditions under which tree search is useful for large language model (LLM) planning within a language agent framework consisting of a generator, a discriminator, and a planning method. The authors compare iterative correction and tree search against simpler re-ranking across two multi-step tasks: text-to-SQL parsing and mathematical reasoning. Their key finding is that advanced planning methods require discriminators with at least 90% accuracy to yield significant improvements over re-ranking. Current LLM-based discriminators fail to meet this threshold, and tree search introduces a 10–20x slowdown without meaningful performance gains. The paper provides code and data for reproducibility, offering practical guidance for deploying planning methods in LLM agents.

Key Features

Language agent framework with generator, discriminator, and planning method (re-ranking, iterative correction, tree search)

Comprehensive analysis of how discriminator accuracy affects planning method performance

Experiments on two distinct multi-step tasks: text-to-SQL parsing and mathematical reasoning

Demonstrates that advanced planning methods (iterative correction, tree search) need at least 90% discriminator accuracy to improve over re-ranking

Shows current LLM-based discriminators are insufficient for achieving such accuracy

Finds tree search is 10-20 times slower than re-ranking and iterative correction with negligible performance gains

Code and data are publicly available for reproducibility

Pros & Cons

Pros

Provides systematic analysis of planning methods for LLMs, highlighting the critical role of discriminator accuracy
Offers practical insights for practitioners: simpler methods like re-ranking may be more efficient than tree search
Open-sourced code and data enable reproducibility and further research
Published in top-tier venue (ACL 2024), ensuring rigorous peer review

Cons

Tree search is 10-20 times slower than re-ranking and iterative correction, limiting real-world applicability
Current LLM-based discriminators cannot achieve the ≥90% accuracy needed for advanced planning methods to outperform re-ranking
Performance gains from tree search are often negligible compared to much simpler re-ranking
The study focuses on only two task domains (text-to-SQL and math reasoning), which may limit generalizability

Best For

Text-to-SQL parsingMathematical reasoningLLM planning in language agent frameworksEvaluating discriminator accuracy for multi-step problem solving

FAQ

What planning methods are compared in the paper?

The paper compares three planning methods within a language agent framework: re-ranking, iterative correction, and tree search.

What tasks were used for evaluation?

Two multi-step tasks were used: text-to-SQL parsing and mathematical reasoning.

What discriminator accuracy is needed for tree search to be useful?

Advanced planning methods like tree search require discriminators with at least 90% accuracy to achieve significant improvements over re-ranking.

Do current LLMs meet that accuracy requirement?

No, the paper finds that current LLM-based discriminators do not achieve the 90% accuracy needed for tree search to provide meaningful gains.

How does tree search compare in efficiency to other methods?

Tree search is 10-20 times slower than re-ranking and iterative correction, but yields negligible performance improvements under current conditions.

Is the code and data for this paper available?

Yes, the code and data are publicly available at the URL provided in the paper.