Planning In Natural Language Improves LLM Search For Code Generation

Free

LLM code generation search via natural language planning

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About Planning In Natural Language Improves LLM Search For Code Generation

PlanSearch is a novel search algorithm that improves LLM-based code generation by searching over natural language plans rather than directly over code solutions. It first generates a diverse set of observations about the problem, then uses these observations to construct candidate plans in natural language. By exploring a more diverse set of high-level solution strategies, PlanSearch mitigates the lack of output diversity that hinders standard repeated sampling. The algorithm achieves state-of-the-art results on LiveCodeBench (pass@200 = 77.0% with Claude 3.5 Sonnet), outperforming both the no-search baseline (pass@1 = 41.4%) and standard repeated sampling (pass@200 = 60.6%). PlanSearch also shows strong results on HumanEval+ and MBPP+.

Key Features

Generates diverse natural language observations about the problem

Constructs multiple candidate plans from observations

Searches over plans instead of code solutions to promote diversity

Provides a direct method to scale inference-time compute for code generation

Achieves state-of-the-art pass@200 on LiveCodeBench (77.0% with Claude 3.5 Sonnet)

Validated on HumanEval+, MBPP+, and LiveCodeBench

Open source code available on GitHub

Pros & Cons

Pros

Significantly increases diversity of generated solutions compared to standard repeated sampling
Provides consistent performance gains across different models and benchmarks
Able to predict search gains based on diversity of generated ideas
Open source and reproducible

Cons

Requires multiple LLM inference passes for plan generation and evaluation
Performance may be limited by the quality of the base LLM's planning ability
May not be necessary for simple code generation tasks

Best For

Improving code generation accuracy for competitive programming problemsEnhancing LLM-based software engineering toolsReducing repetitive failed code samples during inferenceResearch on inference-time compute scaling for LLMs

FAQ

What is PlanSearch?

PlanSearch is a search algorithm that improves LLM code generation by generating diverse natural language plans about the problem and then searching over those plans to find a better solution.

How does PlanSearch differ from standard repeated sampling?

Instead of repeatedly sampling code solutions directly, PlanSearch first generates diverse observations and plans in natural language, leading to more diverse solution strategies and better coverage of the search space.

What benchmarks were used to evaluate PlanSearch?

PlanSearch was evaluated on HumanEval+, MBPP+, and LiveCodeBench, a contamination-free benchmark for competitive coding.

What were the main results?

Using Claude 3.5 Sonnet, PlanSearch achieved a pass@200 of 77.0% on LiveCodeBench, outperforming the best score without search (pass@1 = 41.4%) and standard repeated sampling (pass@200 = 60.6%).