LLMs Still Can’t Plan

Analysis

Why This Paper Matters

This paper is significant because it directly addresses the long-standing question of whether large language models can plan. Planning is a core competence of intelligent agents, and the authors have been tracking this via PlanBench since 2022. The release of OpenAI's o1 model, marketed as a Large Reasoning Model, provides a timely catalyst to reassess the state of planning abilities. The paper's findings are sobering: despite claims and improvements, LLMs still cannot plan reliably. This matters for AI practitioners who may be tempted to use LLMs for tasks requiring sequential decision-making, as it highlights the gap between language understanding and true planning.

The paper also serves as a reality check for the AI community. With the rapid advancement of LLMs, there is a tendency to overstate their capabilities. By systematically evaluating on a benchmark designed to test planning, the authors provide concrete evidence that progress is slow and that even the best models fall short. This is crucial for setting realistic expectations and guiding research toward addressing fundamental limitations.

Technical Contributions

The paper's main technical contribution is the comprehensive evaluation of a wide range of LLMs and LRMs on PlanBench. Key aspects include:

• PlanBench as a benchmark: The authors use their previously developed PlanBench, which is extensible and designed to test planning abilities in a controlled manner.
• Inclusion of o1: They specifically evaluate OpenAI's o1 model, which is a new type of model (LRM) built to escape autoregressive limitations.
• Comparative analysis: They compare o1 against other private and open-source LLMs, providing a clear picture of the current landscape.
• Focus on accuracy, efficiency, and guarantees: The paper goes beyond raw performance to discuss practical considerations for deployment, such as whether the model's outputs are reliable and efficient.

Results

The paper reports that o1's performance on PlanBench is a "quantum improvement" over other models, outpacing the competition significantly. However, it is still far from saturating the benchmark, meaning that even the best model cannot solve all planning tasks. The authors note that progress on PlanBench has been surprisingly slow since GPT-3, despite the influx of new models. This suggests that planning remains a hard problem for LLMs, and the improvements seen in o1 may not generalize to all planning scenarios. The paper does not provide specific numeric results in the abstract, but the qualitative findings are clear.

Significance

The broader impact of this work is twofold. First, it provides a rigorous benchmark for evaluating planning abilities in LLMs, which can guide future research and development. Second, it serves as a cautionary note for deploying LLMs in planning-critical applications, emphasizing the need for guarantees and efficiency. The paper also contributes to the ongoing debate about the nature of reasoning in LLMs, suggesting that even with specialized training, these models do not yet exhibit robust planning capabilities. This has implications for AI safety and reliability, as planning is essential for autonomous agents. The findings encourage the community to continue exploring hybrid approaches that combine LLMs with classical planning methods.