Reasoning with Language Model is Planning with World Model
FreeRAP: Reasoning via Planning with World Models
FreeFree tier
Inputs: textOutputs: text
About Reasoning with Language Model is Planning with World Model
RAP (Reasoning via Planning) is a framework that enhances large language models (LLMs) by repurposing them as both a world model and a reasoning agent. It integrates Monte Carlo Tree Search (MCTS) for strategic planning, enabling LLMs to explore alternative reasoning paths, anticipate future states and rewards, and iteratively refine reasoning steps. The framework addresses the lack of an internal world model in LLMs, improving performance on plan generation, math reasoning, and logical inference tasks. Evaluated on LLAMA-33B, RAP achieved a 33% relative improvement over Chain-of-Thought (CoT) prompting with GPT-4 in plan generation. Code is available, and the paper was published at EMNLP 2023.
Key Features
Repurposes LLM as both a world model and a reasoning agent
Incorporates Monte Carlo Tree Search (MCTS) for strategic exploration
Balances exploration vs. exploitation to find high-reward reasoning paths
Incrementally builds a reasoning tree guided by task-specific rewards
Improves performance on plan generation, math reasoning, and logical inference
Code is publicly available
Pros & Cons
Pros
- Outperforms Chain-of-Thought and least-to-most prompting with self-consistency
- Enables deliberate planning akin to human brain via exploration of reasoning paths
- Works with various LLMs (e.g., LLAMA-33B shown in paper)
- Provides a principled planning algorithm for strategic exploration
- Open source with available code
Cons
- May be computationally more intensive than standard prompting methods due to MCTS
- Requires task-specific reward design for optimal performance
- Limited empirical validation beyond the reported tasks (plan generation, math, logic)
- Dependence on LLM quality as world model may introduce biases
Best For
Plan generation for executing tasks in a given environmentComplex math reasoningLogical and commonsense reasoningAny problem requiring multi-step reasoning with state tracking
FAQ
What is RAP?
RAP (Reasoning via Planning) is a framework that enhances LLMs by using them as both a world model and a reasoning agent, combined with Monte Carlo Tree Search for planning.
How does RAP work?
RAP builds a reasoning tree incrementally, where the LLM acts as an agent and a world model. The agent explores reasoning paths under guidance from the world model and task-specific rewards, balancing exploration and exploitation to find high-reward paths.
What tasks can RAP be applied to?
RAP is demonstrated on plan generation, math reasoning, and logical inference tasks, but can be applied to any problem requiring multi-step reasoning with state tracking.
How does RAP compare to Chain-of-Thought?
RAP outperforms Chain-of-Thought (CoT) prompting across tasks. For example, RAP on LLAMA-33B achieved a 33% relative improvement over CoT with GPT-4 in a plan generation setting.
Is the code available?
Yes, the code is publicly available at the URL provided in the paper.