Neura Market
On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability logo

On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability

Free
FreeFree tier
Type
Open Source
Visit Website GitHub (40 stars)

About On The Planning Abilities of OpenAI's o1 Models: Feasibility, Optimality, and Generalizability

This research paper conducts a pilot study evaluating the planning capabilities of OpenAI's o1 models across several benchmark tasks, focusing on three key aspects: feasibility, optimality, and generalizability. Through empirical evaluations on constraint-heavy tasks (e.g., Barman, Tyreworld) and spatially complex environments (e.g., Termes, Floortile), the paper highlights o1-preview's strengths in self-evaluation and constraint-following while identifying bottlenecks in decision-making and memory management, particularly in tasks requiring robust spatial reasoning. The results show that o1-preview outperforms GPT-4 in adhering to task constraints and managing state transitions, but often generates suboptimal solutions with redundant actions and struggles to generalize effectively in spatially complex tasks. Code for the experiments is provided.

Key Features

Evaluates feasibility, optimality, and generalizability of planning in o1 models
Uses constraint-heavy benchmarks (Barman, Tyreworld) and spatially complex environments (Termes, Floortile)
Highlights o1-preview's strengths in self-evaluation and constraint-following
Identifies bottlenecks in decision-making and memory management
Provides code for reproducibility

Pros & Cons

Pros
  • Outperforms GPT-4 in adhering to task constraints and managing state transitions in structured environments
  • Demonstrates strong self-evaluation and constraint-following abilities
  • Provides open-source code and evaluations for reproducibility
  • Offers foundational insights into planning limitations of LLMs
Cons
  • Often generates suboptimal solutions with redundant actions
  • Struggles to generalize effectively in spatially complex tasks
  • Bottlenecks in decision-making and memory management
  • Pilot study with limited scope; more research needed

Best For

Research on large language model planning capabilitiesBenchmarking and comparing planning performance of o1 models against GPT-4Identifying directions for improving memory, decision-making, and generalization in LLM-based planning

FAQ

What planning aspects does the paper evaluate?
The paper evaluates feasibility, optimality, and generalizability of planning using OpenAI's o1 models across constraint-heavy and spatially complex benchmark tasks.
What benchmarks are used?
Constraint-heavy tasks include Barman and Tyreworld; spatially complex environments include Termes and Floortile.
How does o1-preview compare to GPT-4?
o1-preview outperforms GPT-4 in adhering to task constraints and managing state transitions, but generates suboptimal solutions with redundant actions.
Is the code available?
Yes, code is available at the URL provided in the paper.