Structural Planning for LLM Agents

Analysis

Why This Paper Matters

Enterprise deployment of LLM agents often fails because general-purpose models lack domain-specific process knowledge, leading to disorganized plans and unstable tool calls. Routine addresses this by introducing a structured planning framework that imposes explicit structure and parameter passing, making agent execution more reliable. The paper's results are striking: a simple planning wrapper boosts GPT-4o's tool-calling accuracy from 41.1% to 96.3%, and even a smaller model like Qwen3-14B jumps from 32.6% to 83.3%. This suggests that many agent failures stem from poor planning rather than model capability, and that structured prompting can unlock significant performance gains.

The work also demonstrates a path to distilling domain-specific tool-usage patterns into smaller models. By creating a Routine-following dataset and fine-tuning Qwen3-14B, the authors achieve 88.2% accuracy, and with distillation-based fine-tuning, 95.5%—approaching GPT-4o's performance. This is particularly relevant for enterprises that need cost-effective, on-premise solutions without sacrificing reliability.

Technical Contributions

• Structured Planning Framework: Routine provides a clear, multi-step plan with explicit instructions and parameter passing, guiding the agent's execution module to perform tool calls in a stable manner.
• Training Dataset Construction: The authors build a Routine-following dataset to fine-tune models, improving their adherence to execution plans.
• Distillation Methodology: They use Routine to generate a scenario-specific, multi-step tool-calling dataset, which is then used to fine-tune a smaller model, effectively transferring domain knowledge.
• Real-World Evaluation: The framework is tested in an actual enterprise scenario, providing practical evidence of its effectiveness.

Results

• GPT-4o accuracy improved from 41.1% to 96.3% with Routine.
• Qwen3-14B accuracy improved from 32.6% to 83.3% with Routine.
• Fine-tuning Qwen3-14B on Routine-following data increased accuracy to 88.2%.
• Distillation-based fine-tuning further raised accuracy to 95.5%, nearly matching GPT-4o.

These metrics highlight Routine's ability to significantly enhance tool-calling accuracy, especially for smaller models, making them viable for enterprise use.

Significance

Routine offers a practical, low-cost method to improve agent reliability without requiring new model architectures. By structuring planning and enabling distillation, it allows enterprises to deploy smaller, more efficient models that perform nearly as well as large proprietary ones. This could accelerate the adoption of agent systems in industry and inspire further research into process-aware planning and knowledge distillation for tool use. The paper also contributes to the vision of 'AI for Process' by formalizing how to encode domain processes into agent workflows.