Prompt-Induced Waste

Analysis

Why This Paper Matters

This paper addresses a critical yet underappreciated aspect of AI agent deployment: the cost of prompt wording. While prior work has focused on accuracy or success rates, this study shows that even when two prompts produce the same correct patch, the agent's internal and external resource consumption can differ by an order of magnitude. This is particularly relevant as coding agents become mainstream in software development, where token costs, latency, and tool usage directly impact operational budgets.

The findings reveal that prompt engineering is not just about getting the right answer—it is about designing the agent's work process. The paper identifies two distinct waste mechanisms: one that inflates reasoning (token-borne) and another that inflates tool usage (system-borne). This distinction is crucial because it means that optimizing prompts requires understanding which resource is more constrained in a given deployment. The study's robustness checks (holdout, paraphrases, replication) strengthen confidence in the results.

Technical Contributions

• Preregistered benchmark: 4,644 valid runs across 24 deterministic coding tasks, seven reasoning models, and two real agent harnesses, ensuring methodological rigor.
• Identification of waste mechanisms:
  • Multiple approaches: Prompts encouraging multiple strategies lead to 2.4x–7.4x more reasoning tokens and ~3 discarded solution branches, with no success gain.
  • Maximum certainty: Prompts demanding high confidence trigger repeated verification, leading to 2.5x more tool calls, 3x longer runtime, and 18x cost increase.
• Cost carrier taxonomy: Distinguishes between reasoning-heavy (token-borne) and tool-heavy (system-borne) waste, enabling targeted optimization.
• Harness amplification: Shows that the agent harness (e.g., how tools are exposed) can amplify these effects, changing cost per successful task by 5x–30x.
• Bounded-efficiency prompt: A practical prompt design that preserves diagnostic and validation quality while avoiding waste mechanisms.

Results

The central result is that prompt wording changes where effort is spent, not just how much. For multiple approaches, reasoning token usage increased by 2.4x to 7.4x across all six open models, with about three elaborated but discarded solution branches, yet still only one implemented solution and no success gain. For maximum certainty, runs with high redundant verification cost 18x the clean-run median, executed 2.5x more tool calls, and took 3x longer, again without a success gradient. These effects were consistent across models and harnesses, and survived a frozen holdout, paraphrase tests, a Kimi-K3 replication, and a first-party Claude Sonnet 5 study. Harness design amplified both effects, changing cost per successful task by 5x to 30x. In contrast, bounded-efficiency wording avoided these waste mechanisms while maintaining diagnosis and final validation.

Significance

This paper reframes prompt engineering as work design for AI agents. It provides a framework for understanding how prompt choices influence not only the outcome but also the resource footprint of the agent's work. For practitioners, this means that prompt optimization should consider both accuracy and cost, and that different waste mechanisms require different mitigation strategies. The findings also have implications for agent harness design, suggesting that the interface between the model and tools can be a major cost lever. As AI agents become more autonomous and widely deployed, understanding and controlling these costs will be essential for sustainable and efficient AI systems.