A new research paper introduces EcoAgent-Bench, a benchmark designed to test whether large language model (LLM) agents can make sound economic decisions when resources are limited. The findings, posted to arXiv on 6 Aug 2026, suggest that finishing a task under budget and choosing actions economically are two entirely different skills.
The paper, submitted by lead author Jie Wu with co-authors Ming Gong, Feixiang Cheng, and Qinqin Zhao, argues that current agent benchmarks miss a crucial part of real-world deployment. Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic. In deployment, the choice among local lookup, broad search, composite research tool, stronger model, or human escalation is part of the task itself.
What EcoAgent-Bench Measures
EcoAgent-Bench includes 304 real-derived tasks. These tasks span five families adapted from existing benchmarks: GAIA, HotpotQA, and MuSiQue. Each task comes with priced actions and an explicit budget, forcing agents to weigh the cost of every step they take.
The benchmark tests four specific economic decisions. Agents must avoid unnecessary escalation, escalate when local evidence is insufficient, select an appropriate model tier, and stop when premises are unsupported. These decisions mirror the trade-offs a human analyst faces when deciding how much effort and money to spend on a query.
The evaluation covers seven LLM agents in two settings: tool-API and workspace-CLI. Four oracle scripted controls provide baseline comparisons. The paper is 8 pages long, includes 3 figures and 4 tables, and carries the arXiv ID 2608.05519 under the category cs.AI, with cross-listings in cs.CL and cs.LG.
The Problem With Micro-Averaged Accuracy
The authors identify a flaw in how agent performance is often scored. Micro-averaged accuracy rewards one-sided policies. An agent that always escalates to the most expensive option will score well on tasks that require escalation, while ignoring the cost of doing so.
Always-escalate controls achieve high micro success while failing save-oriented tasks. This means a naive policy can look strong under standard metrics. To counter this, the paper proposes an economic-consistency score, defined as the worse of accuracy on upgrade-oriented and save-oriented family groups. This score exposes the failure of one-sided policies.
The results are stark. Tool-API agents attain only 3.9-24.0% micro strict success. Even the best agent reaches at most 7.3% economic consistency. These numbers indicate that current models struggle to balance cost and correctness in a meaningful way.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
Agents Overspend or Stop Too Early
The analysis shows a pattern of poor judgment. Tool-API agents often either stop before warranted escalation or overspend on cheap tasks. They cannot reliably tell when a simple lookup suffices and when a broader search or stronger model is necessary.
A threshold-crossing budget sweep tested how GPT-5.4 responds to tighter budgets. The escalation rate changed from 0% to only 3%. This suggests that adjusting the budget alone does not teach the agent to escalate more often when it should. The model's behavior remains largely fixed regardless of the financial pressure applied.
The paper concludes that completion under a budget and economical action selection are distinct properties. An agent can finish a task within its budget while still making wasteful choices, and vice versa. The benchmark is designed to separate these two dimensions of performance.
A New Tool for the Research Community
The authors have released several artifacts for further study. These include the task bundle, the transformation pipeline, frozen evaluation environments, and integrity-bound result artifacts. The goal is to let other researchers reproduce the findings and build on the benchmark.
The paper is available under the DOI https://doi.org/10.48550/arXiv.2608.05519. The submission history shows v1 was posted on Thu, 6 Aug 2026 at 01:47:47 UTC, with a file size of 345 KB. The license is listed as "view license," with no specific terms provided.
For the field of LLM agents, this work highlights a gap between capability and cost-awareness. As agents are deployed in settings where every API call or compute minute has a price, the ability to spend wisely becomes as important as the ability to answer correctly. The low scores across all evaluated agents suggest that this skill is far from solved.