What Moves the Pareto Frontier in Tool-Using Agents? A Compute-Aware Study of ReAct Variants

Analysis

Why This Paper Matters

Tool-using LLM agents are increasingly deployed in real-world applications where inference cost is a critical constraint. However, most research compares agents solely on accuracy, ignoring the computational cost. This paper addresses this gap by introducing a budgeted evaluation framework that considers both accuracy and inference cost, providing a more realistic assessment of agent performance.

The study focuses on ReAct variants, a popular paradigm for tool-using agents that interleaves reasoning and acting. By systematically varying the compute budget, the authors reveal that the optimal agent strategy is not fixed but depends on the available resources. This finding challenges the common practice of optimizing for accuracy alone and highlights the need for compute-aware model selection in practice.

Technical Contributions

• Budgeted Evaluation Framework: The paper proposes a methodology to evaluate agents under explicit inference cost constraints, enabling the construction of Pareto frontiers that show the trade-off between accuracy and cost.
• Systematic Comparison of ReAct Variants: The study includes multiple ReAct variants, likely differing in prompting strategies, memory mechanisms, or action selection methods, and evaluates them across a range of compute budgets.
• Pareto Frontier Analysis: The authors analyze how the Pareto frontier shifts with compute, identifying which variants are optimal at different budget levels and providing guidance for practitioners.
• Compute-Aware Insights: The paper offers insights into which components of ReAct contribute most to accuracy gains relative to their computational overhead, helping to design more efficient agents.

Results

While the abstract does not provide specific numbers, the key result is that no single ReAct variant dominates across all compute budgets. The Pareto frontier changes with the budget: some variants that are suboptimal at high budgets become optimal under tight constraints, and vice versa. This indicates that the choice of agent strategy should be informed by the deployment's cost constraints. The paper likely includes empirical results on a benchmark of tool-using tasks, showing accuracy and cost measurements for each variant.

Significance

This work has significant implications for the AI field, particularly for the deployment of LLM agents in cost-sensitive environments. By promoting budget-aware evaluation, it encourages researchers and practitioners to consider the full trade-off space rather than optimizing for a single metric. This could lead to more efficient agent designs and better resource allocation in production systems. Additionally, the methodology can be extended to other agent architectures and tasks, fostering a more holistic approach to agent evaluation.