Scaling Laws for Agent Harnesses

Analysis

Why This Paper Matters

Agent harnesses—the scaffolding that controls tool use, feedback, verification, memory, and repair—are critical to language model performance, yet their scaling behavior has been poorly understood. Traditional scaling laws treat raw compute (tokens, tool calls, wall time, cost) as the primary coordinate, but this paper argues that such measures conflate useful feedback with redundant or unstable interaction. This is a significant conceptual advance because it reframes scaling from "more compute" to "more effective feedback," aligning with the intuition that not all compute is equally valuable.

The paper introduces Effective Feedback Compute (EFC), a trace-level coordinate that quantifies informative, valid, non-redundant, and retained feedback. This is a novel and practical contribution: it provides a measurable quantity that can be extracted from agent traces and used to predict performance. The authors validate EFC across synthetic, real, held-out, and prospective settings, showing that it dramatically outperforms raw compute in scaling law fits. This matters because it offers a new axis for optimizing agent harnesses, potentially leading to more efficient and reliable LLM agents.

Technical Contributions

• EFC definition: Formalizes feedback quality along four dimensions: informativeness, validity, non-redundancy, and retention. This is a principled decomposition that can be operationalized.
• Practical variants: Introduces Estimated-EFC and NRS-EFC to handle realistic traces where oracle feedback is unavailable, making the approach applicable in practice.
• Harness efficiency η: Defines a metric to compare harnesses in terms of feedback per unit of raw compute, enabling efficiency comparisons.
• Task-demand normalization: Proposes D_task to normalize EFC across heterogeneous tasks, allowing cross-task scaling law comparisons.
• Companion control layer: Demonstrates that EFC can be used as a control signal to guide harness execution, improving performance while reducing cost.

Results

The paper reports strong quantitative results. In controlled scaling experiments, Oracle-EFC normalized by task demand (D_task) achieves an R² of 0.99, indicating near-perfect linear scaling. On real traces, NRS-EFC/D_task reaches R²=0.93, whereas raw compute has near-zero or even negative fit, highlighting the inadequacy of raw compute as a scaling coordinate. The companion control layer improves mean pass rate from 61.2% to 68.2% (a relative improvement of ~11%) while reducing mean raw cost from 213.8 to 85.1 (a ~60% reduction) under matched settings. These results are compelling because they show both predictive power and practical utility.

Significance

This paper has broad implications for the AI field. It challenges the prevailing assumption that scaling laws are solely about compute, suggesting that the quality of feedback is a more fundamental driver of agent performance. This could influence how researchers design harnesses, allocate resources, and evaluate agent capabilities. The EFC framework may also serve as a bridge between scaling law research and practical agent engineering, offering a concrete metric to optimize. Future work could extend EFC to other domains, such as multi-agent systems or embodied AI, and explore its relationship with other scaling coordinates like model size and data. Overall, this is a timely and thought-provoking contribution that could reshape how we think about scaling in agentic AI.