StateM: Reaching 95.3% Raw Accuracy, or a \$15 Frontier Run, on Terminal-Bench 2.1 via Harness Scaling

Analysis

Why This Paper Matters

This paper challenges the prevailing assumption that agent performance is primarily limited by the underlying model's capabilities. By focusing on the execution harness—the runtime and procedural scaffolding around the model—StateM demonstrates that significant gains can be achieved without any weight updates. This is particularly relevant as frontier models become more capable but still fail on long-horizon tasks due to state mismanagement, forgotten procedures, or premature termination. The paper's results on Terminal-Bench 2.1, a challenging benchmark, show that harness scaling can boost accuracy by up to 12 percentage points, rivaling or exceeding the gains from upgrading to a more powerful model.

The cost implications are striking: StateM achieves state-of-the-art results with API costs of about $15 compared to $574.68 for the reference, a 97% reduction. This democratizes access to high-performance agent systems, making them feasible for smaller labs and individual researchers. Moreover, the transferability of runbooks across models (e.g., from GPT-5.5 to GPT-5.6) suggests that procedural knowledge can be decoupled from model weights, opening new avenues for sharing and improving agent behaviors.

Technical Contributions

• Durable State Management: StateM maintains explicit, durable state across execution phases, preventing the agent from losing track of mutable variables and ensuring consistency.
• Phase-Local Context: The runtime provides context that is scoped to the current phase, reducing confusion and improving focus on the immediate task.
• Checked Transitions: Transitions between states are validated, catching errors early and enabling recovery.
• Recoverable Runbooks: Runbooks encode step-by-step procedures that can be resumed after failures, ensuring that lessons from earlier executions are not lost.
• Versioned Procedural Practices: Practices are versioned and inspectable, allowing agents and users to collaborate and refine them over time.
• Golden Rules: A set of universal rules that guide the agent's behavior, which can be adapted to new models with minimal effort (less than $38 of adaptation).

Results

On Terminal-Bench 2.1, StateM with GPT-5.5 xhigh achieves 92.1% raw accuracy, up from 83.1% reference, and outperforms GPT-5.6 Sol Ultra at 91.9%. With GPT-5.6 Sol xhigh, StateM reaches 95.3% across 445 trials, succeeding on all 89 tasks at least once. The frozen profile also lifts GPT-5.6 Luna from 76.7% to 85.4%, surpassing the 84.9% Sol xhigh reference. For DeepSeek-V4 Flash, StateM improves accuracy from 82.7% to 88.1% (89.1% on an 88-task common core), and extending latency-sensitive tasks matches the reported 88.8% GPT-5.6 Sol max result. On BusinessBench, family-specific runbooks yield held-out gains of 0.55 macro and 1.34 micro points, with mechanism-matched families improving by 10.04 points. The cost of final-score API usage is about $15 versus $574.68 for the GPT reference, with total DeepSeek expenditure at $52.22.

Significance

StateM introduces a new axis of scaling—harness scaling—that is orthogonal to model scaling. This could shift research focus toward building better execution environments, which are often overlooked. The explicit, inspectable nature of runbooks and practices also addresses the interpretability gap in agent systems, making it easier to debug and improve behavior. The cost efficiency and cross-model transferability suggest that high-performance agents can be built on open-weight models with modest budgets, potentially accelerating adoption in industry. However, the approach requires careful engineering of runbooks and golden rules, which may not generalize to all domains without significant effort. Future work could explore automated discovery of runbooks and rules, further reducing human overhead.