QuoteBench: How Matched Scores Can Hide Command-Path Failures

Analysis

Why This Paper Matters

LLM coding agents are increasingly deployed to execute Bash commands in real environments, yet their evaluation often relies on matched execution scores—whether the command produced the expected outcome. This paper exposes a critical flaw: matched scores can mask failures that occur after generation, such as when the command is serialized, wrapped, or reparsed by the execution interface. QuoteBench systematically isolates these failures by introducing a deliberately unescaped parser at the execution boundary, showing that the same generated command can succeed or fail depending on the transport.

This matters because it reveals that current evaluation practices may overstate model capability. A model that appears to perform well on a benchmark might actually be failing at the command-path level, and its success could be due to compensation mechanisms rather than robust generation. The paper's finding that deployment configuration can reorder model rankings has direct implications for practitioners choosing between models for real-world tasks.

Technical Contributions

• QuoteBench benchmark: 56 one-shot tasks from 14 incident-derived families, with exact final-state validation to ensure that success is measured by the actual system state, not just command output.
• Generation contract vs. execution transport: The paper formalizes the distinction between what the model generates and how that output is processed by the execution environment, introducing a clear boundary for testing.
• Deliberately unescaped added parser: By inserting a parser that does not escape quotes, the authors create a controlled failure point that can be disclosed or hidden, allowing them to measure the impact of boundary awareness.
• Cross-configuration replay: The same reply is replayed through eight different configurations, varying the parser and disclosure, to isolate the effects of generation vs. adaptation.
• Operating point analysis: The paper introduces the concept of an operating point—the specific combination of configuration, contract, and validator—that must be reported for meaningful comparisons.

Results

The results are striking: replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points across the eight configurations. This shows that the execution transport can have a massive impact on performance, independent of the model's generation quality. When the parser boundary is disclosed, models recover 30.4 to 60.7 points for six configurations, but for two configurations, recovery is zero or slightly negative, indicating that some models cannot adapt even when aware of the issue.

Raw generation is nearly saturated at the frontier, meaning that top models generate commands that are almost always correct in the ideal execution path. However, boundary adaptation is what separates models: GPT-5.6-sol's matched gap of -3.6 points hides a -64.3 point damage from the parser and a +60.7 point compensation from adaptation. This decomposition reveals that the model's apparent near-perfect score is actually a balance of two large opposing effects.

Furthermore, the deployment configuration reorders models: among 26 comparable pairs, one reversal is unambiguous and four more sit on single-task margins. This suggests that model rankings are not stable across execution environments, which is a critical concern for practitioners who rely on benchmark scores to select models.

Significance

This paper has significant implications for the evaluation of LLM coding agents. It argues convincingly that matched execution scores are not intrinsic model properties but are contingent on the deployment configuration. The authors propose that evaluations should report the model configuration, generation contract, execution path, operating point, and final-state validator to provide a complete picture.

For the AI field, this work highlights the need for more robust and transparent evaluation methodologies, especially as agents are deployed in increasingly complex environments. It also opens up new research directions into how models can be trained to adapt to execution boundaries, and how to design interfaces that minimize the gap between generation and execution. The QuoteBench benchmark provides a concrete tool for measuring this gap, and its findings will likely influence how future coding agents are evaluated and improved.