A new benchmark aims to answer a question most AI evaluations skip: when a multi-agent orchestration system breaks, can anyone tell why? OrchestraBench, introduced in a paper submitted to arXiv on 5 Aug 2026, uses controlled failure injection to measure not just accuracy, but recovery, decomposition quality, and the spread of errors through a pipeline.
The paper, posted under the cs.AI category with arXiv ID 2608.05263, runs 8 pages with 4 figures. Its authors are Yidian Chen, Yingzi Gu, Natan Vidra, Spurthi Setty, and Sharon Zheng. A DOI is pending registration at https://doi.org/10.48550/arXiv.2608.05263.
From Demos to Production
Multi-agent orchestration frameworks are moving from demos to production, according to the paper. That shift creates a problem. Existing benchmarks typically report task accuracy without diagnosing failure causes, cascade origins, or routing decisions. A system can score well on final outputs while hiding serious structural weaknesses.
OrchestraBench takes a different route. It uses a controlled, seed-reproducible failure-injection harness over templated enterprise workflows. The benchmark introduces two primary metrics: cascade radius and per-failure-mode recovery. It also compares routing policies using bootstrap confidence intervals and paired tests.
Routers Fail on Adversarial Cases
The benchmark includes a 26-case gold-labelled diagnostic designed to test routing decisions. The results were stark. A keyword/flag router scored 0% on adversarial cases with misleading or missing surface flags. An intent-reasoning model router scored 100% on the same adversarial cases, matching the oracle.
That gap matters for production systems. Surface-level signals can mislead simple routers, while models that reason about intent handle ambiguity far better. The diagnostic isolates routing as a distinct failure point, separate from task execution.
Three Tiers of Failure Handling
Controlled mechanism probes with a real Claude agent from Anthropic over a verifiable arithmetic dependency chain revealed three failure-handling tiers across five MAST modes. Tool faults recovered fully, with a recovery rate of 1.0. Ambiguous delegation recovered partially, at 0.30. Three latent or semantic modes never recovered, holding a recovery rate of 0.0.
The ordering of these tiers persisted when the computation was reframed as a loan-approval workflow. It also persisted across Sonnet, Opus, and Haiku models, though absolute rates shifted with context. That consistency suggests the failure-handling hierarchy is structural, not incidental.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
One finding stands out for anyone running agents in production. Blind retry reproduced latent faults and increased time to detection. Retrying without diagnosis does not fix hidden problems. It amplifies them. The paper argues that detection and attribution are necessary for containment. You cannot contain what you cannot name. The probes show that identifying the failure mode is a prerequisite for any meaningful recovery.
Cascade radius also grew with pipeline depth. The mean radius was 0.9 at depth 3 and 4.7 at depth 7. Deeper pipelines spread errors further, making early detection more valuable.
Trusted State, Not Autonomous Smarts
An ablation study on trusted-state repair produced a sobering result. Apparent containment gains came primarily from the trusted-state signal rather than autonomous detection. In other words, the system improved because it was told where the trusted state was, not because it figured it out on its own.
The authors are careful about scope. They describe the results as controlled-chain mechanism probes, not domain-workload claims. The findings show how failure handling works in a controlled setting, not how any specific enterprise workload will behave.
OrchestraBench fills a gap by asking questions other benchmarks avoid. Where did the failure start? How far did it spread? Which failure modes can be recovered, and which cannot? Those questions matter as multi-agent systems move into production environments where errors have real costs.
The paper is available as PDF and HTML, with the HTML version marked experimental. Submission history shows v1 arrived on Wed, 5 Aug 2026 at 17:27:15 UTC, weighing in at 155 KB. The current browse context is cs.AI. References and citations include NASA ADS, Google Scholar, and Semantic Scholar. Bibliographic tools listed include Bibliographic Explorer, Connected Papers, Litmaps, and scite.ai. Code and data tools include alphaXiv, CatalyzeX, DagsHub, Gotit.pub, Huggingface, and ScienceCast. Demos are available through Replicate, Hugging Face Spaces, and TXYZ.AI. Recommenders include Influence Flower and CORE Recommender.
arXivLabs, mentioned in the paper's metadata, is a framework for experimental projects with community collaborators. arXiv itself is committed to values of openness, community, excellence, and user data privacy.
The benchmark does not claim to solve multi-agent reliability. It provides a way to measure it. For teams deploying orchestration frameworks, that measurement is the first step toward understanding what breaks, why it breaks, and whether anything can bring it back.