ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress (2026)

Free

3D reliability surface R(k,ε,λ) unifying consistency, robustness, fault tolerance — chaos engineering for agents; ReAct outperforms Reflexion under stress; pass@1 overestimates reliability by 20–40%

FreeFree tier

Type

Open Source

Visit Website GitHub (40 stars)

About ReliabilityBench: Evaluating LLM Agent Reliability Under Production-Like Stress (2026)

ReliabilityBench is a benchmark designed to evaluate the production readiness of LLM agents by assessing their reliability under stress conditions. It introduces a unified 3D reliability surface R(k,ε,λ) that captures three critical dimensions: consistency under repeated execution (using pass^k), robustness to semantically equivalent task perturbations at intensity ε, and fault tolerance under controlled tool and API failures at intensity λ. The benchmark employs action metamorphic relations to define correctness by end-state equivalence rather than text similarity, and includes a chaos-engineering-inspired fault injection framework that simulates timeouts, rate limits, partial responses, and schema drift. The framework was evaluated on two models (Gemini 2.0 Flash and GPT-4o) and two agent architectures (ReAct and Reflexion) across four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes. Key findings include that perturbations alone reduce success rates from 96.9% (ε=0) to 88.1% (ε=0.2), rate limiting is the most destructive fault type, ReAct outperforms Reflexion under combined stress, and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at significantly lower cost. ReliabilityBench provides a systematic method for stress-testing agent reliability before deployment in production environments.

Key Features

Unified reliability surface R(k,ε,λ) measuring consistency, robustness, and fault tolerance

Action metamorphic relations for correctness based on end-state equivalence

Chaos-engineering fault injection framework (timeouts, rate limits, partial responses, schema drift)

pass^k metric for evaluating consistency under repeated execution

Evaluation across four domains (scheduling, travel, customer support, e-commerce) with 1,280 episodes

Comparison of ReAct and Reflexion architectures on Gemini 2.0 Flash and GPT-4o

Pros & Cons

Pros

Provides a comprehensive reliability evaluation beyond single-run success rates
Incorporates realistic stress factors like perturbations and API failures
Identifies specific failure patterns (e.g., rate limiting is most damaging)
Enables low-cost reliability assessment with Gemini 2.0 Flash achieving comparable results to GPT-4o
Open-source benchmark with systematic methodology for reproducibility

Cons

Limited to two models and two agent architectures in the initial evaluation
May not cover all possible failure modes in diverse production environments
Requires setting up fault injection infrastructure to replicate the benchmark
Benchmark domains are limited to four areas; broader coverage would strengthen conclusions

Best For

Assessing production readiness of LLM agents before deploymentComparing agent architectures (e.g., ReAct vs Reflexion) under realistic failure conditionsIdentifying weak points in agent reliability, such as rate limiting vulnerabilityBenchmarking cost-reliability trade-offs across different models (e.g., Gemini 2.0 Flash vs GPT-4o)

FAQ

What is ReliabilityBench?

ReliabilityBench is a benchmark for evaluating the reliability of LLM agents under production-like stress conditions. It measures three dimensions: consistency (via pass^k), robustness (to task perturbations at intensity ε), and fault tolerance (to tool/API failures at intensity λ), using a unified reliability surface R(k,ε,λ).

What are the key findings from ReliabilityBench?

Key findings include: perturbations alone reduce success from 96.9% to 88.1% at ε=0.2; rate limiting is the most damaging fault type; ReAct is more robust than Reflexion under combined stress; and Gemini 2.0 Flash achieves comparable reliability to GPT-4o at much lower cost.

Which models and architectures were evaluated?

The benchmark evaluated Gemini 2.0 Flash and GPT-4o with two agent architectures: ReAct and Reflexion. The test covered four domains (scheduling, travel, customer support, e-commerce) over 1,280 episodes.