Black-box API bug detection across 7 AI systems
FreeFreeFree tier
About Black-box API bug detection across 7 AI systems
This report presents APIEval-20 v1.0, a black-box benchmark contributed by KushoAI that evaluates seven AI systems (general-purpose LLMs, coding agents, and KushoAI) on API bug detection across 20 live API scenarios with 97 planted functional bugs spanning three complexity tiers. Each system receives only a JSON schema and one valid sample payload, then must generate test cases that expose failures in a live reference API. Key findings show that simple schema-level tests are now table stakes, but complex cross-field and business-logic bugs separate the best systems. KushoAI detects 76% of complex planted bugs, compared with 53% for the strongest coding-agent workflow and 34% for the strongest general-purpose LLM, and shows the lowest run-to-run variance among evaluated systems.
Key Features
Open-source benchmark (APIEval-20 v1.0) for API bug detection
Black-box evaluation using only JSON schema and one sample payload
20 live API scenarios across seven application domains
97 planted functional bugs across three complexity tiers (simple, moderate, complex)
Comparison of 7 systems across 3 groups: general-purpose LLMs, coding agents, KushoAI
Evaluates multiple workflow modes: one-shot prompting, structured test-strategy prompting, prompt chaining, native coding-agent workflows, and native API test generation
Lowest run-to-run variance among evaluated systems (KushoAI)
Pros & Cons
Pros
- Comprehensive benchmark with real planted bugs across multiple domains
- Reveals that plausible-looking test suites can still miss critical bugs
- Identifies that simple schema-level tests are insufficient for production risk assessment
- Highlighting that test composition matters more than test volume
- KushoAI shows highest complex bug detection (76%) and lowest variance
- Provides actionable insights for engineering teams evaluating AI test generation tools
Cons
- Benchmark only covers functional API bugs, not security, performance, or integration errors
- Limited to black-box testing without access to source code or documentation
- Benchmark is a static snapshot; performance may change as AI models evolve
- Benchmark contributed by KushoAI, which is also one of the evaluated systems, introducing potential bias
- Requires running test suites against live reference APIs, not a fully simulated environment
Best For
Evaluating AI coding tools for real-world API test generationUnderstanding whether generated test suites actually catch production-relevant bugsBenchmarking different prompt engineering strategies for API testingAssessing the trade-off between test volume and test compositionSelecting an AI test generation tool for CI/CD pipelines based on consistency and bug detection capability
FAQ
What is APIEval-20 v1.0?
APIEval-20 v1.0 is a black-box benchmark contributed by KushoAI that evaluates AI systems on API bug detection. It includes 20 live API scenarios across seven domains with 97 planted functional bugs.
Which systems were evaluated?
Seven systems were evaluated, grouped into three categories: general-purpose LLMs, coding agents, and KushoAI. Specific names are not listed in the summary.
What are the key findings of the benchmark?
Simple schema-level tests are now table stakes but do not differentiate systems. Complex cross-field and business-logic bugs separate the best performers. KushoAI detected 76% of complex bugs, compared to 53% for the best coding agent and 34% for the best general-purpose LLM. Test composition matters more than test volume, and consistency (low run-to-run variance) is important for CI/CD.
How many bugs were planted and what complexity tiers?
97 planted bugs across three complexity tiers: simple (e.g., missing fields, null values, wrong types), moderate, and complex (e.g., invalid refund state, role hierarchy violations, conflicting recurrence rules).
What is the methodology used?
The evaluation uses a black-box approach: each system receives only a JSON schema and one valid sample payload. No source code, documentation, or hints about where failures are planted are provided. Systems must generate test cases that expose failures in a live reference API.