VAKRA Benchmark Examines AI Agent Reasoning Failures

IBM researchers have released a detailed look at VAKRA, a benchmark designed to test AI agents' reasoning and actions in settings similar to enterprise operations. This tool-grounded evaluation uses executable setups to check multi-step workflows through complete execution records. Agents work with more than 8,000 locally hosted APIs supported by actual databases in 62 domains, plus matching document sets. Tasks demand 3 to 7 steps that mix structured API calls with unstructured searches under natural language limits for tool use. Models score low overall on VAKRA. This post covers task specifics and failure patterns seen across various challenges.

VAKRA Task Capabilities

VAKRA includes four tasks, each targeting distinct skills.

Capability 1 covers API chaining with business intelligence APIs. It has 2,077 test cases over 54 domains using tools from SLOT-BIRD and SEL-BIRD collections (Elder et al., 2026). Tool sets expanded beyond Elder et al. with more domains. Each domain limits to one tool group. Tasks chain 1 to 12 tool calls for the answer.

Example query: "Which football team has a build-up play speed of 31, build-up plan dribbling of 53, and build-up play passing of 32?"

Tool calls include get_data with tool_universe_id="486ea46224d1-aeb8037c5e78", then filters on play_speed=31, play_dribble=53, play_passing=32, and get_team_name on the result. Answer: "FC Barcelona".

Each case links to a JSON data source. MCP servers offer get_data(tool_universe_id=id) first. It sets up the source, gives a data preview (Figure 3), stores full data server-side to cut transfers, sets tool exposure by ID, and matches domain database.

SLOT-BIRD supplies 7 global tools for data handling like filtering and sorting, drawn from Tableau and Google Analytics styles. SEL-BIRD adds specialized ones: shared with SLOT-BIRD or split from categorical args (sort_data ascending bool becomes sort_data_ascending and descending). retrieve_data swaps for query-specific getters; each data key gets a get_KEY_NAME function, averaging 4 per case.

Data preview example:

{ "handle": "retrieved_data_1", "num_records": 2, "key_details": [ {"name": "team_name", "dtype": "str", "first_3_values": ["FC Barcelona", "Manchester City"]), {"name": "play_speed", "dtype": "int32", "first_3_values": [31, 40]}, {"name": "play_dribble", "dtype": "int32", "first_3_values": [53, 30]}, {"name": "play_passing", "dtype": "int32", "first_3_values": [32, 16]} ] }

Capability 2 tests tool selection with dashboard APIs. 1,597 cases over 17 domains use expanded REST-BIRD (Elder et al.). Endpoints handle specific queries via FastAPI wrapped in MCP. Agents pick right APIs from domain sets (Figure 1 example). Domains range 6 to 328 tools, average 116. get_data sets domain APIs.

OpenAI spec caps tool lists at 128, so agents need shortlisting. Repo baselines include basic shortlisting.

Capability 3 adds multi-hop reasoning with dashboard APIs. 869 cases from 38 domains use REST-BIRD, needing 1 to 5 hops (Figure 1). Figure 4 shows hop-type distribution.

Capability 4 handles multi-hop, multi-source reasoning plus policy rules. 644 cases over 41 domains on REST-BIRD. Figure 4 covers hybrid hops without policies. Features include:

Multi-source: Domain document indexes added. Queries mix API and docs per hop, e.g., API-RAG-API. Sources cleaned so info stays in one place per hop.

Multi-turn: Dialogs as context-response pairs; agent answers current turn.

Tool policies: Text rules on sources, e.g., "If a user's query pertains to Technology & Software... make sure you try answering them by only using document retrievers. Do not use other types of tools."

Repo baseline adds to prompt: "You are a helpful assistant with access to tools.\n Tool Usage Constraint: {additional_instructions}."

Evaluation Setup

VAKRA checks tool environments for workflow execution and answer accuracy. Framework reviews final response and full tool trajectory: calls, inputs, results.

Evaluator takes predicted response and trajectory, runs predicted tools against ground truth env.

Waterfall pipeline (Figure 6): Policy check for Cap 4, trajectory compare, then response eval if passed.

Tool sequence check: Executes predicted vs. ground truth responses. Program check if all ground info recovered. If inconclusive (partial, semantic diffs), LLM judge from CRAG (Yang et al., 2024) checks info capture.

Final response: LLM judge verifies grounding in predicted tools and fact match to ground truth.

Rewards full valid processes.

Scoring: Equal weight per capability: Leaderboard_Score = (1/4) * sum Capabilities.

Caps 1-3: correct / total.

Cap 4 weights multi-source x2: (correct_multi x2 + correct_single) / (total_multi x2 + total_single).

Failure Patterns

Errors categorized by first breakdown: tool select, arg provision (no halluc/skip), arg values, response grounding/accuracy.

Sequential assign to earliest fail for disjoint counts.

Cap 1 (2,077 samples): Hard for all, GPT-OSS-120B tops via schema grasp and param choice on optional-heavy tools. Synthesis easier post-correct calls vs. other caps.

SEL-BIRD (600): More tools, fewer params. SLOT-BIRD (1,477): Fewer tools, more params. Models err on SLOT arg names (except GPT-OSS-120B), SEL tool select.

Cap 2 (1,597): Gemini-3-flash-preview leads all errors. High tool/param select errs due to many options/few params. Few param halluc/skip, but synthesis fails even post-correct calls (Gemini, Claude-Sonnet-4-5).

Multi-hop (Cap 3): Performance drops with hops.

Best on 1-hop, worse 2-hop, 3+.

Cap 4 multi-source: Diff by type.

1-hop API > multi-API; RAG/hybrid harder. GPT-OSS-120B skips 1-hop RAG tools, uses params (Wikipedia-style). Gemini strong on 2-hop API-RAG.

Policies: Added difficulty.

No-answer-change policies ok; restricting key source drops most (except Granite-4.0-h-Small-32B). Models violate or fail retrieval/reason.

Policies show models reason tools/sources but falter adding constraints, key for deployment.

Key Takeaways

VAKRA highlights gaps in tool skills vs. full agent reliability. Models handle single calls but fail composition with APIs, docs, dialogs, policies.

Test agents on VAKRA: dataset at https://huggingface.co/datasets/ibm-research/VAKRA, GitHub https://github.com/IBM/vakra, leaderboard submit https://github.com/IBM/vakra?tab=readme-ov-file#submitting-to-the-live-leaderboard.

Related on Neura Market

• AI Tools Directory
• AI Benchmarks
• Agent Frameworks