Evaluation Findings (Sample Run)
Reports results from two evaluation pipelines: agent response scoring and recommendation accuracy, with aggregate metrics and improvement steps.
What this file does
Reports results from two evaluation pipelines: agent response scoring and recommendation accuracy, with aggregate metrics and improvement steps.
When to use it
- You have a multi-agent system and need to present evaluation results to stakeholders
- You want to track regression in agent response quality or recommendation precision
- You need a template for documenting evaluation runs with pass rates and artifact paths
- You are debugging why recommendation metrics are low and need guidance on fixes
Assumes this stack
Evaluation Findings (Sample Run)
Summary of a single run of both pipelines for presentation. Re-run the evals to refresh numbers.
Pipeline 1: Agent response evaluation (DeepEval G-Eval)
Metric: Relevance/helpfulness (LLM-as-judge). Threshold: 0.5.
| test_id | user_query | passed | score (0–1) |
|---|---|---|---|
| greeting_1 | Hi | ✓ | 0.77 |
| vehicles_intent_1 | I'm looking for a used car under $15,000 | ✓ | 0.70 |
| laptops_intent_1 | I need a laptop for programming | ✓ | 0.67 |
| books_intent_1 | Recommend me some sci-fi books | ✗ | 0.10 |
| reset_1 | Start over | ✓ | 1.00 |
Summary: 4/5 passed. The books case failed because the catalog returned no sci-fi books (empty result), so the judge correctly marked the response as unhelpful.
Artifacts: evaluation/agent_response/results/agent_response_eval_results.csv (and .json).
Pipeline 2: Recommendations evaluation
Query → UCP (chat agent filter extraction)
Metric: Fraction of expected UCP keys that match the agent’s produced filters.
| query_id | query (abbreviated) | expected_ucp_match |
|---|---|---|
| q1 | Gaming laptop under $2000 | 1.0 |
| q2 | MacBook coding, ~$1500 | 1.0 |
| q3 | Cheap laptop students under $600 | 1.0 |
| q4 | Dell XPS 16GB for work | 0.8 |
| q5 | Lightweight 14", under $1200 | 1.0 |
| q6 | Laptop for video editing | 1.0 |
| q7 | 32GB RAM for ML | 0.75 |
Aggregate: Mean UCP match rate ≈ 93.6%.
Artifacts: evaluation/recommendations/results/query_to_ucp_results_<ts>.csv.
UCP → recs (merchant/MCP only)
Metrics: Hard constraint pass rate (% of recs satisfying expected UCP); mean soft similarity to ground truth (0 when no ground truth IDs set).
| query_id | query (abbreviated) | hard_pass_rate | mean_soft_similarity | recommended_count |
|---|---|---|---|---|
| q1 | Gaming under $2000 | 0.0 | 0.0 | 10 |
| q2 | MacBook ~$1500 | 0.0 | 0.0 | 5 |
| q3 | Students under $600 | 1.0 | 0.0 | 10 |
| q4–q7 | … | 0.0 | 0.0 | 6–10 |
Aggregate: Mean hard pass rate ≈ 14.3%; mean soft similarity = 0 (ground_truth_product_ids empty in golden set).
Artifacts: evaluation/recommendations/results/ucp_to_recs_results_<ts>.csv.
Note: Full system (query → recs) is not run: the chat agent is multiturn and often returns a question instead of recommendations, so recs-based metrics would be misleading. We evaluate the merchant agent (UCP → recs) separately.
Baseline (optional)
Run without --skip-baseline to compare an out-of-the-box LLM (schema + sample products + user query) with the same hard/soft scoring. Output: evaluation/recommendations/results/baseline_results_<ts>.csv.
Summary JSON (Pipeline 2)
evaluation/recommendations/results/summary_<ts>.json contains:
timestamp,golden_path,num_queriesaggregates.query_to_ucp.mean_ucp_match_rateaggregates.ucp_to_recs.mean_hard_pass_rate,mean_soft_similarityaggregates.baseline(if run)
Use these aggregates for slides or reports.
How to improve numbers
- Pipeline 1: Add or fix book catalog so sci-fi queries return results; add more test cases in
test_cases.json. - Pipeline 2:
- Run
python -m evaluation.recommendations.populate_ground_truthto fillground_truth_product_ids(filter by hard constraints, then top 10 by query–product embedding similarity). - Run
populate_product_embeddingsso embeddings exist for scoring. - Align hard-constraint checks in
scoring.pywith the actual product fields returned by the agent/MCP (e.g. price_cents, good_for_* in attributes).
- Run
What's inside
Two pipeline summaries with tables, aggregate metrics, artifact file paths, and a how-to-improve section.
Change this for your project
- Replace
evaluation/agent_response/results/agent_response_eval_results.csvwith your own output path - Replace
evaluation/recommendations/results/with your results directory - Replace
test_cases.jsonwith your test case file name - Replace
scoring.pywith your scoring module name
Where it goes
Keep alongside your test suite. Used to define and score model evaluations.
Worth borrowing
- Separate evaluation into two pipelines: one for agent response quality, one for recommendation accuracy
- Include a 'How to improve numbers' section that directly links to actionable code changes
Related Documents
IATA BCBP Standard Compliance
Documents which IATA BCBP fields and barcode formats a Swift library implements, including Version 8 gender code support.
Evaluating AI Agent Systems: Metrics, Benchmarks, and Quality Assurance (2024-2026)
Surveys 2024-2026 metrics, benchmarks, and monitoring tools for evaluating AI agent systems, with recommendations for a self-improving coding agent.
Voice AI Leaderboards, Benchmarks, and Evaluation Gaps (Jan 2025 -- Feb 2026)
Surveys 20+ voice AI benchmarks from Jan 2025, Feb 2026, identifies evaluation gaps, and provides leaderboard data for STT, TTS, and end-to-end voice agents.
LLM Evaluation — Deep Dive
Covers why LLM evaluation is hard, benchmark taxonomy, LLM-as-judge methodology, contamination, robustness, and designing an eval suite for a product.