All Documents
3,528 documents available
LLM Evaluation & Benchmarking
Covers why and how to evaluate generative LLMs, from offline metrics to production A/B testing.
Discussion Report: Flat Band Evaluation Methodology & Strategic Direction
Documents a team's decision to replace manual weight tuning with inverse estimation from expert-labeled flat band pairs.
Phase 1.1 Evaluation Framework - Progress Report
Tracks progress building a 100-question golden set and running RAGAS benchmarks for a Buddhist sutra Q&A system.
evaluation/ — Evaluation Skill
Defines an offline evaluation framework that tests a WHO IMCI classification system against 20+ golden scenarios and produces versioned accuracy reports.
Exact match
Covers eval types, metrics, LLM-as-judge patterns, regression testing, CI integration, and tooling for testing non-deterministic AI systems.
Design an Evaluation Pipeline for an LLM-Based Product
Designs an end-to-end evaluation pipeline for LLM-based products, covering offline benchmarks, online A/B tests, safety testing, and human evaluation.
When "Better" Prompts Hurt: Evaluation-Driven Iteration for LLM Applications
Presents an evaluation-driven workflow for LLM applications with a Minimum Viable Evaluation Suite (MVES) framework and shows that generic prompt improvements can degrade structured task performance.
Evaluation Framework Reference
Defines an evaluation framework with scoring metrics, cost/latency/safety measurement, LLM-as-a-judge setup, CI/CD integration, benchmark selection, and contamination prevention.
RAG Evaluation: Complete Guide to Evaluating Retrieval-Augmented Generation Systems
Teaches how to evaluate RAG systems using faithfulness, relevancy, precision, and recall metrics with code examples.
5_Evaluation
Explains why evaluation matters for search and RAG systems, covers metrics like nDCG and Recall@K, and describes golden test sets, demos, and user feedback.
PRD-038: Evaluation, Safety, And Rollout
Defines evaluation metrics, gate criteria, and rollout phases for a face-recognition matcher upgrade, with safety invariants and required tests.
RAG Evaluation Frameworks: Comprehensive Analysis for VERA
Surveys RAG evaluation frameworks and metrics, focusing on grounding verification for a verification engine called VERA.
Air-Gapped RAG: Grounding, Citations, and Evaluation
Defines a three-layer quality loop for air-gapped RAG: grounding prompts with citations, post-generation faithfulness checks, and offline evaluation using local LLM judges.
Epic 3: Working Memory, Evaluation & Production Readiness
Defines 12 stories to bring a cognitive memory MCP system into production with monitoring, resilience, cost control, and stability validation.
LLM as a Judge
Shows how to configure Promptfoo for LLM-as-a-judge evaluations using rubric prompts, model-graded scoring, multi-judge voting, and injection-safe judge prompts.
Evaluating the RAG answer quality
Walks through deploying an evaluation model, generating ground truth, and running bulk evaluations on RAG answer quality.
Evaluating Retrieval Augmented Generation - a framework for assessment
Proposes a four-level framework for evaluating RAG systems: model, data ingestion, semantic retrieval, and end-to-end.
Agent and LLM Evaluation Practices
Organises LLM and agent evaluation into six dimensions, three metric families, five techniques, and AWS-native tooling.
Agent Quality & Evaluation
Defines a quality evaluation framework for AI agents covering failure modes, four pillars, evaluation hierarchy, observability, and a continuous improvement flywheel.
DR-ICU App Review Guide
Guides testers through reviewing a medical reference app for emergency and ICU professionals, covering navigation, features, and device compatibility.
ProtoExtract — Evaluation Approach Using OmniDocBench Methodology
Defines a multi-level evaluation framework for clinical protocol table extraction, adapting OmniDocBench metrics and adding domain-specific accuracy measures.
Evaluation and Testing LLM Systems
Lays out a four-layer evaluation stack for LLM systems, from automated assertions through human review, with concrete testing practices.
evaluation_driven_development
Introduces evaluation-driven development for AI, treating evals as first-class artifacts to measure output quality and guard against regressions.
Evaluation Benchmarking Framework for Token-Optimized RAG
Evaluation & Benchmarking Framework for TokenOptimized RAG