Quality Assurance
EVALS.md · 100 documents
Evaluation of RAG Systems + Presentation Outline
Outlines evaluation strategies for a RAG system and proposes a presentation structure for a project on international student queries.
Topic: Evaluation & Benchmarking
Surveys evaluation and benchmarking for LLMs, covering metrics, methods, pipelines, and trade-offs across offline and online settings.
[BEE-30004] Evaluating and Testing LLM Applications
Explains why traditional testing fails for LLMs and gives a practical evaluation strategy combining offline metrics, LLM-as-judge, red-teaming, and production monitoring.
After LangGraph node execution, convert messages
Explains how to integrate RAGAS evaluation metrics into a LangGraph RAG pipeline, covering offline and online evaluation strategies.
Evaluation Framework
Defines a three-layer evaluation system for an AI investment assistant: guardrails, online RAGAS scoring, and offline golden dataset regression tests.
RAG Evaluation Guide
Defines a repeatable workflow to evaluate RAG retrieval and answer quality across multiple retriever configurations using RAGAS and custom metrics.
The Evals Gap
Explains why LLM non-determinism breaks traditional testing and demonstrates the effect with a temperature experiment.
PRD-010 — Evaluation Framework
Specifies how agent output quality is measured, scored, and gated in CI using LangSmith evaluation primitives.
Agent Evaluation Reference Guide
Documents the agent-eval CLI for evaluating ADK agents with deterministic and LLM-as-judge metrics.
Evaluation Metrics for Language Model Comparative Analysis
Defines three metrics, Accuracy, Bug Frequency, Feature Completeness, for evaluating language model generated Python game scripts.
NEAR Protocol Project Scoring Rubric
Defines an 8-category scoring rubric for evaluating NEAR Protocol projects, each with point ranges and AI evaluation prompts.
Retrieval Evaluation for fasten-answers-ai
Walks through evaluating retrieval accuracy, MRR, precision, and recall using Elasticsearch and a Q&A dataset.
Text Retrieval Benchmark
Runs a text retrieval benchmark using BEIR datasets, measuring recall, precision, NDCG, and MRR without LLM generation.
Architecture - Requirements Review Agent
Guides an AI agent through scoring vendor RFP responses against a rubric, generating a gap analysis and reviewer Q&A.
✅ DANH SÁCH HOÀN THÀNH - CLOUD RESOURCE CLASSIFIER
Serves as a completion checklist and presentation readiness tracker for a cloud resource classifier project with backend, frontend, and documentation.
Plan Evaluation Report
Compares two blog plans across 14 criteria, scoring one 87/100 and the other 93/100, and extracts best practices for editorial workflow, SEO, and cost optimization.
LLMTrace Implementation TODO
Tracks 100+ security and infrastructure features for an LLM proxy, each with acceptance criteria anchored to research papers.
Day 4: Agent Quality - Observability & Evaluation
Teaches observability and evaluation for AI agents, covering logs, traces, metrics, plugins, and automated testing with the ADK framework.
Quick start: Evaluation sets
Walks through creating an evaluation set of questions and optional ground-truth answers to measure RAG application quality before stakeholder review.
From Heuristics to Hybrid: A Methodology for Building a Testable, Sequential Log Anomaly Detection Engine
Describes a sequential rule-then-ML log anomaly detector and a manual golden set evaluation to avoid circular validation.
evaluating-llm
Introduces evaluation concepts for LLMs and agents, then provides a TypeScript framework with scorers, experiment tracking, and JSON persistence.
Architecturally Significant Requirements — Financial Document Analyst
Catalogues seven architecturally significant requirements for a financial RAG system, each with priority, source, and architectural impact.
03 — Evaluation Techniques Deep Dive: LLM-as-Judge, Human Evaluation & Retrieval Metrics
Teaches three evaluation approaches for LLM systems, LLM-as-judge, human evaluation, and retrieval metrics, with trade-offs, code, and cost analysis.
── 数据结构定义 ──────────────────────────────────────────────
Explains why agent evaluation is harder than model evaluation and provides a five-dimension framework plus a runnable Python evaluation class.