Quality Assurance
EVALS.md · 100 documents
Evaluation of RAG Systems + Presentation Outline
How do we evaluate our RAG system?
After LangGraph node execution, convert messages
**RAGAS** (Retrieval-Augmented Generation Assessment) is a specialized evaluation framework designed to measure RAG pipeline performance through reference-free metrics, making it ideal for production systems. **LangGraph** is a state-based orchestration framework that structures AI workflows as directed graphs. Integrating these two creates a powerful system for building and evaluating complex RAG pipelines systematically.
Topic: Evaluation & Benchmarking
Evaluation is widely considered the **hardest unsolved problem** in LLM engineering. Unlike traditional software where a unit test returns pass/fail, LLM outputs are probabilistic, open-ended, and context-dependent -- there is no single "correct" answer for most tasks. Yet every production decision depends on evaluation: which model to deploy, whether a prompt change improved quality, whether a RAG pipeline is hallucinating less after a reranker upgrade. By mid-2025, benchmark saturation (fronti
Evaluation Framework
This document describes how Agent Invest measures quality, detects regressions, and ensures safety. The system uses three evaluation layers: online scoring (every production run), offline evaluation (golden dataset), and guardrails (real-time safety checks).
[BEE-30004] Evaluating and Testing LLM Applications
title: Evaluating and Testing LLM Applications
PRD-010 — Evaluation Framework
title: Evaluation Framework
Agent Evaluation Reference Guide
Complete documentation for the `agent-eval` CLI, metrics, data formats, and customization.
The Evals Gap
It doesn't matter how beautiful your theory is, <br>
RAG Evaluation Guide
This guide explains how to evaluate the RAG (Retrieval-Augmented Generation) performance of the Clarity and Rigor agents using different retriever configurations.
NEAR Protocol Project Scoring Rubric
- **16-20 points**: Deep integration, NEAR standards usage, wallet integration, on-chain innovation
Evaluation Metrics for Language Model Comparative Analysis
This document defines the metrics used to evaluate the performance of different language models in generating Python game scripts. The metrics focus on three key areas: Accuracy, Bug Frequency, and Feature Completeness.
Text Retrieval Benchmark
Run a text retrieval benchmark without generation (no LLM required).
Retrieval Evaluation for fasten-answers-ai
This document provides step-by-step instructions on how to evaluate the retrieval component of the system. Follow the instructions below carefully to ensure correct execution and evaluation.
Architecture - Requirements Review Agent
You will be reviewing the Requirements and the responses to an RFP
✅ DANH SÁCH HOÀN THÀNH - CLOUD RESOURCE CLASSIFIER
- [x] `CloudResourceApplication.java` - Main application
Plan Evaluation Report
**Evaluation Date:** December 2025
LLMTrace Implementation TODO
**Generated from:** `docs/FEATURE_ROADMAP.md`
03 — Evaluation Techniques Deep Dive: LLM-as-Judge, Human Evaluation & Retrieval Metrics
> **Interview Reality:** _"What evaluation techniques do you use and why?"_
evaluating-llm
title: 'Evals - testing AI agents'
Day 4: Agent Quality - Observability & Evaluation
Day 4 teaches you how to ensure **agent quality** through two complementary approaches:
Quick start: Evaluation sets
In the previous guide, we showed you how to collect assessments and measure metrics using those assessments. While a logical next step is to jump into creating a new version of the application to fix any issues you identified, Databricks suggests first creating an Evaluation Set, which is a represensative set of questions you expect your users to ask of your application, optionally with ground-truth answers to those questions.
Architecturally Significant Requirements — Financial Document Analyst
- **Category**: Performance
From Heuristics to Hybrid: A Methodology for Building a Testable, Sequential Log Anomaly Detection Engine
**Authors:** [Author Names]
Evaluation Strategy
Evaluation is a first-class architectural concern, not an afterthought. The system uses a four-tier eval taxonomy, ordered by reliability. Deterministic checks handle clinical correctness and safety. LLM-as-judge is scoped strictly to subjective quality dimensions where programmatic checks are insufficient.