All Documents
3,528 documents available
Claim Extraction Evaluation Matrix
Defines a matrix evaluation framework for comparing YouTube claim extraction quality across multiple LLM models and video types.
LLM Evaluation & Metrics — Complete Guide
Covers the full spectrum of LLM evaluation metrics, from perplexity and BLEU to RAGAS and LLM-as-judge, with production pipeline code.
Traditional function - easy to test
Introduces AI model evaluation concepts, metrics, and code examples for measuring accuracy, reliability, robustness, efficiency, and safety.
Evaluation and Benchmarking
Explains why LLM evaluation is hard, defines 10+ standard benchmarks and metrics, and provides code for pass@k, BERTScore, and Elo simulation.
評估系統
Documents a multi-layered evaluation system for LLM-generated SEO Q&A, covering LLM-as-Judge, reasoning models, retrieval metrics, maturity assessment, and pipeline step-by-step golden sets.
13-02-PLAN
Defines 100 annotated queries for retrieval evaluation and a CSV export service that writes aggregate and detailed metrics to configurable output.
When "Better" Prompts Hurt: Evaluation-Driven Iteration for LLM Applications
Presents an evaluation-driven workflow (Define-Test-Diagnose-Fix) for LLM applications and demonstrates that generic prompt improvements can degrade task-specific performance.
Evaluation Fundamentals
Explains why LLM evaluation is hard and how to build a golden dataset, an eval pipeline, and offline plus online monitoring.
Evaluation and Observability
Defines evaluation methodology, monitoring signals, and feedback loops for LLM applications in production.
Evaluation Findings (Sample Run)
Reports results from two evaluation pipelines: agent response scoring and recommendation accuracy, with aggregate metrics and improvement steps.
EVALS.md — LLM & RAG Evaluation Playbook
Guides model selection, regression testing, and production-grade evaluation for LLMs and RAG systems with research-backed practices.
Chapter 12 — Verification: evaluation inside the loop
Explains why self-verification fails and defines a six-layer verification stack with binary verifiers and score-based evaluators for agent loops.
Evaluation Strategy
Defines a four-tier evaluation taxonomy for a clinical reasoning system, prioritizing deterministic checks over LLM-as-judge for safety and correctness.
When "Better" Prompts Hurt: Evaluation-Driven Iteration for LLM Applications
Presents an evaluation-driven workflow for LLM applications with a Minimum Viable Evaluation Suite (MVES) framework and shows that generic prompt improvements can degrade structured task performance.
Exact match
Covers eval types, metrics, LLM-as-judge patterns, regression testing, CI integration, and tooling for testing non-deterministic AI systems.
Evaluation Framework Reference
Defines an evaluation framework with scoring metrics, cost/latency/safety measurement, LLM-as-a-judge setup, CI/CD integration, benchmark selection, and contamination prevention.
Lesson 01: Evaluation Frameworks Overview
Teaches a taxonomy of LLM evaluation approaches and compares five frameworks for offline and online testing.
🎯 Phase 4 特別計畫：次世代學術評估引擎實作計畫書 (v1.0)
Defines a multi-dimensional 1-10 scoring system with weighted metrics for automated academic answer evaluation, including retry logic and integration points.
LLM Evaluation & Benchmarking
Covers why and how to evaluate generative LLMs, from offline metrics to production A/B testing.
LLM Evaluation — Deep Dive
Covers why LLM evaluation is hard, benchmark taxonomy, LLM-as-judge methodology, contamination, robustness, and designing an eval suite for a product.
The Evaluation & Optimization Loop
Describes a closed-loop system that generates synthetic data, applies human review, optimizes routing with DSPy, evaluates results, and annotates telemetry for continuous improvement.
LLM Evaluation — Interview Grill
Presents 115 active-recall questions across 14 sections to test and reinforce knowledge of LLM evaluation concepts.
Golden Set — Medical-Retrieval Probes
Defines a protocol for building a medical-retrieval golden set that measures how well a hybrid pipeline returns context for proactive guidance turns.
Evaluation Methodology — Practitioner Reference
Distills evaluation methods for open-ended AI systems into actionable guidance with code examples and decision trees.