All Documents
3,528 documents available
Analysis of Paper 50
Structured analysis of a research paper's use of BigCloneBench, answering 12 specific questions about dataset usage and validity.
Day 4: Agent Quality - Observability & Evaluation
Teaches observability and evaluation for AI agents, covering logs, traces, metrics, plugins, and automated testing with the ADK framework.
evaluating-llm
Introduces evaluation concepts for LLMs and agents, then provides a TypeScript framework with scorers, experiment tracking, and JSON persistence.
Day 4 - Podcast Transcript
Summarises a white paper on evaluating AI agent quality, covering pillars, observability, and a continuous improvement flywheel.
Code indexing for AI agents: summarization strategies and evaluation systems
Synthesises 2024-2025 research on code indexing for AI agents, covering summarisation strategies, hybrid retrieval architectures, and evaluation benchmarks.
Training Readiness Checklist
Lists five gates to verify before launching a training or tuning run, with checkboxes for each item.
Page Architecture: Claru
Defines a seven-section landing page layout with animated hero, comparison grid, testimonials, and a terminal-style progressive disclosure form.
From Heuristics to Hybrid: A Methodology for Building a Testable, Sequential Log Anomaly Detection Engine
Describes a sequential rule-then-ML log anomaly detector and a manual golden set evaluation to avoid circular validation.
Quick start: Evaluation sets
Walks through creating an evaluation set of questions and optional ground-truth answers to measure RAG application quality before stakeholder review.
-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Lists 16 arXiv papers on LLM evaluation, fine-tuning, and interpretability, each with title, authors, link, and abstract.
apt_juror_5
Defines metrics, ground truth, audit, and regression plans for evaluating an apartment listing crawler and ranking system.
lib-ai-app-community-rag
Collects community insights, leaderboard links, and production war stories for building RAG systems, especially with multi-modal and large-scale documents.
Evaluation and Benchmarking
Explains why LLM evaluation is hard, defines 10+ standard benchmarks and metrics, and provides code for pass@k, BERTScore, and Elo simulation.
Evaluating AI Agent Systems: Metrics, Benchmarks, and Quality Assurance (2024-2026)
Surveys 2024-2026 metrics, benchmarks, and monitoring tools for evaluating AI agent systems, with recommendations for a self-improving coding agent.
Claim Extraction Evaluation Matrix
Defines a matrix evaluation framework for comparing YouTube claim extraction quality across multiple LLM models and video types.
── 数据结构定义 ──────────────────────────────────────────────
Explains why agent evaluation is harder than model evaluation and provides a five-dimension framework plus a runnable Python evaluation class.
Chapter 12 — Verification: evaluation inside the loop
Explains why self-verification fails and defines a six-layer verification stack with binary verifiers and score-based evaluators for agent loops.
Agent Eval - Complete UI Flow Visualization
Maps every screen of an agent evaluation application to its URL, UI layout, and the evaluation features it implements.
AI System Evaluation & Testing / Đánh Giá và Kiểm Thử Hệ Thống AI
Teaches a four-level evaluation framework, three LLM testing patterns, regression CI pipeline, and production monitoring for AI features.
Configuration Reference
Guides an LLM Evaluator agent through designing evaluation frameworks, test plans, and quality gate decisions for AI systems.
LLM Evaluation Overview
Surveys evaluation methods for LLMs: automated metrics, human eval, AI-as-judge, and task-specific checks, with guidance on dataset design and cadence.
Traditional function - easy to test
Introduces AI model evaluation concepts, metrics, and code examples for measuring accuracy, reliability, robustness, efficiency, and safety.
Superpipe Studio
Introduces a free observability and experimentation app for Superpipe pipelines, with logging, dataset management, and experiment tracking.
Evaluation Fundamentals
Explains why LLM evaluation is hard and how to build a golden dataset, an eval pipeline, and offline plus online monitoring.