Recently Added
3,528 documents
IR-Copilot — Incident Response AI Assistant
Serves as a reference implementation for an agentic RAG system with multi-tenant chat, document ingestion, hybrid search, and subagent delegation.
Using Performance Metrics to Evaluate RAG Systems
Explains how to evaluate RAG systems using Top-K parameter optimization and auto prompt optimization with Qdrant and Relari.
RAG System Testing Methodologies: A Comprehensive Guide
Surveys RAG evaluation frameworks, metrics, and testing methodologies with implementation guidance and tool comparisons.
[BEE-30004] Evaluating and Testing LLM Applications
Explains why traditional testing fails for LLMs and gives a practical evaluation strategy combining offline metrics, LLM-as-judge, red-teaming, and production monitoring.
LLM Evaluation
Guides building an LLM evaluation pipeline with automated metrics, LLM-as-judge, A/B testing, and CI regression tests.
Understanding the Sources of Uncertainty - and Why Our Evals are Biased
Catalogues 10 sources of error in AI evaluations, bias, noise, and their compounding effects, with worked examples and statistical intuition.
📈 Trading RAG Mentor
Answers trading questions using your own video transcripts via a RAG pipeline with ChromaDB, LangChain, and a choice of Groq or Ollama LLMs.
🧠 Big Picture
Explains why API Gateway non-proxy integration with mapping templates is the correct answer for a GenAI model routing exam question, contrasting it with three incorrect options.
Day 20: Evaluation & Benchmarks 📏
Teaches evaluation of generative AI outputs using n-gram metrics, LLM-as-judge, and RAGAS, with runnable code examples for each technique.
Data-Driven RAG Evaluation: Testing Qdrant Apps with Relari AI
Explains how to evaluate RAG applications using Qdrant and Relari, covering Top-K parameter tuning and Auto Prompt Optimization with code examples.
Metrics
Documents evaluation metrics for RAG systems, covering retrieval and generation quality, plus guidance on building golden datasets.
Agents and LLMs
Serves as a personal reference notebook covering LLM families, fine-tuning, RAG, agents, evaluation, and applied AI topics.
Evaluation Scripts
Provides two Python scripts: one interactively builds a golden test set of queries, the other evaluates retrieval quality using IR metrics.
GenAI Benchmarks & Evaluation — Product-Based Companies
Understanding how to **benchmark, evaluate, and compare LLMs** is essential for roles at Google, OpenAI, Anthropic, Cohere, and AI research teams. This file covers the most important benchmarks, evaluation methodologies, and how to build custom evaluation harnesses.
Evaluation Framework
Defines a three-layer evaluation system for an AI investment assistant: guardrails, online RAGAS scoring, and offline golden dataset regression tests.
LLM Evaluation
Guides building an LLM eval pipeline with datasets, automated metrics, LLM-as-judge, A/B testing, and CI regression tests.
Decodable Story Quality Rubric
Guides writers of decodable readers through six story-quality checks, balancing phonics constraints with natural dialogue, clear setup, logical actions, emotional arc, and payoff.
RAG Evaluation Guide
Defines a repeatable workflow to evaluate RAG retrieval and answer quality across multiple retriever configurations using RAGAS and custom metrics.
QualRubric
Defines a qualitative rubric for PhD qualifying exam presentations, focusing on context, organization, and knowledge demonstration rather than numerical scoring.
Instructions for Claude Code: n8n Meal Feedback LLM Evaluation Workflow
Guides building an n8n workflow that uses a thinking model to generate ground truth answers and evaluates cheaper prompts against them.
Prompt Testing Skill
Defines a complete prompt testing and LLM evaluation workflow with hallucination detection, regression, A/B testing, and safety guardrails.
EGG Rubric: Corporate Sustainability Evaluation Framework
Provides a detailed scoring rubric for evaluating corporate sustainability across five themes using commitment strength, specificity, and temporal evolution.
Code Span Semantic Chunking Executive Summary
Recommends an AST-driven chunking strategy for code retrieval, preserving function/class boundaries to improve precision and reduce token waste.
Module 6: Synthesis
Guides you through synthesizing a RAG evaluation project into portfolio artifacts, stakeholder presentations, and technical handoffs.