All Documents
3,528 documents available
RAG System Testing Methodologies: A Comprehensive Guide
Surveys RAG evaluation frameworks, metrics, and testing methodologies with implementation guidance and tool comparisons.
Evaluation Framework
Defines a three-layer evaluation system for an AI investment assistant: guardrails, online RAGAS scoring, and offline golden dataset regression tests.
AI Tester Interview Preparation Guide
Prepares candidates for an AI Tester interview focused on LLM, API, and automation testing in the pharmaceutical industry.
Day 20: Evaluation & Benchmarks 📏
Teaches evaluation of generative AI outputs using n-gram metrics, LLM-as-judge, and RAGAS, with runnable code examples for each technique.
[BEE-30004] Evaluating and Testing LLM Applications
Explains why traditional testing fails for LLMs and gives a practical evaluation strategy combining offline metrics, LLM-as-judge, red-teaming, and production monitoring.
Understanding the Sources of Uncertainty - and Why Our Evals are Biased
Catalogues nine sources of error in AI evaluations, bias and noise, with concrete examples and statistical intuition for each.
Evaluation Scripts
Provides two Python scripts: one interactively builds a golden test set of queries, the other evaluates retrieval quality using IR metrics.
Understanding the Sources of Uncertainty - and Why Our Evals are Biased
Catalogues 10 sources of error in AI evaluations, bias, noise, and their compounding effects, with worked examples and statistical intuition.
🧠 Big Picture
Explains why API Gateway non-proxy integration with mapping templates is the correct answer for a GenAI model routing exam question, contrasting it with three incorrect options.
LLM Evaluation
Guides building an LLM evaluation pipeline with automated metrics, LLM-as-judge, A/B testing, and CI regression tests.
Metrics
Documents evaluation metrics for RAG systems, covering retrieval and generation quality, plus guidance on building golden datasets.
IR-Copilot — Incident Response AI Assistant
Serves as a reference implementation for an agentic RAG system with multi-tenant chat, document ingestion, hybrid search, and subagent delegation.
Agents and LLMs
Serves as a personal reference notebook covering LLM families, fine-tuning, RAG, agents, evaluation, and applied AI topics.
Data-Driven RAG Evaluation: Testing Qdrant Apps with Relari AI
Explains how to evaluate RAG applications using Qdrant and Relari, covering Top-K parameter tuning and Auto Prompt Optimization with code examples.
📈 Trading RAG Mentor
Answers trading questions using your own video transcripts via a RAG pipeline with ChromaDB, LangChain, and a choice of Groq or Ollama LLMs.
GenAI Benchmarks & Evaluation — Product-Based Companies
Understanding how to **benchmark, evaluate, and compare LLMs** is essential for roles at Google, OpenAI, Anthropic, Cohere, and AI research teams. This file covers the most important benchmarks, evaluation methodologies, and how to build custom evaluation harnesses.
Project 2: Full-Stack Application
Defines a graded full-stack project assignment requiring AI modalities, authentication, CI/CD, and Agile documentation.
QualRubric
Defines a qualitative rubric for PhD qualifying exam presentations, focusing on context, organization, and knowledge demonstration rather than numerical scoring.
Decodable Story Quality Rubric
Guides writers of decodable readers through six story-quality checks, balancing phonics constraints with natural dialogue, clear setup, logical actions, emotional arc, and payoff.
RAG Evaluation Guide
Defines a repeatable workflow to evaluate RAG retrieval and answer quality across multiple retriever configurations using RAGAS and custom metrics.
Module 6: Synthesis
Guides you through synthesizing a RAG evaluation project into portfolio artifacts, stakeholder presentations, and technical handoffs.
Reddit Virality Grading Rubric
Defines a weighted scoring system for predicting Reddit post virality, used by an LLM to grade rumours before simulation.
Sprint 0 Marking Scheme
Provides a grading rubric for a Sprint 0 deliverable, scoring summary, competition, backlog, setup, documentation, definition of done, personas, process, and user experience.
Instructions for Claude Code: n8n Meal Feedback LLM Evaluation Workflow
Guides building an n8n workflow that uses a thinking model to generate ground truth answers and evaluates cheaper prompts against them.