All Documents
3,528 documents available
RAG Deep Dive Part 7: Evaluation and Debugging RAG Systems
Teaches you to measure and debug a RAG pipeline with retrieval metrics, generation metrics, and production monitoring, all implemented from scratch in Python.
LLM Evaluation
Guides building an LLM eval pipeline with datasets, automated metrics, LLM-as-judge, A/B testing, and CI regression tests.
AI Tester Interview Preparation Guide
Prepares candidates for an AI Tester interview focused on LLM, API, and automation testing in the pharmaceutical industry.
Data-Driven RAG Evaluation: Testing Qdrant Apps with Relari AI
Explains how to evaluate RAG applications using Qdrant and Relari, covering Top-K parameter tuning and Auto Prompt Optimization with code examples.
GenAI Benchmarks & Evaluation — Product-Based Companies
Understanding how to **benchmark, evaluate, and compare LLMs** is essential for roles at Google, OpenAI, Anthropic, Cohere, and AI research teams. This file covers the most important benchmarks, evaluation methodologies, and how to build custom evaluation harnesses.
Agents and LLMs
Serves as a personal reference notebook covering LLM families, fine-tuning, RAG, agents, evaluation, and applied AI topics.
Topic: Evaluation & Benchmarking
Surveys evaluation and benchmarking for LLMs, covering metrics, methods, pipelines, and trade-offs across offline and online settings.
Day 20: Evaluation & Benchmarks 📏
Teaches evaluation of generative AI outputs using n-gram metrics, LLM-as-judge, and RAGAS, with runnable code examples for each technique.
LLM Evaluation
Guides building an LLM evaluation pipeline with automated metrics, LLM-as-judge, A/B testing, and CI regression tests.
🧠 Big Picture
Explains why API Gateway non-proxy integration with mapping templates is the correct answer for a GenAI model routing exam question, contrasting it with three incorrect options.
Research Report: Using LLMs as Oracle for Entity Matching Ground Truth
Summarises research on using LLMs (DeepSeek, GPT-4, Claude) for entity matching ground truth, covering accuracy, prompt engineering, ensembles, cost analysis, validation, and regression test conversion.
Evaluation Framework
Defines a three-layer evaluation system for an AI investment assistant: guardrails, online RAGAS scoring, and offline golden dataset regression tests.
After LangGraph node execution, convert messages
Explains how to integrate RAGAS evaluation metrics into a LangGraph RAG pipeline, covering offline and online evaluation strategies.
RAG System Testing Methodologies: A Comprehensive Guide
Surveys RAG evaluation frameworks, metrics, and testing methodologies with implementation guidance and tool comparisons.
Understanding the Sources of Uncertainty - and Why Our Evals are Biased
Catalogues nine sources of error in AI evaluations, bias and noise, with concrete examples and statistical intuition for each.
Project Memory
Documents a production-ready RAG system for Paris cultural events with hybrid search, LLM generation, and Streamlit UI.
Reddit Virality Grading Rubric
Defines a weighted scoring system for predicting Reddit post virality, used by an LLM to grade rumours before simulation.
Project 2: Full-Stack Application
Defines a graded full-stack project assignment requiring AI modalities, authentication, CI/CD, and Agile documentation.
Using Performance Metrics to Evaluate RAG Systems
Walks through evaluating RAG systems with Qdrant and Relari, covering Top-K parameter tuning and auto prompt optimization.
Thesis Falsifier
Defines a RAG-based web app that generates 19-point falsification assessments of research papers from uploaded PDFs.
QualRubric
Defines a qualitative rubric for PhD qualifying exam presentations, focusing on context, organization, and knowledge demonstration rather than numerical scoring.
Judging Rubric
Defines a 100-point scoring rubric for an AI-for-social-good hackathon, covering five criteria with detailed score bands and tie-breaking rules.
Criteria 1: Quality of Exploratory Data Analysis (20%) [20]
Defines a 3-criteria marking rubric for a logistic regression assignment, each with four grade bands and specific descriptors.
Module 6: Synthesis
Guides you through synthesizing a RAG evaluation project into portfolio artifacts, stakeholder presentations, and technical handoffs.