Data & Analytics
EVALS.md ยท 41 documents
Using Performance Metrics to Evaluate RAG Systems
Explains how to evaluate RAG systems using Top-K parameter optimization and auto prompt optimization with Qdrant and Relari.
Data-Driven RAG Evaluation: Testing Qdrant Apps with Relari AI
Explains how to evaluate RAG applications using Qdrant and Relari, covering Top-K parameter tuning and Auto Prompt Optimization with code examples.
Day 20: Evaluation & Benchmarks ๐
Teaches evaluation of generative AI outputs using n-gram metrics, LLM-as-judge, and RAGAS, with runnable code examples for each technique.
Instructions for Claude Code: n8n Meal Feedback LLM Evaluation Workflow
Guides building an n8n workflow that uses a thinking model to generate ground truth answers and evaluates cheaper prompts against them.
Using Performance Metrics to Evaluate RAG Systems
Walks through evaluating RAG systems with Qdrant and Relari, covering Top-K parameter tuning and auto prompt optimization.
Client Data Mapping to Existing System
Maps 8 client data tables to existing Airtable tables and identifies 8 new tables plus field-level mappings for a migration.
apt_juror_5
Defines metrics, ground truth, audit, and regression plans for evaluating an apartment listing crawler and ranking system.
LLM Benchmark Report
Documents benchmark results for an LLM agent on an NVIDIA MLOps platform, covering RAGAS, LLM-as-judge, and A/B prompt testing.
Repository Intelligence: Building the Next Generation of Agent Evaluation Data
Describes a pipeline for generating synthetic evaluation data that tests repository-level agent capabilities across QA, code generation, and debugging tasks.
EVALS.md โ LLM & RAG Evaluation Playbook
Guides model selection, regression testing, and production-grade evaluation for LLMs and RAG systems with research-backed practices.
13-02-PLAN
Defines 100 annotated queries for retrieval evaluation and a CSV export service that writes aggregate and detailed metrics to configurable output.
LLM-as-Judge Reliability Patterns
Documents five reliability patterns for LLM-as-Judge systems and maps them against gaps in the LLMTrace security judge implementation.
LLM Evaluation & Metrics โ Complete Guide
Covers the full spectrum of LLM evaluation metrics, from perplexity and BLEU to RAGAS and LLM-as-judge, with production pipeline code.
Exact match
Covers eval types, metrics, LLM-as-judge patterns, regression testing, CI integration, and tooling for testing non-deterministic AI systems.
5_Evaluation
Explains why evaluation matters for search and RAG systems, covers metrics like nDCG and Recall@K, and describes golden test sets, demos, and user feedback.
Golden Set โ Medical-Retrieval Probes
Defines a protocol for building a medical-retrieval golden set that measures how well a hybrid pipeline returns context for proactive guidance turns.
Knowledge MCP Query Reference for Evaluation Timing
Teaches you to query a Knowledge MCP server for evaluation placement, methods, anti-patterns, and decision support, with query templates and result interpretation.
Input Data Format Reference
Defines JSON schemas for a marketing experiment system: product registry, context, hypotheses, creative variants, and metrics.
Marketing Analytics Dashboard - User Guide
Teaches a marketing team how to read and act on a dashboard with 8 KPI cards, channel graphs, and campaign tables.
ANALYTICS CENTER
Defines metrics, dashboards, post analytics, platform comparisons, growth tracking, AI insights, and report generation for a multi-platform marketing analytics module.
After-Action Report: Sports Analytics Framework
Documents a full-stack analytics overhaul that replaced opaque composite metrics with field-position-conditioned conversion rates and a delta profile.
unsigned_map_benchmarks
Compares performance of fea::unsigned_map and fea::flat_unsigned_hashmap against std::map and std::unordered_map across insert, erase, copy, clear, and iterate operations.
erikbern.ann-benchmarks
Benchmarks approximate nearest neighbor algorithms on high-dimensional datasets using Docker containers and precomputed ground truth.
soumith.convnet-benchmarks
Benchmarks forward and backward pass times for convolutional neural network implementations across multiple deep learning libraries on a specific GPU.