All Documents
3,528 documents available
Architecturally Significant Requirements — Financial Document Analyst
Catalogues seven architecturally significant requirements for a financial RAG system, each with priority, source, and architectural impact.
Landing Page Copy: Claru
Provides a complete set of copy and structure for a landing page that positions Claru as an embedded expert intelligence service for AI labs.
Testing
Documents a 3,292-test suite for a markdown vault tool, covering performance, concurrency, fuzzing, security, and retrieval benchmarks against HotpotQA and LoCoMo.
Analysis of Paper 50
Structured analysis of a research paper's use of BigCloneBench, answering 12 specific questions about dataset usage and validity.
evaluating-llm
Introduces evaluation concepts for LLMs and agents, then provides a TypeScript framework with scorers, experiment tracking, and JSON persistence.
Advanced RAG Techniques
Surveys advanced RAG architectures, Agentic, Corrective, Self-RAG, Graph RAG, RAPTOR, Multimodal, Temporal, with design trade-offs and decision guidance.
Training Readiness Checklist
Lists five gates to verify before launching a training or tuning run, with checkboxes for each item.
Weeks 3–4: Production RAG & Evaluation
Teaches production RAG design, chunking, embedding, vector search, reranking, and evaluation with RAGAS and LangSmith over two weeks.
-PLUIE: Personalisable metric with Llm Used for Improved Evaluation
Lists 16 arXiv papers on LLM evaluation, fine-tuning, and interpretability, each with title, authors, link, and abstract.
From Heuristics to Hybrid: A Methodology for Building a Testable, Sequential Log Anomaly Detection Engine
Describes a sequential rule-then-ML log anomaly detector and a manual golden set evaluation to avoid circular validation.
Day 4 - Podcast Transcript
Summarises a white paper on evaluating AI agent quality, covering pillars, observability, and a continuous improvement flywheel.
Quick start: Evaluation sets
Walks through creating an evaluation set of questions and optional ground-truth answers to measure RAG application quality before stakeholder review.
Configuration Reference
Guides an LLM Evaluator agent through designing evaluation frameworks, test plans, and quality gate decisions for AI systems.
Evaluation Findings (Sample Run)
Reports results from two evaluation pipelines: agent response scoring and recommendation accuracy, with aggregate metrics and improvement steps.
Superpipe Studio
Introduces a free observability and experimentation app for Superpipe pipelines, with logging, dataset management, and experiment tracking.
Repository Intelligence: Building the Next Generation of Agent Evaluation Data
Describes a pipeline for generating synthetic evaluation data that tests repository-level agent capabilities across QA, code generation, and debugging tasks.
Evaluation Strategy
Defines a four-tier evaluation taxonomy for a clinical reasoning system, prioritizing deterministic checks over LLM-as-judge for safety and correctness.
── 数据结构定义 ──────────────────────────────────────────────
Explains why agent evaluation is harder than model evaluation and provides a five-dimension framework plus a runnable Python evaluation class.
Agent Eval - Complete UI Flow Visualization
Maps every screen of an agent evaluation application to its URL, UI layout, and the evaluation features it implements.
LLM Judge — Setup & Operations
Explains how to enable and configure a three-tier LLM judge cascade for prompt-injection detection, with shadow-mode rollout and golden-set calibration.
LLM Evaluation Overview
Surveys evaluation methods for LLMs: automated metrics, human eval, AI-as-judge, and task-specific checks, with guidance on dataset design and cadence.
Claim Extraction Evaluation Matrix
Defines a matrix evaluation framework for comparing YouTube claim extraction quality across multiple LLM models and video types.
LLM-as-Judge Reliability Patterns
Documents five reliability patterns for LLM-as-Judge systems and maps them against gaps in the LLMTrace security judge implementation.
When "Better" Prompts Hurt: Evaluation-Driven Iteration for LLM Applications
Presents an evaluation-driven workflow (Define-Test-Diagnose-Fix) for LLM applications and demonstrates that generic prompt improvements can degrade task-specific performance.