Recently Added
3,528 documents
Model and Prompt Evals for a Clinical Charting App Using LLMs in Production
Runs 10 iterations of 8 OpenAI and Anthropic models against a single transcript to compare cost, speed, and output quality for structured SOAP note generation.
Pull Request Review Rubric
Defines a multi-tier rubric for reviewing pull requests against AWS Well-Architected standards across seven architectural dimensions.
Introduction
Defines when WG21 proposals should be reviewed by SG16, the text and Unicode study group.
FIFO Evals Client - Migration Guide
Replaces a synchronous evaluation script that times out on large jobs with an async FIFO queue client that handles up to 10,000 prompts without failure.
Code Challenge 4 Sanitized Rubric
Provides a rubric for grading a React code challenge, with scoring criteria for props, state, code structure, and rendering.
JP Tasks
Runs Japanese language model evaluations across 12 tasks, covering QA, NLI, classification, and summarization benchmarks.
Eval Learnings
Documents six evaluation rounds that validate parameter recovery, autodiff correctness, influence function assembly, and frequentist coverage for a deep inference package.
Instructions to the Evaluator (delete this section)
Provides a peer-evaluation rubric for student teams to assess and improve each other's open-source project materials.
proj1rubric
Scores a software project against a 90-item rubric covering documentation, testing, community, and licensing practices.
Ralph vs Oneshot: Capability Eval Results
Compares two agent harnesses (Ralph vs oneshot) on coding tasks, finding marginal quality gains at 1.5-5x higher cost.
How Zapier 4x'd Their AI Feedback Collection (Vitor)
Summarizes a talk on how Zapier increased AI feedback collection 4x through UX changes and systematic analysis.
llm.md — LLM-Assisted Semantic Repair (DP3)
Specifies an optional LLM-assisted repair step that converts DP2 FAIL outcomes into validated PATCH or NOOP proposals while preserving determinism and safety.
Responsible AI (RAI)
Documents where AI is used, model choices, guardrails, cost controls, and risk mitigations for an LLM-based system.
The ustack: Complete Reference for LLMs
Documents a nine-library web stack built on radical minimalism, data-structure DSLs, and an ES5 subset, with under 3300 total lines.
POS Tagging with LLMs: The Hard Parts
Guides students through building and comparing classical and LLM-based POS taggers, including error analysis and segmentation challenges.
Product Requirements Document: Unleash MCP Server
Defines the product vision, goals, and phased delivery plan for an MCP server that creates and wraps code changes with Unleash feature flags.
DFAH: Determinism-Faithfulness Assurance Harness
A harness for measuring whether LLM agents produce consistent, auditable behavior when given the same input multiple times.
RAG-Lighter Wiki
Documents a modular Python RAG framework with support for multiple LLMs, vector stores, embeddings, and evaluation.
Fission — NEAR Track
Explains how Fission's AI agent meets NEAR track requirements for autonomous governance and data analysis on the NEAR AI Agent Hub.
TODO: Update Tag Filtering Logic in Featured Agents
Outlines steps to deduplicate displayed filter tags in a React component by mapping aliases to a primary tag.
Logging & Evals
Documents a trigger-based logging system, cost tracking, token analytics, and a separate evals dashboard for reviewing LLM interactions.
CDR Evaluation
> How we measure output quality, and what the numbers mean.
Software Carbon Intensity for AI Specification
Extends the SCI methodology to AI systems, defining how to measure and report carbon emissions across AI lifecycles and personas.
Comprehensive Agentic AI Learning Plan for ML Engineers
Outlines an 8-week curriculum for ML engineers to build agentic AI systems, from basic agents to production deployment.