Evalyn Roadmap

This document tracks planned features and completed work. Future roadmap items are listed first, followed by completed features.

---

Roadmap (Planned Features)

Tracing & Instrumentation

• Multi-modal Tracing - Capture images, audio, video in traces
  • Image input/output capture with thumbnails
  • Audio transcription logging
  • Video frame sampling
  • Base64/URL reference storage options
• Streaming Support - Capture streaming LLM responses
  • Streaming response capture (OpenAI, Anthropic, Gemini via StreamingSpanWrapper)
  • Token-by-token capture with timing
  • First-token latency (TTFT) metric
  • Streaming interruption detection
• More LLM Provider Instrumentors
  • Cohere
  • Mistral
  • AWS Bedrock
  • Azure OpenAI
  • Groq
  • Together AI
  • Replicate
• Framework Instrumentors
  • CrewAI
  • AutoGen
  • DSPy
  • Haystack
  • LlamaIndex
  • Semantic Kernel
• Memory/RAG Tracing - Capture retrieval context and memory operations
  • Capture retrieved documents with relevance scores per query
  • Track vector store lookup latency and result count
  • Link retrieval spans to downstream LLM calls that consume them
  • Memory read/write operation logging for stateful agents
• Async/Parallel Call Tracking - Better support for concurrent LLM calls
  • Detect concurrent spans and render as parallel branches in show-trace
  • Measure total wall-clock vs sum of individual span durations
  • asyncio-native context propagation (ContextVar across await boundaries)
  • Thread-pool executor span grouping
• Trace Export to OTel Backends - Export traces to Jaeger, Zipkin, or any OpenTelemetry collector
  • OTLP gRPC exporter alongside existing SQLiteSpanExporter
  • OTLP HTTP/JSON exporter for firewall-friendly environments
  • Configurable export filters (only export errors, only export slow spans)
  • Dual-write mode: SQLite for evalyn + OTLP for observability platform
• Trace Replay - Re-run a captured trace against a different model to compare outputs
  • Extract input messages from each LLM span for replay
  • Swap model name and re-execute captured prompts
  • Generate side-by-side diff of original vs replayed outputs
  • Cost comparison report between original and replayed model
• Cost Budget Alerts - Warn or stop when cumulative LLM cost exceeds a configurable threshold
  • Per-session budget limit in evalyn.yaml
  • Per-run budget limit as --max-cost flag
  • Warning at 80% threshold, hard stop at 100%
  • Budget tracking across multiple eval runs in a session
• Trace Diff - Side-by-side comparison of two traces showing divergent spans
  • Align spans by name/type and highlight added/removed/changed spans
  • Show output text diff for matching spans
  • Cost and latency delta per span
  • ASCII and HTML diff output formats
• Trace Search Query Language - Filter traces by span attributes, duration, cost, or error status
  • SQL-like syntax: "spans where type=llm_call and duration_ms > 5000"
  • Attribute filtering: model name, token count, error status
  • Aggregate queries: "traces with total_cost > $0.10"
  • Integration with list-calls command via --query flag
• PII Redaction - Scrub sensitive data from inputs/outputs before storage
  • Regex-based patterns for emails, phone numbers, SSNs, credit cards
  • Named entity recognition for names and addresses
  • Configurable redaction strategy: mask, hash, or remove
  • Pre-storage hook in SQLiteSpanExporter and SQLiteStorage
• Trace Sampling Rate - Capture only N% of traces in production to reduce storage overhead
  • Configurable sample rate in evalyn.yaml (0.0 to 1.0)
  • Priority-based sampling: always capture errors and slow traces
  • Per-project sampling rate override
• Distributed Trace Propagation - Pass trace context across service boundaries via HTTP headers
  • W3C Trace Context (traceparent/tracestate) header injection
  • HTTP client instrumentation to propagate headers on outbound calls
  • Incoming header extraction to attach child spans to external parent
• Trace Size Limits - Cap span payload size with configurable truncation for large inputs/outputs
  • Max input/output size in bytes with tail truncation
  • Configurable per span type (larger limit for llm_call, smaller for tool_call)
  • Truncation marker in span metadata when content is clipped
• Custom Span Types - Register user-defined span types beyond the built-in set (llm_call, tool_call, etc.)
  • Registration API: register_span_type(name, icon, color)
  • Custom span type validation in span creation
  • Custom types rendered in show-trace with user-defined icons
• Span Tagging at Trace Time - Add custom key-value tags to spans during execution for later filtering
  • API: tag_current_span(key, value) callable inside traced functions
  • Tags stored in span metadata and queryable via list-calls
  • Standard tags: environment, user_id, experiment_id, variant
• Native Embedding and Reranker Span Types - First-class span types for embedding and reranking operations
  • "embedding" span type capturing model name, input text, vector dimensions
  • "reranker" span type capturing query, documents, and re-ranked scores
  • "guardrail" span type capturing check name, pass/fail, and blocked content
  • Update SPAN_KIND_TO_TYPE mapping in conventions.py (currently mapped to "custom")
• Span Attribute Extraction Plugins - Pluggable attribute extractors for SpanConverter
  • Plugin interface for extracting custom attributes from OTEL spans
  • Provider-specific extractors (e.g. extract function_call from OpenAI tool use spans)
  • Configurable truncation limits per attribute (currently hardcoded 1000 chars)
• Trace Compression - Compress span payloads before SQLite storage to reduce database size
  • gzip or zstd compression for input/output fields exceeding size threshold
  • Transparent decompression on read in SQLiteStorage
  • Compression ratio reporting in storage-stats command
  • Configurable compression level and minimum payload size for compression
• Span Dependency Graph - Auto-detect causal data flow between spans within a trace
  • Detect when output of span A appears as input to span B (content overlap heuristic)
  • Build directed dependency graph from data flow analysis
  • Visualize as Mermaid or ASCII DAG in show-trace
  • Identify bottleneck spans that block the most downstream work
• Hot Path Detection - Identify the most frequently executed span sequences across traces
  • Extract sequential span-type patterns (e.g. llm_call->tool_call->llm_call)
  • Rank patterns by frequency and cumulative cost
  • Highlight optimization opportunities for repeated expensive patterns
• Trace Density Heatmap - Time-based visualization showing trace volume across hours and days
  • Hour-of-day x day-of-week grid showing trace counts
  • Overlay cost or error rate on the heatmap
  • ASCII heatmap for terminal, HTML for reports
• Provider SDK Version Tracking - Capture installed SDK versions of instrumented providers in span metadata
  • Record openai, anthropic, google-generativeai package versions at instrumentation time
  • Store as span attributes (evalyn.provider_sdk_version)
  • Surface version mismatches across traces in show-trace output
• Trace Anonymization Export - Export traces with user content replaced by synthetic equivalents for sharing
  • Replace input/output text with length-preserving placeholder content
  • Preserve span structure, timing, token counts, and cost data
  • evalyn export-traces --anonymize for safe sharing and bug reports
• Trace Flame Graph - Flame graph rendering for span durations within a trace
  • Stacked bar visualization where width represents wall-clock time per span
  • Color-code by span type (llm_call, tool_call, node, etc.)
  • ASCII flame graph for terminal, SVG for HTML reports
  • Identify time-dominant spans at a glance vs nested show-trace tree
• Trace Summary Generation - LLM-generated natural language summary of trace behavior
  • Summarize what the agent did: tools called, decisions made, output produced
  • evalyn summarize-trace --id <id> producing 2-3 sentence summary
  • Batch summaries for dataset items to understand coverage
• Trace Metadata Inheritance - Child spans automatically inherit parent's custom tags
  • Inheritance rules configurable: inherit-all, inherit-listed, no-inherit
  • Override inherited tags at child level
  • Useful for propagating environment, user_id, experiment_id down the span tree
• Trace Cost Breakdown by Phase - Attribute cost to trace phases (reasoning, tool use, output)
  • Classify spans into phases based on span type and position in tree
  • Per-phase cost aggregation in show-trace and analyze output
  • Identify which phase consumes the most tokens/cost
• Trace Correlation with External Events - Link traces to deployments, incidents, or config changes
  • evalyn mark-event --type deploy --label "v2.1 rollout" recording event timestamp
  • Overlay events on trend charts to correlate metric changes with deploys
  • Query traces around an event: evalyn list-calls --around-event <event-id>
• Trace Complexity Score - Single numeric score summarizing trace complexity for quick triage
  • Weighted combination of span depth, breadth, total span count, and tool call count
  • Score stored in FunctionCall metadata for filtering in list-calls --sort complexity
  • Threshold alerts: flag traces exceeding expected complexity for the project
• Trace Template Matching - Detect if a trace matches known execution patterns
  • Built-in templates: "RAG pattern" (retrieve->generate), "retry loop", "fan-out/fan-in"
  • Custom pattern definitions in evalyn.yaml as span-type sequences
  • evalyn classify-traces showing which pattern each trace matches
  • Pattern coverage report: what % of traces match known patterns vs are novel
• Span Type Distribution - Per-project statistics on span type frequencies over time
  • Count and percentage of each span type (llm_call, tool_call, node, agent, etc.)
  • Trend: how span type distribution shifts across weeks
  • Useful for detecting architectural changes (e.g. suddenly more tool calls)
• Instrumentation Compatibility Report - Track which provider SDK versions have been tested
  • Record provider package version on first instrumentation in session
  • evalyn check-compat showing tested vs current SDK versions
  • Warning when using an untested SDK version
• Trace Lineage Graph - Visualize how one trace's output becomes another trace's input
  • Detect session-level chaining where output of call A is input to call B
  • Render as directed graph showing data flow across function calls
  • evalyn show-lineage --session <id> producing Mermaid or ASCII graph
• Orphan Span Recovery - Detect and attach spans captured outside an active trace context
  • Orphan spans collected in _orphan_spans list (context.py) are currently lost
  • Match orphans to the nearest active FunctionCall by timestamp proximity
  • Report recovered vs truly lost orphan spans in show-trace
• Context Propagation Diagnostics - Verify ContextVar propagation across async and thread boundaries
  • evalyn check-context that spawns test async tasks and threads to verify span hierarchy
  • Detect when ThreadPoolExecutor breaks ContextVar inheritance
  • Recommend workarounds when propagation failures are detected
• Instrumentation Toggle API - Hot-toggle instrumentation on/off at runtime without restart
  • evalyn_sdk.toggle_instrumentation(enabled=False) to pause tracing
  • Useful for excluding specific code sections from tracing overhead
  • Toggle state visible in show-projects output
• Span Collector Statistics - Report collected, orphaned, and lost spans per session
  • Track spans collected vs expected (from OTEL SpanProcessor callbacks)
  • Warning when span loss exceeds threshold (e.g. >5% lost)
  • Statistics available via evalyn show-call --stats flag
• Instrumentation Dry-Run - Show what would be patched without actually applying instrumentation
  • evalyn check-instrumentation --dry-run listing SDK methods that would be wrapped
  • Report detected SDK versions and instrumentation strategy per provider
  • Useful for verifying compatibility before enabling auto-instrumentation

Trace Lifecycle Management

• Trace Archival - Move old traces to cold storage instead of deleting
  • evalyn archive-traces --older-than 90d moving traces to archive.sqlite
  • Archive is read-only and queryable via --db archive flag
  • Restore from archive: evalyn restore-traces --from archive --id <id>
• Post-Hoc Trace Annotation - Add notes and tags to existing traces after capture
  • evalyn tag-trace --id <id> --tag "regression-candidate"
  • evalyn annotate-trace --id <id> --note "Root cause: stale prompt cache"
  • Tags and notes queryable in list-calls and build-dataset filters
• Trace Bookmarking - Mark interesting traces for later review or inclusion in datasets
  • evalyn bookmark --id <id> --reason "edge case: empty input"
  • evalyn list-bookmarks showing all bookmarked traces
  • --bookmarked-only flag on build-dataset to create datasets from bookmarks

Provider-Specific Feature Capture

• Gemini Safety Rating Capture - Capture safety ratings from Gemini responses
  • Extract safetyRatings array from GenerateContent responses
  • Store per-category ratings (harassment, hate, dangerous, sexual) in span attributes
  • Surface safety blocks in show-trace output
• Gemini Grounding Metadata Capture - Capture search grounding results from Gemini
  • Extract groundingMetadata and searchEntryPoint from grounded responses
  • Store grounding sources and confidence in span attributes
  • Link grounding data to grounding metrics (source_attribution, claim_verification)
• @trace Decorator Span Upgrade - Upgrade @trace from event-based to span-based tracing
  • Create proper Span objects instead of TraceEvent pairs (start/end)
  • Automatic parent-child hierarchy via span_context stack
  • Visible in show-trace as child spans alongside LLM and tool spans
• Anthropic Thinking Block Capture - Capture extended thinking/reasoning from Claude responses
  • Extract thinking content blocks from Anthropic Messages API responses
  • Store thinking text in span attributes alongside output content
  • Display thinking blocks in show-trace with distinct styling
  • Enable reasoning quality evaluation on captured thinking content
• Metric-Specific Provider Routing - Use different judge providers for different metric categories
  • Route safety metrics to Gemini, quality metrics to OpenAI, etc.
  • Provider routing config per metric in evalyn.yaml
  • Cost optimization: use cheap models for simple metrics, expensive for nuanced ones

Instrumentation & Decorator Enhancements

• Selective Instrumentation - Only instrument specific methods or classes, not entire SDK
  • Allowlist/blocklist of method names to instrument per provider
  • Config in evalyn.yaml: instrument.openai.methods: ["chat.completions.create"]
  • Reduce overhead by skipping low-value calls (e.g. embeddings, moderation)
• Instrumentation Health Check - Verify instrumentation is capturing spans correctly
  • evalyn check-instrumentation that runs a test call and verifies span capture
  • Report which providers are instrumented, which failed, and why
  • Warning when instrumented SDK is imported before evalyn_sdk
• Instrumentation Overhead Measurement - Measure performance impact of tracing
  • Benchmark: instrumented vs uninstrumented call latency
  • Report added overhead in ms and % per provider
  • Auto-disable instrumentation if overhead exceeds threshold
• Experiment Tracking - Group traces by experiment ID for A/B comparisons
  • @eval(experiment="prompt-v2") decorator parameter
  • Filter traces by experiment in list-calls and build-dataset
  • Cross-experiment metric comparison in analyze command
• Conditional Tracing - Only trace when runtime conditions are met
  • Sample-based: trace 10% of calls via @eval(sample_rate=0.1)
  • Predicate-based: @eval(trace_if=lambda args: args["user_id"] in sample_set)
  • Environment-based: only trace in production, skip in unit tests

Onboarding & Templates

• Quickstart Templates - Framework-specific guided templates beyond generic quickstart
  • evalyn quickstart --template rag for RAG pipeline setup
  • evalyn quickstart --template chatbot for conversational agent setup
  • evalyn quickstart --template multi-agent for multi-agent orchestration
  • Each template pre-selects relevant metric bundles
• Interactive Tutorial Mode - Step-by-step in-terminal tutorial for learning evalyn
  • evalyn tutorial that walks through trace/build/eval/analyze cycle
  • Bundled sample traces so tutorial works without API keys
  • Progressive disclosure: each step explains what happened and why
• Example Agent Gallery - Bundled working example agents for each supported framework
  • example_agents/ directory with one example per framework
  • Each example includes: agent code, pre-built dataset, expected results
  • evalyn example --framework openai to scaffold from template

Config & Project Management

• Config Inheritance - Base config with per-project overrides
  • Global ~/.evalyn/config.yaml for shared settings (API keys, provider defaults)
  • Project-level evalyn.yaml inherits and overrides global config
  • Per-dataset config override via meta.json
• Project Scaffolding - evalyn new-project to create standard project structure
  • Create data/ directory, evalyn.yaml, and .gitignore entries
  • Optional: create example agent file for chosen framework
  • Optional: create GitHub Actions workflow for CI evaluation
• Multi-Project Dashboard - View and compare metrics across multiple projects
  • evalyn projects showing all projects with latest run status
  • Cross-project regression detection
  • Unified cost tracking across projects

Confidence & Judge Robustness

• Confidence Method Comparison - Run all confidence methods on same data and compare calibration
  • Side-by-side comparison of logprobs, deepconf, consistency, verbalized methods
  • Calibration curve: confidence score vs actual correctness
  • Recommend best method per metric/provider combination
• Hybrid Confidence - Combine multiple confidence methods into a single robust score
  • Weighted ensemble of available methods
  • Fall back gracefully when a method is unavailable (e.g. no logprobs)
  • Bayesian combination with learned weights
• Structured Output Enforcement - Force JSON mode on judge LLM calls for reliable parsing
  • Use provider-native JSON mode (Gemini response_mime_type, OpenAI response_format)
  • Schema enforcement via provider-specific structured output features
  • Fallback to regex extraction when JSON mode unavailable
• Judge Output Retry - Automatically retry judge calls when output fails to parse
  • Configurable max retries (default 2)
  • Append "respond with valid JSON" on retry attempts
  • Track parse failure rate per metric for diagnostics
• Judge Latency Optimization - Reduce judge call overhead for large-scale evaluation
  • Prompt caching: reuse system prompt prefix across items
  • Batch multiple items into single judge call where possible
  • Model-specific prompt length optimization

Evaluation Units & Views

• Custom Unit Builder Plugins - User-defined evaluation boundaries via pluggable builders
  • Register custom EvalUnitBuilder subclasses via entry points
  • Builder configuration in evalyn.yaml per metric
  • Example builders: per-paragraph, per-code-block, per-citation
• Unit Type Auto-Detection - Infer best EvalUnit type from trace structure
  • Detect multi-turn patterns from sequential LLM spans
  • Detect tool-use patterns from tool_call/tool_result span pairs
  • Default to outcome when trace structure is flat
• Unit-Level Reporting - Per-unit-type metric breakdowns in analysis
  • Separate pass rates for outcome vs single_turn vs tool_use units
  • Unit type distribution chart in analysis output
  • Filter analysis by unit type: --unit-type single_turn

Batch Evaluation Enhancements

• Batch Job Persistence - Save batch job state to disk for recovery after crash or restart
  • Write BatchJob to .evalyn/batch_jobs/ as JSON on submit
  • evalyn batch-status to list pending/completed batch jobs
  • evalyn batch-resume to collect results from a previously submitted batch
• Mixed-Mode Evaluation - Use batch API for large runs, real-time for small runs
  • Auto-select mode based on item count threshold (e.g. batch if > 50 items)
  • --mode auto/batch/realtime flag on run-eval
  • Cost/speed comparison in dry-run output
• Batch Progress Polling - Live progress updates while batch job is processing
  • Poll provider API for completion percentage
  • Display progress bar with ETA during batch wait
  • Configurable poll interval (default 30s)
• Multi-Provider Batch Splitting - Split a single evaluation batch across multiple providers
  • Route N% of items to gemini, M% to openai for cost/latency comparison
  • Provider-aware retry: re-route failed items to alternate provider
  • Unified result merging regardless of which provider evaluated each item
• Streaming Partial Results - Start analyzing results before the full batch completes
  • Process completed items as they arrive from batch polling
  • Live-updating analysis dashboard during batch wait
  • Early termination: stop batch if enough results show clear pass/fail

Session Management

• Session-Level Analysis - Aggregate metrics across all calls within an eval_session
  • Group traces by session_id in analysis output
  • Per-session pass rate, cost, and latency summaries
  • Cross-session comparison for the same user journey
• Session Replay - Re-execute a full session against a different model or prompt version
  • Extract all inputs from session traces in order
  • Replay with swapped model/provider
  • Session-level diff: compare original vs replayed outputs turn by turn

Reproducibility

• Deterministic Evaluation Mode - Ensure runs produce identical results given identical inputs
  • Fixed random seed for all sampling operations
  • Temperature 0 enforcement for judge LLM calls
  • --seed flag on run-eval for reproducible runs
• Run Manifest - Record every parameter that could affect evaluation results
  • Store: evalyn version, Python version, provider versions, metric hashes, config hash
  • Manifest file alongside eval run results
  • evalyn verify-manifest to check reproducibility of a past run
• Custom Cost Models - User-defined pricing for custom or self-hosted models
  • Per-model cost-per-token config in evalyn.yaml
  • Override default pricing for Ollama and other local models
  • Cost model versioning for tracking price changes over time

Cost Intelligence

• Auto-Update Pricing Tables - Fetch latest model pricing from provider APIs
  • Scrape/fetch pricing from OpenAI, Anthropic, Google pricing pages
  • evalyn update-pricing command to refresh COST_PER_1M_TOKENS in _shared.py
  • Warn when using a model not in the pricing table
• Prompt Cache Savings Report - Show how much prompt caching saved per run
  • Aggregate cache_creation_tokens and cache_read_tokens from spans
  • Calculate: actual cost vs hypothetical cost without caching
  • Recommend caching strategy based on prompt repetition patterns
• Context Window Utilization Alerts - Warn when spans approach context limits
  • Alert when context_utilization_pct exceeds configurable threshold (default 80%)
  • Per-run summary: max utilization, mean utilization, models hitting limits
  • Suggest model upgrade when context is consistently near capacity

Confidence Enhancements

• Adaptive Consistency Sampling - Stop early when judge agreement is already clear
  • Sequential sampling: stop after 3 samples if all agree (skip remaining 2)
  • Configurable early-stop threshold (e.g. 100% agreement after 3 of 5 samples)
  • Cost savings report: samples skipped vs full sampling
• Confidence-Based Re-Evaluation - Re-evaluate uncertain items with a stronger model
  • Identify items where confidence score < threshold after initial eval
  • Automatically re-run those items with a more capable model (e.g. flash -> pro)
  • Merge re-evaluated scores back into the run results
• Confidence Threshold Tuning - Find optimal confidence cutoff per metric
  • Binary search for threshold that maximizes alignment with human annotations
  • Per-metric optimal threshold stored in calibration record
  • evalyn tune-confidence command

Config Enhancements

• Config Profiles - Named environment profiles (dev/staging/prod) in evalyn.yaml
  • profiles: section with per-profile overrides
  • --profile flag on all commands to select active profile
  • Profiles inherit from base config, override specific keys
• Environment Variable Validation - Check all required env vars at command startup
  • Required vars per command (e.g. run-eval needs GEMINI_API_KEY)
  • Validate key format and basic connectivity before starting long operations
  • Clear error messages: "GEMINI_API_KEY is set but invalid (HTTP 401)"

Evaluation Enhancements

• Span-Level Evaluation - Evaluate individual spans within a trace
  • Per-LLM-call quality metrics
  • Tool call success/failure analysis
  • Node-level evaluation for graph agents
  • Span-specific rubrics
• Multi-Turn Evaluation - Specialized evaluation for conversations
  • Turn-by-turn quality assessment
  • Conversation flow metrics
  • Context carryover evaluation
  • Memory consistency across turns
  • Topic drift detection
  • Response latency patterns
• Pairwise Comparison - A vs B evaluation mode
  • Side-by-side LLM judge comparison
  • Elo rating system for models
  • Win/loss/tie statistics
• Reference-Free Evaluation - Metrics that don't need ground truth
  • Self-consistency checking (via --confidence consistency)
  • Uncertainty quantification (via confidence module)
• Evaluation Budget Control - Stop early if token or cost budget is exceeded mid-run
  • --max-tokens and --max-cost flags on run-eval
  • Real-time budget tracking in ProgressCallback
  • Graceful stop: finish current item, checkpoint, report partial results
  • Budget summary in EvalRun metadata
• Differential Evaluation - Only re-evaluate items that changed between dataset versions
  • Hash-based change detection using datasets.hash_inputs
  • Carry forward unchanged MetricResults from previous run
  • --diff-from flag to specify baseline run ID
  • Report showing only changed items and their score deltas
• Evaluation Caching - Skip re-computing unchanged metric/item pairs across runs
  • Content-addressable cache keyed by (item_hash, metric_id, prompt_hash)
  • Cache stored in SQLite alongside eval runs
  • --no-cache flag to force re-evaluation
  • Cache hit/miss statistics in run summary
• Evaluation Dry-Run - Estimate token cost and wall-clock time before executing
  • Count items x metrics, estimate tokens per metric type
  • Cost estimate by provider (Gemini, OpenAI pricing)
  • --dry-run flag that prints estimate and exits
  • Wall-clock estimate based on historical run data
• Cross-Validation Evaluation - K-fold scoring for statistically robust metric estimates
  • --cv-folds N flag to split dataset into N folds
  • Stratified splitting by metadata or score
  • Per-fold and aggregate metric statistics with std deviation
  • Identify items with high variance across folds
• Evaluation Replay - Re-run a past evaluation with different judge prompts or providers
  • --replay-run flag to reuse items/metrics from a previous run
  • Override provider, model, or calibrated prompts
  • Automatic comparison report between original and replayed run
• Conditional Metrics - Run expensive subjective metrics only if cheap objective metrics pass first
  • Metric dependency declaration: "run helpfulness only if json_valid passes"
  • Gate conditions: pass/fail, score threshold, or custom predicate
  • Skip tracking: report which items had metrics skipped and why
• Evaluation Profiles - Named configs (fast/thorough/cost-optimized) bundling workers, providers, and metric sets
  • Profile definitions in evalyn.yaml (fast: 8 workers, objective only; thorough: all metrics, 2 workers)
  • --profile flag on run-eval
  • Built-in profiles: smoke-test, standard, comprehensive
• Evaluation Tagging - Tag runs with custom labels for filtering and organization
  • --tag flag on run-eval (multiple tags allowed)
  • Tags stored in EvalRun metadata and queryable via list-runs
  • Filter list-runs by tag: --filter-tag experiment-v2
• Async Evaluation Strategy - Native asyncio execution strategy alongside sequential and parallel
  • AsyncStrategy using asyncio.gather for concurrent metric calls
  • Semaphore-based concurrency control (replaces ThreadPoolExecutor)
  • Compatible with async LLM client libraries (httpx, aiohttp)
  • --strategy flag: sequential, parallel, async
• Distributed Evaluation - Fan out metric evaluation across multiple machines via task queue
  • Redis/RabbitMQ task queue for distributing metric evaluations
  • Worker process that pulls and evaluates metric tasks
  • Centralized result collection and checkpoint merging
  • --distributed flag with queue URL configuration
• Canary Evaluation - Run eval on a small random subset first; abort full run if pass rate is below threshold
  • --canary N flag to evaluate N items before committing to full run
  • Configurable abort threshold (default: 20% pass rate on canary)
  • Cost savings report: how much was saved by aborting early
• Evaluation Warm-Up - Discard first K results to reduce cold-start score variance from judge LLM
  • --warmup K flag discarding first K item scores
  • Re-evaluate warm-up items after LLM cache is primed
  • Measure score variance reduction from warm-up vs no warm-up
• Multi-Language Auto-Detection - Detect output language and apply language-appropriate metric rubrics automatically
  • Language detection via character set and n-gram analysis (no external API)
  • Route to language-matched rubric variant when available
  • Report language distribution across dataset items
• Metric Score Normalization - Normalize scores across metrics to a common scale for fair cross-metric comparison
  • Z-score normalization using historical score distributions per metric
  • Min-max normalization to [0, 1] range
  • --normalize flag on analyze and compare for normalized views
• Evaluation Resource Monitoring - Track memory and CPU usage during evaluation to detect resource issues
  • Per-worker memory tracking via psutil (optional dependency)
  • Warning when memory exceeds configurable threshold
  • Resource usage summary in eval run metadata
• Evaluation Abort Conditions - Compound abort rules beyond simple pass rate threshold
  • Rule syntax: "abort if any safety metric < 50% on any item"
  • Multiple abort conditions combinable with AND/OR logic
  • --abort-on flag on run-eval with condition expression
• Human-AI Hybrid Scoring - Route uncertain items to human annotator during evaluation
  • Confidence threshold below which items are queued for human review
  • Interactive prompt during eval for human labels on flagged items
  • Merge human and judge scores in final EvalRun results
• Evaluation Result Changelog - LLM-generated summary of differences between two runs
  • Natural language description: "3 items regressed on helpfulness, all related to multi-step queries"
  • evalyn changelog --run1 <id> --run2 <id> producing human-readable diff
  • Highlight most impactful changes
• Metric Execution Priority Queue - Run most-likely-to-fail metrics first for faster feedback
  • Priority based on historical failure rate per metric
  • Surface failing metrics early in progress output
  • Combine with abort conditions for fast-fail workflows
• Evaluation Retry Budget - Bound total retries across all items with a per-run budget
  • --max-retries-total flag (default: unlimited)
  • Track retries consumed vs budget in progress output
  • Prevent retry storms from consuming excessive tokens
• Evaluation Progress API - Structured progress events for external monitoring tools
  • Emit JSON events to a file or socket: item_started, item_complete, metric_scored
  • Enable integration with CI dashboards, Slack bots, and custom UIs
  • --progress-file flag on run-eval writing JSONL progress events
• Evaluation Throttle Control - Dynamically adjust concurrency based on API response times
  • Reduce workers when latency exceeds threshold (provider overloaded)
  • Increase workers when latency is low (headroom available)
  • Adaptive mode: --workers auto on run-eval
• Evaluation Split-Model Routing - Route objective metrics to local compute, subjective to API
  • Automatic: objective metrics skip API entirely, subjective use configured provider
  • Cost savings report showing how much was saved by local objective evaluation
  • --local-objectives flag (default: true) on run-eval
• Evaluation Partial Result Access - Query in-progress evaluation results before run completes
  • evalyn show-run --id <id> works on actively running evaluations via checkpoint data
  • Live pass rate estimate from completed items
  • Useful for monitoring long-running evaluations without waiting for completion
• Evaluation Comparison Auto-Trigger - Automatically compare against pinned baseline after each run
  • When a baseline run is pinned, run-eval auto-runs compare at the end
  • Regression summary appended to run-eval output
  • --no-auto-compare flag to disable
• Evaluation Isolation Mode - Run each metric in a subprocess to prevent crashes from affecting other metrics
  • --isolate flag spawning each metric evaluation in a child process
  • Crash in one metric produces error result without killing the run
  • Useful for untested custom metrics or unstable provider connections
• Evaluation Result Signing - Cryptographic hash of results for tamper detection
  • SHA-256 hash of all MetricResults stored in EvalRun metadata
  • evalyn verify-run --id <id> checking result integrity against stored hash
  • Detect if results were manually edited after evaluation
• Evaluation Item-Level Cost Attribution - Track exact LLM cost per dataset item
  • Sum input/output tokens across all metrics for each item
  • Per-item cost in show-run output and export formats
  • Identify most expensive items for cost optimization
• Evaluation Output Diff - Show exact text differences between expected and actual output per item
  • evalyn diff-outputs --run <id> showing per-item expected vs actual text diff
  • Highlight added/removed/changed text with color coding
  • Filter to only items where expected reference is available
• Judge Debiasing - Mitigate known LLM judge biases (position, length, verbosity)
  • Position-bias mitigation: swap answer order in pairwise comparisons and average
  • Length-controlled scoring: GLM correction for length preference (AlpacaEval approach)
  • Regression-based bias correction from small human-annotated calibration set
  • Report bias metrics per judge model in calibration output
• Agent Goal Completion Metrics - Evaluate whether agents achieve stated objectives
  • ToolCallAccuracy: sequence + argument correctness (Ragas-inspired)
  • ToolCallF1: unordered tool call matching
  • AgentGoalAccuracy: end-state vs expected outcome assessment
  • TopicAdherence: domain boundary enforcement for conversational agents
• Automatic Test Case Generation from Behaviors - Generate diverse scenarios from behavior descriptions
  • Bloom-style pipeline: understand behavior -> generate scenarios -> execute -> score
  • Mine production traces for challenging evaluation cases (Arena-Hard BenchBuilder pattern)
  • Synthesize adversarial variants of existing test cases
• DAG-Based Deterministic Evaluation - Decision-tree scoring as middle ground between rules and LLM judge
  • DAGMetric: LLM-powered decision trees for structured scoring (DeepEval-inspired)
  • Deterministic evaluation paths based on input characteristics
  • Lower cost than full LLM judge, more flexible than regex rules
• Statistical Evaluation Reporting - Confidence intervals and power analysis for all metrics
  • Bootstrap confidence intervals (1000 resamples) on metric scores
  • Power analysis: recommend minimum sample size for target precision
  • Significance testing for run-to-run comparisons (two-proportion z-test)

Calibration & Optimization

• More Optimizers
  • DSPy MIPROv2 - Multi-stage instruction optimization
  • TextGrad - Gradient-based prompt optimization
  • EvoPrompt - Evolutionary prompt optimization
  • PromptBreeder - Self-referential prompt evolution
• Rubric Optimization - Auto-generate and refine evaluation rubrics
  • LLM-generated rubric from example pass/fail items
  • Iterative rubric refinement based on disagreement analysis
  • Rubric clarity scoring (can a different LLM interpret it consistently?)
  • A/B test rubric variants for inter-judge agreement
• Few-Shot Example Selection - Optimize which examples to include in prompts
  • Select maximally informative examples from annotation pool
  • Diversity-based selection: cover different failure modes
  • Leave-one-out evaluation to measure example contribution
  • Dynamic example count optimization (find optimal k)
• Judge Ensemble - Combine multiple judges for robust evaluation
  • Majority vote across N judges (same or different models)
  • Weighted ensemble based on per-judge calibration accuracy
  • Disagreement flagging: items where judges disagree go to human review
  • Cost-aware ensemble: use cheap judge first, expensive only on uncertain items
• Active Learning - Smart sample selection for annotation
  • Uncertainty sampling: prioritize items where judge confidence is lowest
  • Disagreement sampling: prioritize items where judge and heuristics disagree
  • Diversity sampling: ensure coverage of input space
  • Batch-mode active learning with configurable batch size
• Transfer Calibration - Apply calibration learned on one metric to similar metrics
  • Metric similarity detection based on rubric text embedding
  • Shared preamble transfer with metric-specific rubric
  • Transfer effectiveness validation on held-out samples
• Calibration Staleness Detection - Warn when calibration age or dataset drift exceeds threshold
  • Track calibration date and dataset hash at calibration time
  • Alert when dataset changes exceed drift threshold (new items, distribution shift)
  • Re-calibration recommendation with estimated alignment degradation
• Cross-Provider Calibration - Calibrate for consistency when switching judge providers
  • Run same calibration set across providers (Gemini, OpenAI, Ollama)
  • Provider-specific preamble adjustments
  • Cross-provider agreement metrics
• Calibration A/B Testing - Compare calibrated vs uncalibrated prompts on the same dataset
  • Side-by-side evaluation run with original and calibrated prompts
  • Per-item comparison showing score changes
  • Statistical significance test for improvement
• Calibration Rollback - Revert to a previous calibration if the new one degrades alignment
  • Calibration history stored in CalibrationRecord
  • --rollback flag on calibrate command
  • Automatic rollback suggestion when validation metrics drop
• Multi-Objective Calibration - Optimize jointly for accuracy and cost (fewer tokens per judgment)
  • Pareto front of accuracy vs token count
  • Prompt compression as optimization objective
  • Configurable accuracy/cost trade-off weight
• Calibration Cost Tracking - Report total LLM cost of the calibration process itself
  • Per-optimizer token usage tracking (extend TokenAccumulator)
  • Cost breakdown by calibration phase (alignment, optimization, validation)
  • Historical cost trends across calibration runs
• Calibration Curriculum - Start optimization on easy examples, progressively add harder ones
  • Sort calibration examples by judge confidence (easy = high confidence)
  • Progressive expansion: start with top-50% easiest, add harder items
  • Early stopping if optimizer plateaus before reaching hard examples
• Calibration Convergence Visualization - Plot alignment score vs optimization step to diagnose optimizer behavior
  • Record per-step alignment scores during optimization
  • Detect plateau, oscillation, and divergence patterns
  • ASCII convergence chart in terminal, SVG in HTML reports
  • Recommend optimizer parameter changes based on convergence shape
• Prompt Length Regularization - Penalize prompt length during calibration to keep judge prompts concise
  • Add token count penalty term to optimizer objective function
  • Configurable weight: --length-penalty 0.1 (default 0, no penalty)
  • Report prompt token savings vs alignment trade-off
• Calibration Data Augmentation - Augment calibration examples by paraphrasing to improve optimizer generalization
  • LLM-powered paraphrase of calibration inputs preserving semantics
  • Expand calibration set 2-5x without additional human annotation
  • Validate paraphrased items preserve original labels
• Calibration Difficulty Weighting - Weight alignment errors by item difficulty so hard items count more
  • Difficulty estimate from cross-annotator disagreement or judge confidence
  • Weighted accuracy metric in optimizer objective
  • Prevent optimizer from gaming easy items while ignoring hard ones
• Per-Score-Level Calibration - Calibrate separately for each score level to reduce systematic bias
  • Detect if judge systematically over/under-scores at specific levels
  • Score-level-specific preamble adjustments
  • Confusion matrix per score level showing calibration effectiveness
• Calibration Ensemble Fusion - Run multiple optimizers and fuse outputs via tournament selection
  • Run 2-3 optimizers in parallel on same calibration data
  • Tournament: evaluate each optimizer's prompt on held-out set
  • Select best-performing prompt or blend top-K prompts
• Calibration Sensitivity Analysis - Measure alignment sensitivity to small prompt perturbations
  • Perturb calibrated prompt (word swaps, paraphrase, reorder)
  • Measure alignment variance across perturbations
  • Flag calibrations that are fragile (small change causes large alignment drop)
• Few-Shot Example Ordering - Optimize the order of examples in few-shot judge prompts
  • Test permutations of example order and measure alignment impact
  • Heuristics: put hardest examples last, group by failure type
  • Store optimal order in CalibrationRecord
• Calibration Diagnostic Report - Detailed analysis of why calibration improved or degraded alignment
  • Per-item breakdown: which items flipped from wrong to right (and vice versa)
  • Prompt diff showing exactly what changed in the preamble
  • Categorize improvements by item type (false positive fixes vs false negative fixes)
• Calibration Freeze - Lock a calibration record to prevent accidental overwriting
  • evalyn freeze-calibration --id <id> marking calibration as immutable
  • Prevent calibrate command from overwriting frozen records
  • evalyn unfreeze-calibration to unlock when intentional re-calibration is needed
• Calibration Comparison Dashboard - Side-by-side view of multiple calibration attempts
  • evalyn compare-calibrations --ids <id1> <id2> showing alignment metrics
  • Prompt diff between calibration versions
  • Per-item score change matrix across calibrations
• Calibration Checkpoint - Save optimizer state mid-run for resuming long calibrations
  • Atomic checkpoint writes at configurable intervals during optimization
  • evalyn calibrate --resume to continue from last checkpoint
  • Prevent wasted compute on interrupted calibration runs
• Calibration Human Validation - Present calibrated prompt to human for approval before committing
  • Show before/after prompt diff and alignment metrics change
  • Interactive confirm/reject/edit before writing CalibrationRecord
  • --auto-accept flag to skip validation in CI
• Calibration Memory - Remember what approaches failed in past calibration runs
  • Store failed prompt variants and their alignment scores
  • Optimizer avoids re-exploring previously failed regions of prompt space
  • Accumulated across calibration runs for the same metric
• Calibration Scope Control - Calibrate only for specific item subsets
  • --scope flag: calibrate for long inputs only, or specific metadata values
  • Scope-specific preambles stored separately in CalibrationRecord
  • Apply scope-matched calibration at eval time based on item characteristics
• Calibration Time Budget - Stop optimization after N minutes regardless of convergence
  • --max-time flag on calibrate command (e.g. --max-time 10m)
  • Return best prompt found within time budget
  • Report whether optimizer converged or was time-limited
• Calibration Alignment Curve - Plot alignment vs annotation count to find diminishing returns
  • Re-calibrate with increasing annotation subsets (10%, 25%, 50%, 75%, 100%)
  • Plot alignment improvement vs annotation count
  • Recommend minimum annotation count for acceptable calibration quality
• Calibration Negative Example Mining - Find the hardest examples where calibrated prompt still fails
  • After calibration, identify items where the calibrated judge still disagrees with humans
  • Cluster these remaining failures by pattern
  • Use as targeted additions to calibration set for next round
• Calibration Prompt Templates - Reusable preamble templates for common calibration patterns
  • Built-in templates: "strict evaluator", "lenient evaluator", "domain expert"
  • --template flag on calibrate to start from a template instead of blank
  • Save successful calibration preambles as custom templates
• Calibration Batch Processing - Calibrate multiple metrics in one command
  • evalyn calibrate --metrics all calibrating every metric with annotations
  • Parallel calibration of independent metrics for speed
  • Combined calibration report showing per-metric alignment improvements
• SAMMO-Style Structural Optimization - Treat prompts as symbolic DAGs with structural mutations
  • Represent prompt as sections (instruction, context, examples, rubric) with structural operators
  • Mutations: paraphrase section, drop section, reformat, reorder examples
  • Multi-objective search: accuracy vs prompt length vs cost
• Annotation Queue Flywheel - Closed loop where human labels improve judge, reducing future annotation needs
  • Track judge accuracy on human-labeled items over time
  • Identify metrics where judge is now reliable enough to skip human review
  • Gradually reduce annotation requirement as calibration improves
• CAPO Optimizer - Current SOTA prompt optimization algorithm
  • Implement CAPO (Confidence-Aware Prompt Optimization) as new optimizer
  • Add to OPTIMIZER_REGISTRY alongside existing 9 optimizers
  • Benchmark against existing optimizers on standard calibration tasks
• Specialized Judge Model Support - Fine-tuned evaluation models outperform general LLM-as-judge
  • Support custom model endpoints as judge providers (Patronus Lynx pattern)
  • Configurable per-metric: use specialized model for safety, general model for quality
  • Track and compare judge model accuracy across calibration rounds

Multi-Modal Evaluation

• Image Evaluation Metrics
  • Image-text alignment (CLIP score)
  • Visual quality assessment
  • OCR accuracy for generated images
  • Style consistency
• Audio Evaluation Metrics
  • Speech clarity
  • Transcription accuracy (WER)
  • Prosody and tone
• Video Evaluation Metrics
  • Frame consistency
  • Temporal coherence
  • Action recognition accuracy

Agent-Specific Evaluation

• Tool Use Evaluation
  • Tool selection appropriateness
  • Parameter correctness
  • Error recovery patterns
  • Tool chain efficiency
• Planning Evaluation
  • Plan completeness
  • Step ordering correctness
  • Resource efficiency
  • Replanning quality
• Reasoning Evaluation
  • Chain-of-thought faithfulness
  • Logical consistency
  • Evidence usage
  • Conclusion validity
• Multi-Agent Communication Scoring - Evaluate quality of inter-agent communication
  • Communication Score (1-5 per utterance): relevance, clarity, information density
  • Collaborative efficiency: ratio of useful exchanges to total messages
  • Milestone-based KPIs: track which coordination milestones are achieved (MARBLE approach)
• Agent Consistency Testing - Measure reliability across repeated runs
  • Run agent N times on same input, measure consistency of tool calls and outputs
  • Research finding: 60% single-run success drops to 25% at 8-run consistency
  • Report consistency score alongside pass rate
• Agentic Benchmark Integration - Run standard agent benchmarks within evalyn
  • SWE-bench integration for coding agent evaluation
  • WebArena integration for web agent evaluation
  • GAIA integration for general agent evaluation
  • Unified reporting across benchmarks

Graph & Multi-Agent Evaluation

• Graph Topology Extraction - Extract and visualize LangGraph execution topology from traces
  • Build DAG from graph/node spans captured by LangGraphInstrumentor
  • Identify critical path (longest execution chain through nodes)
  • Detect cycles and redundant node executions
  • evalyn show-graph --call <id> rendering ASCII or Mermaid diagram
• Node-Level Metric Attribution - Attribute eval failures to specific graph nodes
  • Map MetricResult failures back to the node span that produced the failing output
  • Per-node pass rate aggregation across dataset items
  • Identify "bottleneck nodes" that cause the most failures
• Subagent Cost Allocation - Track cost per subagent in multi-agent traces
  • Aggregate token/cost from Claude Agent SDK's SubagentContext hierarchy
  • Per-subagent cost breakdown in show-trace and analyze output
  • Identify most expensive subagent paths for optimization
• Agent Decision Tree Visualization - Render agent's tool selection choices as a tree
  • Build decision tree from tool_call/tool_result span sequences
  • Highlight decision points where agent chose between tools
  • Compare decision trees across different runs or models

Pipeline Customization

• Custom Pipeline Definitions - User-defined step sequences beyond the fixed 7-step pipeline
  • Pipeline definition in evalyn.yaml with ordered step list
  • Skip/include steps declaratively (instead of --skip-annotation flags)
  • Custom step plugins: user-defined Python functions as pipeline steps
• Pipeline Templates - Preset pipelines for different evaluation goals
  • "quick-check" template: build-dataset -> objective metrics only -> analyze
  • "full-audit" template: all 7 steps + simulation + deep insights
  • "ci-gate" template: objective metrics + threshold check + exit code
  • evalyn one-click --template quick-check
• Pipeline Comparison - Compare results of two one-click pipeline runs
  • evalyn compare-pipelines <dir1> <dir2>
  • Step-by-step comparison: dataset size, metric count, scores, cost
  • Identify which pipeline changes improved or degraded results

Infrastructure & Platform

• Web Dashboard - Browser-based UI for viewing traces, datasets, and results
  • Trace viewer with span tree navigation (like Phoenix/LangSmith)
  • Dataset browser with item search, sort, and filter
  • Eval run comparison view with metric charts
  • Real-time run progress monitoring
  • Lightweight server (Flask/FastAPI) bundled with evalyn
• CI/CD Integration - GitHub Actions for automated testing and evaluation on PR
  • GitHub Action YAML template for evalyn run-eval
  • PR comment bot posting eval results as markdown table
  • Regression gate: fail CI if metrics drop below threshold
  • Artifact upload of HTML reports and datasets
  • GitLab CI and Jenkins pipeline examples
• GitHub Action for Evalyn - Dedicated reusable GitHub Action for PR evaluation
  • braintrustdata/eval-action-style: run eval, post diff as PR comment
  • Caching of previous run results for fast comparison
  • Quality gate: configurable pass/fail threshold as PR check status
• Regression Detection - Automatic alerts when metrics drop below threshold
• Multi-model Comparison - Compare same prompts across different LLM providers
  • --models flag to run same eval across multiple providers in one command
  • Cross-model comparison table (rows=items, columns=models)
  • Cost/latency/quality trade-off analysis per model
  • Best-model-per-item analysis
• Cost Tracking Dashboard - Visualize LLM API costs over time
  • Per-run cost breakdown by metric and provider
  • Cumulative cost chart across all runs
  • Cost-per-item and cost-per-metric averages
  • Budget forecast based on historical usage
• API Server Mode - REST API for programmatic access
  • REST endpoints: /runs, /traces, /datasets, /metrics
  • Trigger eval runs via POST /runs with JSON config
  • WebSocket endpoint for real-time run progress
  • API key authentication for multi-user access
• Team Collaboration - Multi-user annotation with conflict resolution
  • User identity tracking on annotations
  • Assignment queue: distribute items across annotators
  • Conflict detection when multiple users annotate same item
  • Resolution strategies: majority vote, senior override, discussion
• Cloud Storage Backend - Optional S3/GCS storage for large datasets
  • S3-compatible backend implementing StorageBackend protocol
  • GCS backend with service account authentication
  • Hybrid mode: SQLite for metadata, cloud for large payloads
  • Configurable via evalyn.yaml storage section
• Storage Compaction - Vacuum and optimize SQLite database on demand
  • evalyn compact command to VACUUM and ANALYZE
  • Auto-compaction trigger when DB exceeds size threshold
  • Orphan cleanup: remove spans not linked to any function_call
• Data Retention Policies - Auto-delete traces and runs older than a configurable threshold
  • retention_days setting in evalyn.yaml
  • evalyn purge --older-than 30d command
  • Exempt pinned/starred runs from auto-deletion
  • Dry-run mode showing what would be deleted
• Storage Migration - Export/import data between different storage backends
  • evalyn export-db --format sqlite/json/parquet
  • evalyn import-db to load from another backend
  • Schema version validation on import
  • Incremental export: only new data since last export
• Encrypted Storage - At-rest encryption for sensitive trace and evaluation data
  • SQLCipher integration for encrypted SQLite
  • Key management via environment variable or keyring
  • Selective encryption: encrypt input/output payloads, keep metadata queryable
• Storage Statistics - Show database size, row counts, and growth rate over time
  • evalyn storage-stats command
  • Row counts per table (function_calls, eval_runs, annotations, otel_spans)
  • Size breakdown: data vs index vs free space
  • Growth rate: new rows per day/week
• Plugin System - Third-party metric, instrumentor, and storage backend plugins via entry points
  • Python entry_points discovery for evalyn.metrics, evalyn.instrumentors, evalyn.storage
  • Plugin manifest with version compatibility declaration
  • evalyn list-plugins command
  • Plugin isolation: plugins cannot modify core behavior
• Webhook Notifications - Trigger HTTP webhooks on eval completion, failure, or regression
  • Configurable webhook URLs in evalyn.yaml
  • Event types: run_complete, regression_detected, annotation_needed
  • Payload includes run summary, metric scores, and delta from previous
  • Retry with exponential backoff on delivery failure
• Rate Limit Awareness - Respect LLM provider rate limits with automatic throttling during evaluation
  • Per-provider rate limit config (RPM, TPM) in evalyn.yaml
  • Adaptive backoff when 429 errors received
  • Token bucket rate limiter shared across parallel workers
  • Rate limit status in progress callback output
• Connection Pooling - Reuse SQLite connections for high-throughput multi-threaded evaluation
  • Thread-local connection pool with configurable max size
  • Connection health checking and recycling
  • WAL mode auto-enable for concurrent readers
• Incremental Backup - Periodic automatic backup of database to a secondary location
  • SQLite online backup API integration
  • Configurable backup schedule and destination path
  • Backup rotation: keep last N backups
• Auto Model Selection - Choose judge model based on task complexity (fast model for easy items, smart model for hard ones)
  • Complexity heuristic based on input length, output length, and metric type
  • Model routing: flash-lite for simple items, flash for complex items
  • Cost savings report showing how much auto-selection saved vs always-smart
• Storage Partitioning - Partition SQLite databases by time period for better performance at scale
  • Monthly or weekly database files (evalyn_2026_03.sqlite)
  • Transparent cross-partition queries via ATTACH DATABASE
  • Auto-archive old partitions to reduce active DB size
• Storage Integrity Checks - Verify referential integrity between tables
  • Check function_calls referenced by eval_runs still exist
  • Check otel_spans have valid parent span references
  • evalyn storage-check producing integrity report with fixable/unfixable issues
• Storage Schema Introspection - Show current database schema and statistics
  • evalyn storage-schema listing table schemas, column types, index definitions
  • Schema version and migration history
  • Useful for debugging and plugin development
• Storage Merge - Merge two SQLite databases from different machines with conflict resolution
  • evalyn storage-merge --source <db2> --into <db1>
  • Deduplication by primary key (function call ID, span ID, run ID)
  • Conflict strategy: skip, overwrite, or rename
• Storage Index Tuning - Auto-create indexes based on common query patterns
  • Profile slow queries in list-calls, list-runs, build-dataset
  • evalyn storage-tune creating recommended indexes
  • Report query speedup after index creation
• Storage Query Logging - Log SQL queries for performance debugging and optimization
  • EVALYN_QUERY_LOG=1 env var enabling query logging to .evalyn/queries.log
  • Log query text, execution time, rows returned
  • Identify slowest queries for index tuning
• Storage Cross-Reference Report - Show relationships between stored entities
  • evalyn storage-xref showing: traces -> datasets -> runs -> annotations linkage
  • Identify orphaned entities (runs referencing deleted datasets, etc.)
  • Entity count summary per relationship type
• Storage Connection Diagnostics - Report SQLite configuration and health
  • evalyn storage-diag showing WAL mode, journal mode, page size, cache size
  • File lock status and concurrent access warnings
  • Recommend optimal SQLite pragmas for current workload
• Storage Snapshot/Restore - Point-in-time snapshots for safe experimentation
  • evalyn storage-snapshot --name "before-cleanup" creating named copy
  • evalyn storage-restore --name "before-cleanup" reverting to snapshot
  • Snapshot list with timestamps and sizes
• Storage Usage Forecast - Predict storage growth based on current usage rate
  • Compute growth rate from last 7/30/90 days
  • Estimate when storage will reach configurable size threshold
  • evalyn storage-forecast showing projected growth chart
• Storage Migration Versioning - Formal migration version tracking with up/down support
  • Version table tracking which migrations have been applied
  • Down-migration support for rolling back schema changes
  • evalyn storage-migrate --status showing current schema version
• Storage Read-Only Mode - Prevent accidental writes during analysis
  • EVALYN_DB_READONLY=1 env var opening database in read-only mode
  • Useful when sharing databases or running analysis on production data
  • Clear error message when write is attempted in read-only mode
• Storage Multi-DB Queries - Query across prod and test databases simultaneously
  • evalyn list-calls --db all searching both prod.sqlite and test.sqlite
  • Cross-database comparison: production traces vs test traces
  • ATTACH DATABASE under the hood with transparent result merging
• Storage WAL Monitoring - Monitor Write-Ahead Log size and checkpoint frequency
  • evalyn storage-wal showing WAL file size, checkpoint status
  • Warning when WAL exceeds configurable size threshold
  • Auto-checkpoint recommendation based on write patterns
• Storage Auto-Vacuum Scheduling - Schedule automatic vacuum based on database growth
  • auto_vacuum_threshold setting in evalyn.yaml (e.g. 500MB)
  • Run VACUUM automatically when DB crosses threshold during write operations
  • Log vacuum events with space reclaimed
• Storage Data Checksums - Verify data integrity with per-row checksums
  • Store SHA-256 hash of critical fields (input, output, spans) alongside rows
  • evalyn storage-verify checking all rows against stored checksums
  • Detect corruption from concurrent writes or filesystem errors
• Storage Anonymous Export - Strip identifying information when sharing databases
  • evalyn storage-export --anonymous replacing PII-like content with placeholders
  • Preserve data structure, metadata, and statistics while removing content
  • Useful for sharing databases for debugging without exposing user data
• Denormalized Storage Optimization - Flatten trace hierarchy for query performance
  • Langfuse found 10x dashboard speedup by denormalizing trace attributes onto span rows
  • Store trace-level metadata (project, session_id, user_id) on every span row
  • Eliminate JOIN overhead for common query patterns (list spans with trace context)

Data & Dataset

• Dataset Versioning - Track dataset changes over time with diff view
  • Content-hash versioning on each build-dataset invocation
  • Diff view: items added, removed, and modified between versions
  • Version log stored alongside dataset.jsonl
  • Rollback to previous version via evalyn dataset-rollback
• Synthetic Data Generation
  • Adversarial example generation
  • Edge case mining
  • Demographic variation
  • Domain-specific generators
• Data Augmentation - Automatically expand datasets
  • Paraphrase generation: rephrase inputs preserving semantics
  • Input perturbation: typos, casing, formatting variations
  • Language translation: generate multilingual variants
  • Context expansion: add/remove context to test robustness
• Golden Set Management - Curate and maintain evaluation benchmarks
  • evalyn golden-set create/add/remove commands
  • Lock golden set items from modification
  • Track golden set coverage: % of metrics with golden examples
  • Periodic validation: re-evaluate golden set to detect model drift
• Dataset Splitting - Train/test/validation splits with stratification by metadata fields
  • evalyn split-dataset --ratio 0.7/0.15/0.15
  • Stratification by metadata keys (tag, source, difficulty)
  • Deterministic splitting with configurable random seed
  • Output as separate JSONL files in split/ subdirectory
• Dataset Statistics - Auto-compute input/output length distributions, token counts, label balance
  • evalyn dataset-stats command
  • Input/output token count histograms
  • Metadata field value distributions
  • Expected reference coverage (% items with ground truth)
  • Duplicate detection report
• Dataset Merge and Diff - Combine two datasets or show item-level differences between them
  • evalyn dataset-merge --deduplicate
  • evalyn dataset-diff showing added/removed/changed items
  • Conflict resolution for items with same ID but different content
• External Format Import - Import from HuggingFace datasets, LMSYS Arena, or custom CSV schemas
  • evalyn import --format huggingface --dataset-name <name>
  • CSV import with column mapping config
  • LMSYS Arena format (conversation pairs with human preference)
  • Auto-detect format from file extension and content
• Schema Evolution - Handle format changes across dataset versions with automatic migration
  • Version field in dataset header line
  • Automatic migration on load (old format to current)
  • Migration log showing which transformations were applied
• Dataset Sampling Preview - Show sample items and summary stats before building full dataset
  • --preview flag on build-dataset showing 5 sample items
  • Summary: item count, avg input/output length, metadata distribution
  • Confirmation prompt before writing full dataset
• Dataset Pinning - Lock a dataset version hash for reproducible evaluations across environments
  • SHA-256 hash stored in dataset metadata
  • --pinned flag on run-eval to verify hash before evaluation
  • Pin file (.evalyn-pin) for CI/CD reproducibility
• Dataset Lineage - Track which traces and runs produced each dataset item
  • Source trace ID and function_call ID in item metadata
  • Lineage query: "which traces contributed to this dataset?"
  • Reverse lineage: "which datasets use this trace?"
• Dataset Filtering DSL - Query-based item filtering (e.g. "items where output_length > 500 and tag=production")
  • --filter flag on build-dataset and run-eval
  • Operators: =, !=, >, <, contains, matches (regex)
  • Compound filters with AND/OR
  • Filter on metadata fields, input/output length, and item ID patterns
• Incremental Dataset Build - Append new traces to an existing dataset without full rebuild
  • --append flag on build-dataset
  • Track last-build timestamp to only process new traces
  • Deduplication against existing items using hash_inputs
• Dataset Health Check - Validate dataset quality before evaluation
  • Reference coverage: % of items with ground truth (uses _dataset_has_reference logic)
  • Empty/null field detection in input, output, and metadata
  • Duplicate input detection via hash_inputs
  • evalyn dataset-health command with pass/warn/fail summary
• Dataset Decontamination - Detect items that overlap with known LLM benchmark/training data
  • N-gram overlap check against common benchmarks (MMLU, HumanEval, GSM8K)
  • Configurable contamination threshold (default: 13-gram exact match)
  • evalyn dataset-decontaminate --report showing contaminated items
  • Auto-exclude contaminated items from evaluation datasets
• Dataset Drift Detection - Statistical tests comparing input distributions between dataset versions
  • Kolmogorov-Smirnov test on input length, token count distributions
  • Chi-square test on categorical metadata field distributions
  • Embedding centroid shift measurement between versions
  • evalyn dataset-drift --v1 <path1> --v2 <path2> with drift severity score
• Dataset Annotation Coverage Map - Visualize which items have annotations and which need them
  • Per-metric coverage percentage across dataset items
  • ASCII heatmap: items on Y-axis, metrics on X-axis, filled/empty cells
  • Prioritize unannotated items in items with lowest judge confidence
• Dataset from Production Logs - Import HTTP request/response logs as trace-like dataset items
  • Parse common log formats (JSON, Apache, nginx) into DatasetItem input/output
  • evalyn import-logs --format json --input-field request --output-field response
  • Auto-deduplicate against existing traces in storage
• Dataset Snapshot Comparison - Compare two dataset versions showing item-level content diffs
  • Side-by-side text diff for modified items (input or output changed)
  • Summary: items added, removed, modified, unchanged
  • evalyn dataset-snapshot-diff --before <v1> --after <v2>
• Dataset Complexity Scoring - Auto-compute per-item difficulty from input features
  • Heuristics: input length, vocabulary diversity, question complexity indicators
  • Store complexity_score in item metadata for filtering and stratification
  • evalyn dataset-stats --complexity showing difficulty distribution
• Dataset Bias Auditing - Detect systematic biases in input distribution
  • Topic distribution analysis via LLM classification
  • Length and vocabulary skew detection
  • evalyn dataset-audit producing bias report with recommendations
• Dataset Curation Suggestions - LLM-powered gap analysis suggesting items to add
  • Analyze current dataset coverage against metric requirements
  • Suggest input types, edge cases, and scenarios not yet represented
  • evalyn dataset-suggest --dataset <path> producing curation plan
• Dataset A/B Split Generator - Create matched pairs for controlled model comparison
  • Stratified pairing by complexity, topic, and metadata fields
  • Ensure balanced splits for statistical validity
  • evalyn dataset-ab-split --dataset <path> producing split_a.jsonl and split_b.jsonl
• Dataset Subset Extraction - Extract semantically meaningful subsets via clustering
  • Cluster items by embedding similarity into N groups
  • evalyn dataset-subset --clusters N --dataset <path> extracting per-cluster subsets
  • Useful for focused evaluation on specific input categories
• Dataset Embedding Index - Pre-compute and store embeddings for fast similarity queries
  • Build embedding index on build-dataset using SentenceTransformer
  • Store embeddings alongside dataset.jsonl as embeddings.npy
  • Enable fast nearest-neighbor queries for sampling, dedup, and clustering
• Dataset Interleaving - Round-robin merge from multiple datasets for balanced evaluation
  • evalyn dataset-interleave --datasets d1/ d2/ d3/ producing merged dataset
  • Interleave by metadata field (e.g. alternate "production" and "synthetic" items)
  • Source tracking: tag each item with originating dataset
• Dataset Quality Gate - Block evaluation start if dataset fails quality checks
  • Configurable rules in evalyn.yaml: min_items, max_duplicate_rate, required_metadata_fields
  • run-eval refuses to start unless gate passes (--skip-quality-gate to override)
  • Gate report showing which checks passed and failed
• Dataset Item Clustering Report - Show natural clusters with LLM-generated descriptions
  • Auto-cluster items by embedding similarity into K groups
  • LLM-generated label per cluster describing what the items have in common
  • evalyn dataset-clusters --k 5 showing cluster summary with example items
• Dataset Changelog - Automatic log of all build-dataset operations and parameters
  • Append entry to data/changelog.jsonl on each build-dataset invocation
  • Record: timestamp, filters used, item count, sampling mode, hash
  • evalyn dataset-changelog showing chronological build history
• Dataset Cross-Contamination Check - Verify no item leakage between train/test/calibration splits
  • Hash-based check that no item appears in both train and test splits
  • Embedding-based check for near-duplicate items across splits
  • evalyn dataset-xcontam --train <path1> --test <path2> reporting contamination
• Dataset Item Semantic Search - Find items by natural language query using embeddings
  • evalyn dataset-search --query "user asks about refund policy" finding nearest items
  • Uses pre-built embedding index (from Dataset Embedding Index feature)
  • Return top-K matches with similarity scores
• Dataset Format Autodetect - Auto-detect and load from multiple formats without explicit --format flag
  • Detect JSONL, JSON array, CSV, and TSV from file content and extension
  • Auto-map columns to input/output/metadata fields using heuristics
  • Warn when auto-detection is ambiguous and suggest explicit format
• Dataset Metadata Schema Enforcement - Validate item metadata against a defined schema
  • Schema definition in meta.json: required_fields, field_types, allowed_values
  • Validation on build-dataset and import, rejecting non-conforming items
  • evalyn dataset-validate --schema showing validation results

Reporting & Analytics

• Custom Report Templates - User-defined HTML report layouts
  • Jinja2 template engine for HTML report customization
  • Template variables: run data, analysis, insights, charts
  • Built-in templates: executive summary, technical deep-dive, compliance
  • evalyn export --template custom_template.html
• Slack/Discord Notifications - Alert on evaluation completion or failures
  • Slack webhook integration with rich message formatting
  • Discord webhook with embedded metric summary
  • Configurable alert thresholds: only notify on regression or failure
  • Channel routing: different alerts to different channels
• Metric Correlation Analysis - Understand relationships between metrics
• Failure Root Cause Analysis - Automated diagnosis of failures
  • LLM-powered analysis of common patterns in failed items
  • Feature attribution: which input features correlate with failure
  • Failure clustering by root cause category (prompt, data, model, tool)
  • Actionable fix suggestions per failure cluster
• Trend Anomaly Detection - Alert on unusual metric patterns
  • Z-score based anomaly detection on metric time series
  • Configurable sensitivity threshold
  • Automatic alert when anomaly detected during trend analysis
  • Visual anomaly markers in trend charts
• Cohort Analysis - Compare metrics across user-defined item groups (by metadata, input length, etc.)
  • --cohort-by flag on analyze command (split by metadata field)
  • Per-cohort metric statistics and pass rates
  • Cross-cohort comparison table
  • Identify worst-performing cohort with improvement suggestions
• Statistical Significance Testing - P-values and confidence intervals for run-to-run comparisons
  • Two-proportion z-test for pass rate differences
  • Bootstrap confidence intervals for score means
  • Effect size (Cohen's d) alongside p-values
  • Automatic significance flag in compare output
• Judge Confusion Matrix - Visualize agreement/disagreement patterns between judge and human
  • 2x2 matrix: TP/FP/TN/FN per metric
  • ASCII table and HTML heatmap renderers
  • Per-metric confusion matrix in annotation-stats
  • Aggregate confusion matrix across all metrics
• Jupyter Notebook Export - Generate .ipynb with pre-built charts and analysis from eval runs
  • evalyn export --format notebook
  • Pre-built cells: data loading, metric charts, distribution plots, correlations
  • Interactive widgets for filtering by metric, item, or cohort
  • nbformat-based generation (no Jupyter dependency required)
• Metric Budget Analysis - Estimate cost savings from dropping low-signal metrics
  • Compute information gain of each metric (redundancy with others)
  • Cost attribution: how much each metric costs per run
  • Recommended metric subset that preserves N% of signal at minimum cost
• Regression Bisection - Binary search across dataset items to pinpoint exact cause of a regression
  • evalyn bisect --baseline <run1> --current <run2>
  • Identify items that changed from pass to fail
  • Cluster newly-failing items by input features
  • Rank items by regression severity (score delta)
• Comparative Heatmap - Visual heatmap of metric scores across items and runs
  • Items on Y-axis, metrics on X-axis, color = score
  • Multi-run heatmap: side-by-side comparison
  • ASCII heatmap for terminal, HTML/SVG for reports
  • Sort by worst-performing items or metrics
• Failure Taxonomy - Auto-categorize failures into a structured taxonomy (prompt, model, data, tool)
  • LLM-powered categorization of failure reasons
  • Built-in taxonomy: prompt_ambiguity, model_limitation, data_quality, tool_error, hallucination
  • Custom taxonomy definition in evalyn.yaml
  • Taxonomy distribution chart in analysis output
• Analysis Snapshots - Save analysis state at a point in time for later comparison
  • evalyn snapshot --name "pre-refactor" saves RunAnalysis + InsightsReport
  • evalyn compare-snapshots for before/after comparison
  • Snapshots stored in .evalyn/ directory as JSON
• Item Difficulty Estimation - Compute per-item difficulty scores based on cross-run fail rates
  • Aggregate pass/fail across multiple eval runs per item
  • Difficulty score: inverse of average pass rate across runs
  • Rank items by difficulty in analysis output
  • Use difficulty scores to weight calibration and sampling
• Metric Interaction Effects - Detect non-linear interactions between metrics beyond pairwise correlation
  • Chi-square test for co-failure: items failing both A and B more than expected by chance
  • Interaction strength score per metric pair
  • Surface metric pairs with strong interactions in insights report
• Improvement Priority Ranking - Rank metrics by expected ROI: which improvement would raise overall pass rate most
  • Compute marginal gain: if metric M improved by 10%, how much does overall pass rate increase
  • Factor in metric weight from weighting profiles
  • Actionable ranking in insights output: "Fix metric X first for maximum impact"
• Score Distribution Normality Testing - Verify if metric scores follow expected distributions
  • Shapiro-Wilk test per metric score distribution
  • Flag metrics with non-normal distributions (bimodal, heavy-tailed)
  • Recommend appropriate statistical tests based on distribution shape
• Cross-Run Stability Analysis - Measure how stable metric scores are across repeated runs of same data
  • Run same eval N times and compute per-metric coefficient of variation
  • Flag metrics with high variance as unreliable
  • Recommend increasing samples or switching judge model for unstable metrics
• Metric Contribution Analysis - SHAP-style attribution of each metric's contribution to overall pass/fail
  • Compute marginal contribution of each metric to overall item pass rate
  • Identify metrics that are decisive (flip overall pass/fail) vs redundant
  • Visualization: waterfall chart showing per-metric contribution
• Worst-Case Item Identification - Surface items that fail across the most metrics simultaneously
  • Rank items by number of failed metrics (cross-metric failure count)
  • Highlight items that are "universally bad" vs "edge case failures"
  • Useful for prioritizing which agent behaviors to fix first
• Time-to-Fix Tracking - Track how many runs it takes for failing items to start passing
  • Per-item pass/fail history across consecutive runs
  • Average time-to-fix per metric and per failure category
  • Identify persistently failing items that resist fixes
• Analysis Report Diff - Diff two RunAnalysis outputs showing what changed
  • evalyn analysis-diff --run1 <id> --run2 <id>
  • Delta per metric: pass rate change, score mean change, new/resolved failures
  • ASCII table with color-coded improvements/regressions
• Run Quality Score - Composite score summarizing overall run health
  • Weighted combination: pass rate, cost efficiency, coverage, judge confidence
  • Single 0-100 score for quick run quality assessment
  • Configurable weights in evalyn.yaml
• Trend Forecasting - Predict future metric values using time series extrapolation
  • Linear regression and exponential smoothing on metric pass rates over runs
  • Forecast next N runs with confidence bands
  • Alert when forecast predicts metric dropping below threshold
• Analysis Natural Language Summary - LLM-generated plain English analysis report
  • Summarize key findings, regressions, and recommendations in 3-5 paragraphs
  • evalyn analyze --summary producing human-readable narrative
  • Useful for sharing results with non-technical stakeholders
• Metric Volatility Index - Measure historical stability of each metric across runs
  • Coefficient of variation across last N runs per metric
  • Classify metrics as stable, moderate, or volatile
  • Recommend increasing judge samples or switching models for volatile metrics
• Analysis Change Attribution - Attribute metric changes to dataset, model, or prompt factors
  • Detect which factor changed between compared runs (dataset hash, source hash, prompt hash)
  • Attribute score deltas to the changed factor
  • "Pass rate dropped 15%, likely due to dataset change (12 new items added)"
• Analysis Comparison Template - Configurable comparison layouts for different audiences
  • Executive template: overall pass rate, top regressions, cost summary
  • Engineering template: per-metric details, failed item list, prompt diffs
  • --template flag on compare command
• Analysis What-If Simulator - Interactively model "what if metric X improved by N%"
  • evalyn what-if --metric helpfulness --improve 20% showing projected overall pass rate
  • Model multiple simultaneous improvements
  • Identify the minimum improvement per metric needed to reach a target pass rate
• Analysis Dashboard Theming - Configurable chart colors and styles for HTML reports
  • Theme definitions in evalyn.yaml: primary color, accent, chart palette
  • Built-in themes: corporate, academic, dark-mode, print-friendly
  • Custom CSS injection for branded reports
• Analysis Data Export API - Export analysis data as structured Python objects for custom analysis
  • evalyn.analyze_to_dict(run) returning dict-of-lists for pandas DataFrame construction
  • evalyn export --format feather producing columnar format for direct notebook loading
  • Enable custom statistical analysis beyond built-in insights
• Analysis Time Series Decomposition - Separate trend, seasonality, and noise in metric time series
  • Decompose metric pass rates across runs into systematic trend and random variation
  • Distinguish genuine improvement from normal score fluctuation
  • Visualize decomposed components in trend analysis output

Interoperability

• Phoenix/Langfuse Trace Export - Native export to popular LLM observability platforms
  • evalyn export-traces --format phoenix to produce Phoenix-compatible JSONL
  • evalyn export-traces --format langfuse for Langfuse import format
  • Preserve span hierarchy and OpenInference attributes in export
• Trace Import from External Platforms - Bring existing traces into evalyn for evaluation
  • evalyn import-traces --format phoenix/langfuse/otel
  • Map external span types to Evalyn span types via conventions.py
  • Deduplicate against existing traces by span ID
• OpenInference Full Compliance - Complete implementation of OpenInference semantic conventions
  • Full document/retrieval attribute capture (DocumentAttributes, RetrievalAttributes)
  • Embedding attribute capture (EmbeddingAttributes.EMBEDDINGS, TEXT)
  • Session and user attribute propagation (SessionAttributes)
  • Reranker score capture and display in show-trace
• Eval Result Export to Observability Platforms - Push evaluation scores back to trace viewers
  • Annotate Phoenix spans with evalyn metric scores
  • Push eval results as Langfuse scores
  • Bi-directional sync: traces in, scores out

Resilience & Error Handling

• Circuit Breaker for Providers - Stop calling a provider after N consecutive failures
  • Configurable failure threshold (default: 5 consecutive errors)
  • Cool-down period before retrying (exponential backoff)
  • Automatic fallback to alternative provider when circuit opens
  • Circuit state visible in progress output
• Graceful Item-Level Failure - Continue evaluation when individual items fail
  • Catch and log per-item errors without stopping the run
  • Record failure reason in MetricResult.details
  • Summary of failed items at end of run with error categories
  • --fail-fast flag to override and stop on first error
• Provider Fallback Chain - Automatically try alternative providers on failure
  • Ordered provider list: [gemini, openai, ollama]
  • Fall back to next provider on timeout, rate limit, or API error
  • Log which provider was actually used per item
• Evaluation Timeout Per Item - Prevent single slow items from blocking the entire run
  • --item-timeout flag (default: 120s per item)
  • Timeout recorded as failure with reason "timeout"
  • Separate timeout for objective vs subjective metrics

Output & Formatting

• Color-Coded Terminal Output - ANSI colors for pass/fail/warning states
  • Green for pass, red for fail, yellow for warning across all commands
  • Respect NO_COLOR env var and --no-color flag for CI environments
  • Color-coded score ranges in analyze and compare output
• Compact Output Mode - Minimal output for CI logs and scripting
  • --compact flag producing single-line summaries per command
  • Summary format: "RUN <id> PASS 85% (17/20) COST $0.12 TIME 45s"
  • Pair with exit codes for CI gate integration (exit 1 if pass rate < threshold)
• PDF Report Export - Generate PDF reports from HTML dashboards
  • evalyn export --format pdf using headless browser or weasyprint
  • Page breaks between sections, print-friendly layout
  • Cover page with run metadata, date, project name
• HTML Report Dark Mode - Dark theme option for HTML dashboards and insights
  • CSS dark mode support via prefers-color-scheme media query
  • Manual toggle button in report header
  • Dark-friendly Chart.js color palette

Code Change Tracking

• Source Code Diff Correlation - Track agent code changes alongside metric changes
  • Store source_hash from _extract_code_meta in each eval run
  • Detect when source code changed between consecutive runs
  • Correlate code diffs with metric deltas in compare output
  • evalyn code-diff --run1 <id> --run2 <id> showing code changes alongside score changes
• Prompt Version Tracking - Track judge prompt changes across calibration rounds
  • Hash judge prompts and store in MetricResult metadata
  • Warn when comparing runs that used different prompt versions
  • Prompt changelog: show how each metric's prompt evolved over time

Programmatic SDK

• Python API for Running Evaluations - Run evaluations from Python code without CLI
  • evalyn.run(dataset, metrics, provider) returning EvalRun object
  • evalyn.analyze(run) returning RunAnalysis directly
  • evalyn.compare(run_a, run_b) returning comparison dict
  • Async variants: await evalyn.run_async(...)
• Event Callback Hooks - Register functions that fire on evaluation events
  • on_item_complete(callback) for per-item processing
  • on_metric_complete(callback) for per-metric processing
  • on_run_complete(callback) for post-run triggers
  • Hook registration via evalyn.yaml or Python API
• Context Manager Tracing - Manual span creation with with syntax
  • with evalyn.span("name", "type") as s: for explicit span boundaries
  • Automatic parent-child linking via context propagation
  • Span attribute setting: s.set_attribute("key", "value")
• Embedding as Library - Use evalyn as imported library in test suites
  • pytest plugin: @pytest.mark.evalyn(metrics=["helpfulness"])
  • Assert on metric scores: assert result.metrics["helpfulness"].passed
  • Integration with pytest-xdist for parallel testing
• Declarative Evaluation API - Single-call evaluation matching industry patterns
  • Braintrust-style: evalyn.Eval("project", data=fn, task=fn, scores=[...])
  • Weave-style: evalyn.Evaluation(dataset=..., scorers=[...]).run(model)
  • Both patterns return structured results with .to_pandas() support
• Semantic Caching for Judge Calls - Cache identical LLM judge calls to reduce cost
  • Content-addressable cache keyed by hash(prompt + input + output + model)
  • Research finding: up to 68.8% API call reduction (GPTCache benchmark)
  • Optional embedding-based fuzzy matching for similar-but-not-identical inputs

Testing & Quality Enhancements

• Snapshot Testing for Metrics - Detect unintended changes to metric scoring behavior
  • Record expected scores for a golden dataset
  • Flag when metric output changes (new code, model update)
  • evalyn test-metrics --update-snapshots to accept changes
• Performance Benchmark Suite - Track and prevent performance regressions in evalyn itself
  • Benchmarks for: dataset loading, metric scoring, analysis, export
  • Baseline timings stored in repo
  • CI check: fail if any benchmark regresses > 20%
• Fuzz Testing for Parsers - Stress-test JSON/judge output parsing with malformed inputs
  • Fuzz _extract_json_object and extract_json_list with random strings
  • Fuzz _parse_passed with edge case values
  • Ensure no unhandled exceptions on any input
• Sandboxed Agent Evaluation - Safe execution environment for agent evals where models run code
  • Docker-based sandbox for executing agent tool calls safely (Inspect AI pattern)
  • Configurable timeout and resource limits per sandbox
  • Capture sandbox output as part of trace spans
• Composable Assertion Framework - PromptFoo-style assertion primitives for evaluation
  • Assertion types: contains, not_contains, regex_match, llm_rubric, similar, cost_below
  • Composable with AND/OR logic for complex pass/fail criteria
  • YAML-configurable assertions in metrics definition
• Evaluation Result Schema Standard - Define a JSON schema for evaluation results
  • Enable cross-platform evaluation result exchange
  • Schema covers: items, metrics, scores, metadata, provenance
  • No universal standard exists yet (industry gap evalyn could fill)
• Knowledge Graph Test Generation - Generate evaluation questions from document knowledge graphs
  • Extract entities and relationships from source documents (Ragas pattern)
  • Generate questions that test understanding of specific relationships
  • Configurable question types: factual, inferential, multi-hop

Packaging & Distribution

• Docker Image - Official Docker image for CI/CD and isolated evaluation environments
  • Dockerfile with evalyn pre-installed and all optional dependencies
  • Configurable via environment variables (API keys, config path)
  • Docker Compose example with SQLite volume mount for data persistence
  • GitHub Actions example using the Docker image for eval-on-PR
• Standalone Binary - Single-file executable without Python dependency
  • PyInstaller or Nuitka build for Linux, macOS, Windows
  • GitHub Releases automation for versioned binaries
  • Install script: curl -sSL https://evalyn.dev/install | sh
• evalyn version and Update Check - Version management and update notifications
  • evalyn version showing installed version and latest available
  • Optional update check on startup (configurable, off by default)
  • evalyn self-update command to upgrade in place

Documentation Generation

• CLI Reference Auto-Generation - Generate CLI docs from argparse definitions
  • evalyn docs --format markdown producing per-command reference pages
  • Include all flags, defaults, examples, and cross-references
  • Auto-update on release via CI
• Metric Catalog - Auto-generated browsable catalog of all 133 metrics
  • evalyn docs --metrics producing metric reference with rubrics, categories, scopes
  • HTML format with search and filter by category/type
  • Include metric bundle membership and recommended use cases
• Config Reference - Auto-generated documentation for evalyn.yaml options
  • Generate from evalyn.yaml.example with type annotations and valid values
  • Show default values, environment variable overrides, and CLI flag mappings

Deprecation & Migration

• Deprecation Warnings - Warn when using deprecated config keys, flags, or APIs
  • Deprecation registry mapping old names to new names
  • Yellow warning on first use, error after N versions
  • evalyn migrate-config to auto-update deprecated config keys
• Breaking Change Detection - Detect when upgrading evalyn would break existing runs
  • Compare metric version hashes between installed version and pinned run manifest
  • Warn before evaluation if metric behavior changed since last run
  • Migration guide output for each detected breaking change

Rubric Engineering

• Multi-Language Rubrics - Judge prompts and rubrics in languages other than English
  • Rubric translation support in JUDGE_TEMPLATES (locale field per template)
  • Language-matched judging: use rubric language matching the output language
  • Cross-language evaluation: judge non-English outputs with English rubrics vs native rubrics
• Community Rubric Library - Import and export rubrics from a shared repository
  • evalyn rubric-export --metric <id> producing a portable YAML rubric file
  • evalyn rubric-import from URL or local file
  • Rubric metadata: author, version, tested-on, accuracy stats
• Rubric Testing - Validate that a rubric produces consistent scores on test cases
  • evalyn test-rubric --metric <id> running rubric against a set of known pass/fail items
  • Consistency score: same rubric, same item, N runs, measure agreement
  • Edge case detection: find items where rubric is ambiguous (close to threshold)
• Domain-Specific Rubric Packs - Downloadable rubric sets for specialized domains
  • Medical: HIPAA compliance, clinical accuracy, patient safety, drug interaction checks
  • Legal: jurisdictional accuracy, precedent citation, privilege preservation
  • Finance: SEC compliance, fiduciary duty, risk disclosure completeness
  • evalyn install-rubric-pack medical

Dashboard Interactivity

• Embeddable Widget Mode - Iframe-friendly dashboard for embedding in other tools
  • evalyn dashboard --embed producing minimal HTML without navigation chrome
  • Configurable widget size and chart selection
  • PostMessage API for parent page communication (filter events, score updates)
• In-Dashboard Data Export - CSV/JSON export buttons on each chart in HTML reports
  • Download button per chart exporting underlying data as CSV
  • Full dataset export button in failed items section
  • Copy-to-clipboard for individual metric summaries
• Comparison Overlay Dashboard - Overlay two runs on same charts for visual comparison
  • evalyn dashboard --compare <run1> <run2>
  • Dual bar charts, overlaid radar plots, side-by-side heatmaps
  • Toggle visibility of each run for clean comparison

Audit & Governance

• Evaluation Audit Trail - Immutable log of who ran what and when
  • Record: user, timestamp, command, args, config hash, result summary
  • Append-only audit log in .evalyn/audit.jsonl
  • evalyn audit-log showing evaluation history with filters
• Data Governance Metadata - Track data provenance and compliance attributes
  • Dataset-level tags: PII-present, internal-only, customer-data, synthetic
  • Eval run compliance flag: was evaluation run on approved infrastructure?
  • Exportable governance report for compliance audits
• Structured Logging - JSON-formatted logs with configurable verbosity
  • --log-level flag (debug, info, warning, error) on all commands
  • JSON log format for machine parsing in production environments
  • Log file output: --log-file evalyn.log

Security

• API Key Rotation Support - Gracefully handle key rotation without interrupting evaluation runs
  • Accept multiple API keys per provider in evalyn.yaml (primary + fallback)
  • Automatic fallback to secondary key when primary returns 401/403
  • evalyn rotate-key --provider gemini to update key and verify connectivity
• Secrets Backend Integration - Load API keys from external secret managers instead of plaintext config
  • Support AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault
  • evalyn.yaml secrets_backend: "aws" with ARN references
  • Environment variable passthrough as default (no config change needed)
• Trace Content Redaction Policies - Configurable rules for what gets stored in trace payloads
  • Policy definitions in evalyn.yaml: never store full messages, only store first/last N chars
  • Per-project redaction rules (strict for production, relaxed for test)
  • Redaction audit: report showing how much content was redacted per trace
• Prompt Injection Detection Metric - Objective metric detecting prompt injection attempts in inputs/outputs
  • Tier 1: 4-category regex patterns (instruction override, role injection, prompt extraction, encoding signals)
  • Tier 2: optional LLM-based classification for higher accuracy
  • Tier 3: optional vector similarity against known attack embeddings (self-hardening via Rebuff pattern)
  • Scoring: 0.0 (injection detected) to 1.0 (clean), configurable sensitivity
• Embedding PII Safety Check - Detect whether stored embeddings could leak PII via inversion attacks
  • Warn when embedding vectors are stored alongside PII-containing text
  • Research finding: 93-98% text recovery from ada-002 embeddings via inversion
  • Recommend PII stripping before embedding or Eguard-style defense
• EU AI Act Compliance Report - Auto-generate evaluation documentation for regulatory compliance
  • Document evaluation methodology, benchmarks used, and results
  • Export as PDF/HTML for regulatory submission
  • Cover NIST AI RMF and ISO 42001 reporting requirements

Offline & Air-Gapped Mode

• Fully Offline Evaluation - Run complete evaluation pipeline without internet access
  • Objective-only mode: all 73 objective metrics work offline with no API calls
  • Ollama provider for subjective metrics using local models
  • Pre-download and cache model artifacts for sentence-transformers embeddings
  • evalyn run-eval --offline flag that errors if any metric would require internet
• Local Model Performance Baselines - Benchmark local models against API models for judge quality
  • evalyn benchmark-judges --local ollama:llama3 --api gemini comparing alignment
  • Per-metric local vs API agreement scores
  • Recommend which metrics are safe to evaluate locally

Scale & Performance

• Large Dataset Optimization - Handle 10k+ item datasets without memory issues
  • Streaming evaluation: process items without loading full dataset into memory
  • Chunked metric result storage: write results in batches to avoid OOM
  • Progress checkpointing every N items (currently only on interrupt)
  • Memory usage monitoring and warning when approaching system limits
• SQLite Full-Text Search - FTS5 index for searching trace content and outputs
  • FTS index on function_call inputs and outputs
  • evalyn search "user asked about refund policy" finding matching traces
  • Search integration with build-dataset for content-based dataset curation
• Aggregation Queries - Efficient database queries for cost and usage analytics
  • Cost by project, by date range, by model
  • Trace count and token usage per provider
  • evalyn stats --project <name> --since 2026-03-01 for project-level analytics

---

Completed Features

Setup & Configuration

• evalyn init - Initialize evalyn.yaml config file
• evalyn one-click - Run complete pipeline in one command
• evalyn help - Show available commands with examples
• Environment Variables - GEMINI_API_KEY, OPENAI_API_KEY, EVALYN_NO_HINTS, EVALYN_AUTO_INSTRUMENT

Tracing & Instrumentation

• @eval decorator - Automatic function call tracing
• Auto-instrumentation - Automatic LLM SDK patching (OpenAI, Anthropic, Gemini, LangChain, LangGraph)
• Span tree capture - Hierarchical trace of LLM calls, tool calls, graph nodes
• Token & cost tracking - Automatic token counting and cost estimation
• evalyn list-calls - List captured traces with filtering and sorting
• evalyn show-call - View detailed call information
• evalyn show-trace - Phoenix-style span tree visualization
• evalyn show-projects - Project summary with trace counts
• Streaming response capture - StreamingSpanWrapper for OpenAI, Anthropic, Gemini
• GenAI semantic convention attributes - OpenTelemetry gen_ai.* attributes on spans
• Span-metric attribution - Link metric results to specific spans with relevance scoring
• Context window utilization tracking - Track context usage in spans
• --db flag - Switch between prod/test databases
• Short ID support - 8-character ID prefixes for convenience

Dataset Management

• evalyn build-dataset - Build dataset.jsonl from traces
• evalyn validate - Validate dataset format
• evalyn status - Show comprehensive dataset status
• --latest flag - Auto-resolve most recent dataset
• Production/simulation filtering - Separate real vs synthetic traces
• Date range filtering - --since and --until options

Metrics System

• 73 Objective Metrics - Deterministic code-based evaluation
  • Efficiency: latency_ms, cost, token_length, compression_ratio
  • Structure: json_valid, json_schema_keys, regex_match, xml_valid, syntax_valid
  • Correctness: bleu, rouge_l, rouge_1, rouge_2, exact_match, levenshtein_similarity
  • Robustness: tool_call_count, llm_call_count, tool_success_ratio, retry_count
  • Grounding: url_count, citation_count, source_diversity
  • Style: word_count, sentence_count, avg_sentence_length, vocabulary_diversity
  • Diversity: unique_ngrams, type_token_ratio
• 60 Subjective Metrics - LLM judge evaluation
  • Safety: toxicity_safety, pii_safety, manipulation_resistance, bias_detection
  • Correctness: helpfulness_accuracy, factual_accuracy, technical_accuracy
  • Style: tone_alignment, formality_match, brand_voice_consistency
  • Instruction: instruction_following, constraint_adherence, format_compliance
  • Grounding: hallucination_risk, source_attribution, claim_verification
  • Agent: reasoning_quality, tool_use_appropriateness, planning_quality
  • Domain: medical_accuracy, legal_compliance, financial_prudence
  • Conversation: context_retention, memory_consistency, empathy, patience
• evalyn list-metrics - List all available metrics
• evalyn suggest-metrics - Suggest metrics for a function
  • basic mode - Fast heuristic-based
  • bundle mode - Pre-configured metric sets
  • llm-registry mode - LLM picks from registry
  • llm-brainstorm mode - LLM generates custom metrics
  • auto mode - Uses function hints or defaults
• evalyn select-metrics - Interactive LLM-guided selection

Metric Bundles (17 Curated Sets)

• Conversational AI
  • chatbot - Safety, helpfulness, multi-turn memory
  • customer-support - Empathy, patience, escalation handling
• Content Generation
  • content-writer - Style, engagement, readability
  • summarization - Compression, reference overlap, grounding
  • creative-writer - Originality, engagement, vocabulary diversity
• Knowledge & Research
  • rag-qa - Grounding, citations, factual accuracy
  • research-agent - Citations, grounding, tool use
  • tutor - Pedagogical clarity, examples, patience
• Code & Technical
  • code-assistant - Syntax validity, complexity, technical accuracy
  • data-extraction - JSON validity, schema compliance
• Agents & Orchestration
  • orchestrator - Tool success, planning, error handling
  • multi-step-agent - Planning, context retention, memory
• High-Stakes Domains
  • medical-advisor - Medical accuracy, safety, ethics
  • legal-assistant - Legal compliance, citations, accuracy
  • financial-advisor - Financial prudence, safety, ethics
• Safety & Translation
  • moderator - Toxicity, bias, PII, manipulation
  • translator - BLEU, Levenshtein, cultural sensitivity

Evaluation Engine

• evalyn run-eval - Run evaluation on dataset
• Parallel execution - Multi-threaded metric evaluation (--workers)
• Batch API mode - 50% cost savings for large-scale evaluation (--batch)
  • Gemini batch provider
  • OpenAI batch provider
  • Anthropic batch provider
• Confidence estimation - Confidence scores for LLM judgments (--confidence)
  • Logprobs-based confidence (OpenAI/Ollama)
  • DeepConf confidence (Meta AI's bottom-10% strategy)
  • Self-consistency confidence (multi-sample agreement)
  • Perplexity and entropy methods
• Multi-provider support - Choose judge provider (--provider)
  • Gemini (default)
  • OpenAI
  • Ollama (local)
• Token usage tracking - Track LLM API token consumption per eval run
  • Per-metric input/output token counts
  • Aggregated usage summary in EvalRun
  • Display in run-eval output and show-run command
• Checkpoint & resume - Save progress on interrupt, resume later
• HTML reports - Interactive visualization with Chart.js
• evalyn list-runs - List past evaluation runs
• evalyn show-run - View run details
• --use-calibrated - Apply calibrated prompts

Analysis & Insights

• evalyn analyze - Analyze evaluation results
• evalyn compare - Compare two runs side-by-side
• evalyn trend - View metric trends over time
• evalyn cluster-failures - Cluster failed items by failure reason
• evalyn cluster-misalignments - Cluster judge vs human disagreements
• Pass rate charts - ASCII bar charts in terminal
• Score distributions - Mini histograms
• Failed item breakdown - List items with failure reasons
• evalyn insights - Comprehensive diagnostic, prescriptive, and proactive analysis
  • Metric correlations, regressions, distributions, feature analysis
  • Prioritized recommendations
  • LLM expert panel (--deep) with 4 expert roles + moderator synthesis
  • Interactive HTML dashboard (--format html) with Chart.js charts

Annotation Enhancements

• Inter-Annotator Agreement - Track and visualize consistency between multiple annotators
  • Cohen's Kappa and Krippendorff's Alpha per metric
  • Pairwise agreement matrix across annotators
  • Identify items with highest disagreement for re-annotation
  • Agreement trend over time as annotators calibrate
• Annotation Delegation - Assign specific items to specific annotators by expertise
  • Annotator profiles with domain expertise tags
  • Auto-assignment based on item metadata and annotator expertise match
  • Workload balancing across annotators
  • Progress dashboard per annotator
• Bulk Pre-Annotation via LLM - Use LLM to pre-fill annotations for human review and correction
  • evalyn pre-annotate --provider gemini to generate draft annotations
  • Confidence-based triage: auto-accept high-confidence, human-review low-confidence
  • Track pre-annotation accuracy vs human corrections
  • Use corrections to improve pre-annotation prompts
• Annotation Guidelines Generator - Auto-generate annotation guidelines from metric definitions
  • Convert metric rubrics to annotator-friendly instructions
  • Include concrete pass/fail examples from existing annotations
  • Export as markdown document or HTML with examples
• Annotation Conflict Resolution UI - Side-by-side view when annotators disagree, with tiebreaker workflow
  • Display both annotators' labels with their confidence and reasoning
  • Third-party tiebreaker annotation with full context
  • Resolution policies: majority vote, senior override, discussion required
• Annotation UX Improvements - Faster, more forgiving annotation workflow
  • Undo/edit previous annotation without re-annotating from scratch
  • Skip items with "s" key (mark as skipped, return to later)
  • Keyboard shortcuts: y=pass, n=fail, 1-5=confidence, s=skip, u=undo
  • Batch mode: present N items at once for rapid annotation
• Annotation Session Persistence - Save and resume annotation progress
  • Track annotated item IDs in session file per annotator
  • evalyn annotate --resume to continue where last session ended
  • Session statistics: items/hour, agreement rate over time

Human Annotation

• evalyn annotate - Interactive annotation interface
  • Simple mode - Overall pass/fail
  • Per-metric mode - Agree/disagree with each metric
  • Span mode - Annotate individual LLM/tool calls
• evalyn annotation-stats - Show annotation coverage
• evalyn import-annotations - Import from JSONL
• evalyn export-for-annotation - Export for external tools
• Confidence scores - 1-5 scale for annotation certainty
• Immediate save - Each annotation saved instantly

Calibration (LLM Judge Optimization)

• evalyn calibrate - Optimize judge prompts
  • Basic method - Single-shot LLM analysis of disagreements
  • APE method - Search-based optimization with UCB selection
  • OPRO method - Trajectory-based optimization
  • GEPA method - Evolutionary prompt optimization (external library)
  • GEPA-Native method - Evolutionary optimization with token tracking
  • EvoPrompt method - Population-based mutation/crossover
  • TextGrad method - Iterative critique-revise refinement
  • MIPROv2 method - Joint instruction + few-shot demo optimization
  • PromptBreeder method - Self-referential prompt evolution
  • BaseOptimizer base class + factory dispatch
• evalyn list-calibrations - List calibration records
• Alignment metrics - Accuracy, precision, recall, F1, Cohen's Kappa
• Validation split - Test calibration on held-out samples

Simulation (Synthetic Data)

• evalyn simulate - Generate synthetic test data
  • similar mode - Variations of existing queries
  • outlier mode - Edge cases and unusual inputs
• Temperature control - Separate temps for similar/outlier
• Seed sampling - Control number of seed examples
• Persona-Based Simulation - Generate inputs as specific user personas (novice, expert, adversarial)
  • Built-in personas: novice user, power user, adversarial attacker, non-native speaker
  • Custom persona definitions in evalyn.yaml
  • Persona tag in generated item metadata for cohort analysis
• Multi-Turn Simulation - Generate full multi-turn conversations, not just single queries
  • Configurable conversation length (2-10 turns)
  • Follow-up generation based on agent response
  • Conversation flow patterns: clarification, topic shift, error recovery
• Adversarial Simulation - Deliberately craft inputs targeting known failure modes
  • Prompt injection attempts
  • Boundary inputs: empty, max length, special characters, unicode edge cases
  • Contradiction inputs that conflict with system prompt
  • Jailbreak pattern variations
• Domain Transfer Simulation - Adapt seed inputs from one domain to another (e.g. medical to legal)
  • LLM-powered domain rewriting preserving query structure
  • Domain vocabulary substitution
  • Complexity preservation across domain transfer
• Regression Simulation - Re-generate past failure inputs to verify they no longer fail
  • Extract failure patterns from cluster-failures output
  • Generate new inputs matching each failure pattern
  • Track fix rate: % of previously-failing patterns now passing
• Conditional Simulation - Generate inputs that specifically test edge conditions (empty input, max length, unicode)
  • Edge condition library: empty, null, max_length, unicode, mixed_language
  • Combinatorial generation across edge conditions
  • Configurable via --conditions flag
• Simulation Validation - Auto-verify that generated items match expected statistical distributions
  • Input length distribution comparison (generated vs seed)
  • Vocabulary overlap check between generated and seed
  • Deduplication against both seed and existing dataset
• Parallel Simulation - Generate synthetic data with configurable concurrency for large-scale runs
  • --workers flag on simulate command
  • Batch LLM calls for generation efficiency
  • Progress bar with items generated / total target
• Structured Input Simulation - Generate dict/JSON inputs, not just text prompts
  • Infer input schema from seed dataset items (detect keys, types, value ranges)
  • Generate valid structured inputs conforming to detected schema
  • Configurable field-level variation (mutate one field at a time for targeted testing)
• Seed Selection Optimization - Choose which seed items produce the most diverse simulations
  • Score seeds by diversity of generated outputs
  • Greedy selection: pick seeds that maximize coverage of unexplored input space
  • Drop seeds that produce near-duplicate simulations
• Simulation with Reference Answers - Generate both inputs and expected outputs for automatic golden set creation
  • LLM generates input-output pairs where the output serves as ground truth
  • Configurable quality threshold: only keep pairs where LLM confidence is high
  • Useful for bootstrapping evaluation datasets with expected references
• Simulation Coverage Report - Compare embedding space coverage of simulated vs production traces
  • Compute coverage overlap between simulated and real item embeddings
  • Identify production input regions not represented in simulated data
  • Recommend additional simulation targets to fill coverage gaps
• Simulation Budget Optimizer - Given a token budget, optimize the mix of similar/outlier/adversarial items
  • Estimate token cost per simulation mode based on prompt complexity
  • Maximize diversity under budget constraint via greedy allocation
  • Report actual vs budgeted cost after generation
• Constraint-Guided Simulation - Generate inputs satisfying specific constraints
  • --constraint "topic=refunds AND length>200" flag on simulate command
  • LLM-guided generation with constraint verification loop
  • Reject and regenerate items that fail constraint checks
• Simulation Diversity Metrics - Quantify how diverse the generated set is vs seed set
  • Embedding spread: average pairwise distance in generated set
  • Vocabulary uniqueness ratio vs seed set
  • Novelty score: fraction of generated items far from all seed items
• Simulation Evaluation Loop - Generate, evaluate, and iterate on simulated data in one command
  • evalyn simulate-and-eval --rounds 3 running simulate + run-eval in a loop
  • Each round generates items targeting previous round's failure patterns
  • Convergence tracking: stop when pass rate stabilizes
• Simulation with Tool Schemas - Generate inputs that exercise specific tool call patterns
  • Provide tool definitions in evalyn.yaml; simulator generates queries requiring those tools
  • Coverage tracking: % of tools exercised by generated inputs
  • Useful for testing tool selection and parameter correctness
• Simulation Seed Clustering - Cluster seeds before simulation to ensure diverse coverage
  • Auto-cluster seed items into groups by embedding similarity
  • Sample proportionally from each cluster for simulation seeds
  • Prevent simulation from over-representing one cluster of similar inputs
• Simulation Template Library - Pre-built simulation configs for common use cases
  • Templates: customer-support, rag-qa, code-review, multi-step-agent
  • Each template defines persona mix, edge case types, output format constraints
  • evalyn simulate --template customer-support
• Simulation Difficulty Grading - Auto-tag generated items with estimated difficulty level
  • Difficulty heuristics: input complexity, number of constraints, ambiguity level
  • Tag in metadata as difficulty: easy/medium/hard
  • Ensure generated set has balanced difficulty distribution
• Simulation Quality Score - Evaluate generated items for naturalness compared to seed set
  • LLM-based naturalness rating: does this look like a real user query?
  • Statistical comparison: generated vs seed item length/vocabulary distributions
  • Auto-reject generated items scoring below quality threshold
• Simulation Provider Diversity - Use multiple LLM providers to increase variety in generated items
  • Round-robin across configured providers (Gemini, OpenAI, Ollama)
  • Merge results with provider tag in metadata
  • Compare generation quality per provider
• Simulation Cost Estimation - Estimate token cost before running simulation
  • --dry-run flag on simulate showing estimated tokens and cost
  • Cost breakdown: similar mode vs outlier mode estimates
  • Useful for budgeting large-scale simulation runs
• Simulation Reproducibility Seed - Deterministic seed for exact reproduction of generated items
  • --seed flag on simulate command for reproducible LLM outputs (temperature + seed)
  • Record seed in simulation metadata for audit trail
  • Verify reproducibility: re-run with same seed produces identical items
• Simulation Feedback Injection - Inject specific failure patterns into simulation prompts
  • Accept failure cluster labels from cluster-failures as simulation targets
  • Generate items specifically designed to trigger each failure mode
  • Coverage tracking: % of known failure patterns with generated test cases
• Evol-Instruct Data Evolution - Evolve evaluation items through iterative complexity increases
  • In-depth evolution: add constraints, reasoning steps, edge cases to existing items
  • In-breadth evolution: generate topic variations and domain transfers
  • Quality scoring: rate evolved items on clarity, depth, structure, relevance
  • Auto-filtering: reject evolved items that degrade below quality threshold
• Persona Hub Integration - Generate diverse user personas for simulation
  • Large-scale persona generation from behavior descriptions
  • Persona-to-Persona expansion for combinatorial diversity
  • Structured diversity controls: ensure coverage across demographics, expertise, intent
• Cascade Model Routing for Evaluation - Use cheap models for easy items, expensive for hard
  • Difficulty estimation from input complexity heuristics
  • Route easy items to flash-lite, hard items to flash/pro
  • 87% cost reduction benchmark (ETH Zurich finding)
  • Quality estimator to determine when to escalate

Sampling

• Importance Sampling - Weight sample selection by item difficulty or model uncertainty
  • Weight by inverse pass rate from previous eval run
  • Weight by judge confidence (low confidence = high importance)
  • Configurable weight function via Python callable
• Curriculum Sampling - Order samples from easy to hard for progressive evaluation
  • Difficulty estimation from input length, complexity heuristics, or past scores
  • Progressive disclosure: evaluate easy items first, add harder ones
  • Early stopping if easy items already fail
• Time-Weighted Sampling - Prefer recent traces over older ones during dataset construction
  • Exponential decay weighting by trace timestamp
  • Configurable half-life parameter (e.g. 7 days, 30 days)
  • Minimum representation guarantee for older traces
• Coverage-Aware Sampling - Maximize coverage of the input feature space
  • Embedding-based coverage using existing SentenceTransformer infrastructure
  • Greedy maximal-diversity selection
  • Coverage report: % of embedding space represented
• Balanced Sampling - Ensure equal representation across metadata categories or labels
  • Balance by any metadata field (tag, source, difficulty)
  • Undersample majority or oversample minority categories
  • Report sampling ratio adjustments applied
• Adversarial Sampling - Select items most likely to trigger model failures based on past results
  • Prioritize items that failed in previous runs
  • Select items near decision boundaries (scores close to threshold)
  • Include items from underperforming cohorts
• Score-Stratified Sampling - Ensure representation across the full metric score range
  • Bin items by score range (0-0.2, 0.2-0.4, ..., 0.8-1.0)
  • Equal sampling from each bin
  • Useful for calibration datasets needing score diversity
• Embedding Drift Sampling - Prioritize items whose embeddings shifted most between dataset versions
  • Compute per-item embedding delta between old and new dataset
  • Sample items with largest cosine distance change
  • Useful for targeting evaluation on items most affected by data updates
• Cost-Aware Sampling - Prefer shorter/cheaper items when evaluation budget is constrained
  • Estimate per-item evaluation cost from input/output token counts
  • Greedy selection maximizing item count within token/cost budget
  • --max-eval-cost flag on build-dataset to cap total evaluation expense
• Human Disagreement Sampling - Prioritize items where annotators previously disagreed
  • Query annotation store for items with divergent human labels
  • Weight by disagreement severity (binary flip vs minor score difference)
  • Useful for building targeted calibration datasets
• Cluster Boundary Sampling - Sample items near cluster decision boundaries for maximum information gain
  • Identify items closest to cluster centroids vs farthest from all centroids
  • Preferentially sample boundary items that are hardest to classify
  • Combine with existing clustered sampling mode
• Bootstrap Resampling - Generate bootstrap samples for confidence interval estimation on metrics
  • --bootstrap N flag on run-eval to create N resampled evaluation runs
  • Report 95% confidence intervals for each metric from bootstrap distribution
  • Useful for small datasets where point estimates are unreliable
• Similarity-Based Sampling - Sample items most or least similar to a given reference item
  • --similar-to <item-id> flag selecting nearest neighbors by embedding distance
  • --dissimilar-to <item-id> for maximum diversity from a reference
  • Useful for focused investigation around a specific failure or success case
• Error-Pattern Sampling - Preferentially sample items matching known failure patterns
  • Extract failure patterns from cluster-failures output
  • Match new items against known patterns via embedding similarity
  • Ensures calibration and evaluation sets include known-hard cases
• Progressive Sampling - Start with small sample, expand if metrics are statistically inconclusive
  • Initial sample of N items, evaluate, check confidence intervals
  • Expand sample size if CI width exceeds threshold
  • Stop when statistical power is sufficient or budget exhausted
• Metadata-Conditional Sampling - Variable sample rates by metadata field values
  • Config: sample 100% of "production" items, 20% of "test" items
  • Per-field rate definitions in evalyn.yaml or --sample-by flag
  • Report actual sampling ratios applied per metadata value
• Novelty Sampling - Prioritize items most unlike the existing labeled/annotated set
  • Compute embedding distance from each unlabeled item to nearest labeled item
  • Sample items with maximum novelty for annotation or calibration
  • Expand labeled set coverage efficiently
• Sampling Reproducibility Report - Log exactly which items were selected and why
  • Record sampling mode, seed, parameters, and selected item IDs in meta.json
  • Verify reproducibility: re-run with same params produces identical selection
  • Audit trail for dataset construction decisions
• Multi-Stage Sampling Pipeline - Chain arbitrary sampling strategies in sequence
  • Config: sampling_pipeline: [deduplicate, stratified, diverse] in evalyn.yaml
  • Each stage feeds its output as input to the next
  • Per-stage statistics showing how many items survived each filter
• Sampling Impact Analysis - Estimate how sample size affects metric confidence intervals
  • Given historical run data, compute expected CI width for different sample sizes
  • evalyn sample-impact --dataset <path> --sizes 50,100,200 showing precision vs cost
  • Recommend minimum sample size for target precision level
• Locale-Aware Sampling - Sample proportionally by language or region for i18n testing
  • Detect language/locale from input text or metadata field
  • Ensure minimum representation per locale in sample
  • --sample-by locale flag on build-dataset
• Embedding Model Selection - Configurable embedding model for diversity and clustered sampling
  • embedding_model setting in evalyn.yaml (default: all-MiniLM-L6-v2)
  • Support custom models from HuggingFace or local paths
  • Cache embeddings keyed by model name to avoid recomputation
• Reservoir Sampling - Online sampling for streaming dataset construction
  • Build dataset from continuous trace stream without knowing total count upfront
  • Maintain fixed-size sample with uniform probability guarantees
  • Useful for production monitoring: always keep a representative sample of recent traces
• Coreset Sampling - Find minimal representative subset preserving distribution properties
  • Greedy coreset construction minimizing maximum approximation error
  • Guarantee that statistics computed on coreset approximate full dataset within bounds
  • --coreset N flag on build-dataset for maximum compression with minimal information loss
• IRT-Based Tiny Benchmarks - Use Item Response Theory to find minimal representative subset
  • Psychometrics-inspired: 100 items can replace 14K (140x reduction) within 2% error
  • Estimate item difficulty and discrimination from historical eval data
  • Select items maximizing information at target ability level
  • evalyn dataset-optimize --method irt --target-size 100
• BenchBuilder Auto-Curation - Automatically curate evaluation prompts from production traces
  • Cluster production traces by topic (Arena-Hard pattern)
  • Score each trace for quality and difficulty
  • Select diverse, high-quality traces as evaluation dataset
  • 98.6% human correlation at $20 cost (Arena-Hard benchmark)

Export & Reporting

• evalyn export - Export results in multiple formats
  • JSON - Full structured data
  • CSV - Spreadsheet-compatible
  • Markdown - Human-readable report
  • HTML - Standalone interactive report
• evalyn export-for-annotation - Export for external annotation tools

Additional Export Formats

• Parquet Export - Columnar format for big data tooling and ML pipelines
  • evalyn export --format parquet using pyarrow (optional dependency)
  • Schema: one row per (item, metric) pair with score, passed, details columns
  • Efficient for loading into pandas, DuckDB, or Spark
• OpenAI Evals Format Export - Compatibility with OpenAI's evaluation framework
  • evalyn export --format openai-evals producing JSONL in OpenAI evals schema
  • Map evalyn MetricResult to OpenAI eval sample format
  • Include system prompt and messages for replay in OpenAI's eval harness
• Experiment Tracker Integration - Push eval results to W&B, MLflow, or Neptune
  • evalyn export --format wandb logging metrics as W&B runs
  • evalyn export --format mlflow logging as MLflow experiments
  • Configurable tracker URL and credentials in evalyn.yaml

Developer Experience

• Context-aware hints - Suggests next steps after each command
• --quiet flag - Suppress hints
• --format flag - table/json output for all commands
• --last flag - Quick access to most recent item
• Short IDs - 8-character prefixes for easier use
• Error messages with hints - Helpful troubleshooting suggestions

CLI Enhancements

• Interactive TUI Mode - Rich terminal UI with navigation, filtering, and drill-down
  • Textual or Rich-based TUI framework
  • Views: trace list, run list, metric dashboard, item detail
  • Keyboard navigation: j/k scroll, enter drill-down, q quit
  • Real-time eval progress view with per-metric status
• Shell Completion - Bash/zsh/fish tab completion for all commands and flags
  • argcomplete integration for automatic completion generation
  • Complete command names, flag names, and flag values (run IDs, dataset paths)
  • Installation helper: evalyn --install-completion
• Watch Mode - Auto-rerun evaluation when dataset or config file changes
  • File watcher on dataset.jsonl and evalyn.yaml
  • Debounce: wait 2s after last change before re-running
  • Diff output: only show changed metrics since last run
  • --watch flag on run-eval command
• Profile Command - Show storage size, run counts, disk usage, and system health
  • Database file size and table row counts
  • Total eval runs, traces, and annotations
  • Disk usage by data directory
  • Python environment info: version, installed providers, API key status
• Config Validation Command - Check evalyn.yaml for errors, missing fields, and deprecations
  • Schema validation against expected evalyn.yaml structure
  • Warn on unknown keys, deprecated fields, and type mismatches
  • Suggest fixes for common misconfigurations
  • evalyn config-check command
• evalyn doctor - Diagnose common setup issues (missing API keys, stale data, broken config)
  • Check API key validity for each configured provider
  • Verify database accessibility and schema version
  • Check disk space and write permissions
  • Verify Python dependencies are installed (sentence-transformers, etc.)
  • Generate diagnostic report for bug reports
• evalyn playground - Interactive prompt testing with live metric scoring in the terminal
  • Enter input, see agent output, instantly score with selected metrics
  • Side-by-side: original prompt vs modified prompt
  • Score history across playground iterations
  • Save good examples to dataset
• evalyn diff - Diff two evaluation runs showing changed scores per item
  • Per-item score delta table sorted by largest regression
  • Metric-level summary: improved/regressed/unchanged counts
  • --threshold flag to only show items with delta > N
  • ASCII color coding: green for improvement, red for regression
• evalyn gc - Garbage collect orphaned data (stale checkpoints, runs without datasets)
  • Identify orphaned checkpoint files without matching runs
  • Find runs referencing deleted datasets
  • Remove temporary files in .evalyn/ directory
  • --dry-run mode showing what would be cleaned
• Piped JSON Mode - Machine-readable JSON output for scripting and CI pipeline integration
  • --output json on all commands producing structured JSON to stdout
  • JSONL streaming for long-running operations (progress events)
  • Exit codes: 0=pass, 1=fail, 2=error for CI gate integration
  • jq-friendly output structure
• CLI Plugin System - Register custom commands via Python entry points
  • evalyn.commands entry point group for third-party command modules
  • Auto-discovery and registration at startup
  • evalyn list-plugins showing installed command plugins
• CLI Alias Support - User-defined command aliases in evalyn.yaml
  • aliases: section mapping short names to full commands (e.g. "q" -> "quickstart")
  • Aliases can include default flags (e.g. "fast-eval" -> "run-eval --workers 8 --provider ollama")
  • evalyn alias list showing configured aliases
• CLI Command History - Record and replay command sequences for reproducible workflows
  • Auto-log commands to .evalyn/history.jsonl with timestamps and exit codes
  • evalyn history showing recent commands
  • evalyn replay --from <timestamp> to re-run a sequence of commands
• CLI Batch Script - Run multiple commands from a script file
  • evalyn batch commands.txt executing one command per line
  • Stop-on-error vs continue-on-error modes
  • Variable substitution: $DATE, $LATEST_RUN, $LATEST_DATASET
• CLI Output Pagination - Built-in pager for long terminal outputs
  • Auto-page when output exceeds terminal height
  • Respect PAGER env var, default to less
  • --no-pager flag to disable for piping
• CLI Notification on Completion - System notification when long-running commands finish
  • Desktop notification via notify-send (Linux), osascript (macOS), or toast (Windows)
  • --notify flag on run-eval, calibrate, and one-click commands
  • Include pass/fail summary in notification body
• CLI Config Show - Display effective merged configuration from all sources
  • evalyn config-show displaying global + project + env var + flag overrides
  • Highlight which source each setting comes from
  • Useful for debugging "why is this provider being used?"
• CLI Compare Shorthand - Quick comparison shortcuts for common comparison patterns
  • evalyn compare --last-2 comparing two most recent runs
  • evalyn compare --latest-vs-pinned comparing latest against pinned baseline
  • evalyn compare --latest-vs-previous for sequential regression checking
• CLI Checkpoint Inspection - View and manage evaluation checkpoints
  • evalyn checkpoints listing all saved checkpoints with item counts and timestamps
  • evalyn checkpoint-info <id> showing checkpoint details
  • evalyn checkpoint-delete <id> cleaning up stale checkpoints
• CLI Pipeline Visualization - Show pipeline steps as ASCII flowchart before execution
  • evalyn one-click --show-plan displaying step sequence with estimated times
  • Indicate which steps will be skipped based on flags
  • Confirm before executing the visualized plan
• CLI Side-by-Side View - Display two outputs side by side in terminal
  • evalyn compare --side-by-side rendering left/right columns for two runs
  • Per-item comparison with visual diff markers
  • Automatic column width adjustment based on terminal size
• CLI Progress Dashboard - Unified progress view for all concurrent operations
  • Multi-bar display: per-metric progress within a run
  • ETA estimation based on completed items and average per-item time
  • Rich-based dashboard with live updates (optional dependency)
• CLI Command Chaining - Pipe output of one command as input to another
  • evalyn build-dataset | evalyn run-eval passing dataset path automatically
  • --stdin flag reading dataset path or run ID from standard input
  • Useful for scripting multi-step workflows without temp variables
• CLI Time Tracking - Track total time spent per command type for operational analytics
  • Auto-log command name and duration to .evalyn/timing.jsonl
  • evalyn timing-stats showing per-command average/total time
  • Identify slowest commands for optimization opportunities
• CLI Quick Rerun - Rerun last command with modified flags
  • evalyn !! repeating last command exactly
  • evalyn !! --workers 8 repeating with flag override
  • Command history stored in .evalyn/history.jsonl
• CLI Color Theme Configuration - User-configurable terminal color scheme
  • theme setting in evalyn.yaml: default, solarized, monokai, high-contrast
  • EVALYN_THEME env var for quick switching
  • Separate from NO_COLOR which disables all colors entirely
• CLI Output Width Control - Respect terminal width for table and chart formatting
  • Auto-detect terminal width and adjust table column widths accordingly
  • --width N flag to override detected width (useful for piping to files)
  • Truncate long cell values to fit within available space
• CLI Execution Audit Log - Log every CLI command with full arguments for reproducibility
  • Auto-append to .evalyn/command_log.jsonl: timestamp, command, args, exit code, duration
  • evalyn audit-log showing chronological command history
  • Distinct from evaluation audit trail (covers all commands, not just eval runs)

Run Management

• Run Naming - Give eval runs human-readable names instead of only UUIDs
  • --name flag on run-eval: evalyn run-eval --name "prompt-v3-experiment"
  • Name stored in EvalRun metadata, displayed in list-runs
  • Resolve runs by name: evalyn show-run --name "prompt-v3-experiment"
• Run Pinning - Mark a run as baseline for automatic comparison
  • evalyn pin-run --id <id> marking a run as the project baseline
  • Subsequent analyze and compare commands auto-compare against pinned run
  • evalyn list-runs showing pinned run with a marker
• Run Cleanup - Bulk delete runs matching criteria
  • evalyn cleanup-runs --older-than 30d --keep-pinned
  • evalyn cleanup-runs --below-pass-rate 0.3 for removing low-quality runs
  • --dry-run mode showing what would be deleted with total storage savings

Metrics Enhancements

• Custom Metric DSL - Define metrics via YAML config without writing Python code
  • YAML metric definition: name, type, prompt template, threshold, scoring rubric
  • Variable interpolation: {{input}}, {{output}}, {{expected}} in prompt templates
  • Custom objective metrics via Python expressions (e.g. "len(output) < 500")
  • Hot-reload: modify YAML, re-run eval without code changes
• Metric Composition - Combine multiple metrics into weighted composite scores
  • Composite metric definition: weighted average of child metrics
  • Min/max/mean aggregation strategies
  • Pass threshold on composite score
  • Drill-down: see child metric contributions to composite
• Metric Weighting Profiles - Named weight sets for different evaluation use cases
  • Profile definitions in evalyn.yaml (e.g. "safety-first": safety=3x, quality=1x)
  • --weight-profile flag on analyze and compare commands
  • Weighted pass rate and weighted overall score
• Metric Versioning - Track when metric implementations change and flag affected runs
  • Hash metric prompt + scoring logic as version identifier
  • Store metric version in MetricResult metadata
  • Warn when comparing runs with different metric versions
  • evalyn metric-history showing version changes over time
• Metric Benchmarking - Measure computation cost and latency per metric
  • Per-metric timing in evaluation runner
  • Token usage and cost per metric type
  • Benchmark report: slowest metrics, most expensive metrics
  • Optimization suggestions for costly metrics
• Inter-Rater Reliability - Compute agreement stats when multiple judges score the same items
  • Run same metric with N different judges (models or prompts)
  • Fleiss' Kappa for multi-rater agreement
  • Identify items with lowest agreement for human review
  • Recommend judge selection based on reliability
• Metric Sensitivity Analysis - Measure score stability across small input perturbations
  • Perturb inputs (typos, rephrasing) and measure score variance
  • Flag metrics with high sensitivity to minor input changes
  • Robustness score per metric
• Metric Correlation Pruning - Auto-suggest removing redundant metrics that track the same signal
  • Pearson/Spearman correlation matrix across all metrics
  • Flag pairs with r > 0.95 as candidates for pruning
  • Recommend minimal metric set preserving signal coverage
• Metric Dependencies - Declare that metric B requires metric A to run first (dependency graph)
  • Dependency declaration in MetricSpec
  • Topological sort of metrics before evaluation
  • Pass metric A results as context to metric B prompt
• Conditional Metric Chains - If metric A fails, automatically run a diagnostic follow-up metric B
  • Chain definition: "if toxicity_safety fails, run toxicity_type_classifier"
  • Diagnostic metrics produce detailed failure categorization
  • Chain results stored alongside primary metric results
• Metric Namespacing - Organize metrics by project/team namespace to avoid collisions
  • Namespace prefix: "team-safety/toxicity" vs "team-quality/toxicity"
  • Namespace-scoped metric search in list-metrics
  • Cross-namespace metric comparison
• Metric Score Explanations - Return human-readable explanations for objective metric scores
  • Per-metric explain() function describing why the score is what it is
  • Example: "json_valid: FAIL - parse error at line 3, column 12: unexpected token"
  • Include explanations in show-run and failed item breakdown output
• Metric Warmup Averaging - Run each subjective metric N times and average to reduce LLM variance
  • --metric-samples N flag on run-eval (default 1)
  • Report per-metric score variance across samples
  • Flag items where samples disagree (high variance) for review
• Metric Runtime Estimation - Predict eval duration per metric based on historical timing data
  • Store per-metric median execution time from past runs
  • Estimate total run time before execution starts
  • Surface slow metrics in dry-run output with time contribution
• Metric Compatibility Matrix - Show which metrics work with which evaluation unit types
  • Matrix: metrics on Y-axis, unit types (outcome, single_turn, tool_use, multi_turn) on X-axis
  • evalyn list-metrics --compatibility showing supported unit types per metric
  • Warn when user selects metrics incompatible with their trace structure
• Metric Score Binning - Configurable score-to-grade mapping for human-friendly reporting
  • Grade definitions in evalyn.yaml (e.g. A=0.8-1.0, B=0.6-0.8, C=0.4-0.6, F=0-0.4)
  • Grade distribution chart in analyze output
  • Custom grade labels and thresholds per project
• Reference-Adaptive Metrics - Auto-switch metric rubric based on whether expected reference is present
  • Detect reference availability per item via _dataset_has_reference
  • Use reference-based rubric when available, reference-free rubric otherwise
  • Report which rubric variant was used per item in MetricResult details
• Metric Debug Mode - Verbose logging of the complete judge interaction per item
  • --debug-metrics flag showing: prompt sent, raw response, parsed result per item
  • Log to .evalyn/metric_debug.jsonl for post-hoc analysis
  • Useful for diagnosing why a metric scores differently than expected
• Metric Template Variables - Custom variables in judge prompt templates beyond standard input/output/expected
  • User-defined variables in evalyn.yaml: template_vars: {domain: "healthcare", persona: "clinician"}
  • Variable interpolation in judge prompts: "Evaluate from the perspective of a {{persona}}"
  • Per-dataset variable overrides in meta.json
• Metric Registry Freeze - Lock the metric set for a project to prevent accidental changes
  • evalyn freeze-metrics --project <name> locking current metrics.json
  • Warn when attempting to modify frozen metric set
  • evalyn unfreeze-metrics to unlock for intentional changes
• Metric Output Post-Processing - Pluggable post-processors on raw judge output before scoring
  • Post-processor chain in evalyn.yaml per metric (e.g. normalize, clamp, round)
  • Built-in processors: score_clamp(0,1), binary_threshold(0.5), invert_score
  • Custom Python post-processor functions via entry points
• Metric Deprecation Lifecycle - Formal deprecation with migration path and sunset date
  • Deprecation metadata on MetricSpec: deprecated_since, replacement, sunset_date
  • Warning when using deprecated metrics in run-eval
  • evalyn list-metrics --deprecated showing deprecated metrics with migration hints
• Metric Category Pass Rates - Aggregate reporting by subjective category (safety, correctness, style, etc.)
  • Group metrics by CATEGORIES mapping in analyze output
  • Per-category pass rate bar charts
  • Identify weakest category for targeted improvement
• Metric Rubric Preview - Show exact judge prompt before evaluation starts
  • evalyn preview-metric --id helpfulness_accuracy showing full prompt with rubric
  • Include template variable substitution with sample input/output
  • Verify rubric looks correct before committing to expensive evaluation
• Metric Cross-Reference View - Show which bundles include each metric
  • evalyn list-metrics --show-bundles displaying bundle membership per metric
  • Inverse view: evalyn list-bundles --show-metrics for bundle contents
  • Useful for understanding metric coverage across different evaluation profiles
• Metric Score Curve Fitting - Fit parametric distributions to historical metric scores
  • Fit beta/normal/bimodal distributions to score history per metric
  • Detect distribution changes between runs (shift, spread, shape)
  • Use fitted distribution for anomaly detection on new scores
• Metric Prompt Token Count - Show estimated prompt token count per metric before evaluation
  • Estimate tokens from metric prompt template + average input/output sizes
  • evalyn list-metrics --show-tokens displaying per-metric token cost
  • Factor into cost estimation in dry-run mode
• Metric A/B Variant Testing - Evaluate same items with two rubric variants of the same metric
  • Define variant rubrics in evalyn.yaml: helpfulness_v1 vs helpfulness_v2
  • Run both variants in a single eval, compare scores and agreement
  • Select the variant with better alignment to human annotations
• Metric Cold Start Detection - Detect when a metric's first N items score differently than the rest
  • Compare score distribution of first K items vs remaining items per metric
  • Statistical test (KS or Mann-Whitney) for distribution shift
  • Recommend warm-up if cold start effect is significant

Metric Bundle Customization

• User-Defined Bundles - Define custom metric bundles in evalyn.yaml
  • bundles: section in evalyn.yaml with named metric lists
  • evalyn suggest-metrics --mode bundle --bundle my-custom-bundle
  • Inherit from built-in bundles and override (e.g. extend "chatbot" with custom metrics)
• Bundle Composition - Combine multiple bundles into one with deduplication
  • evalyn suggest-metrics --bundle chatbot+safety merging two bundles
  • Automatic deduplication when combining overlapping bundles
  • Conflict resolution when same metric appears with different configs
• Bundle Recommendation - Auto-suggest bundle based on captured trace patterns
  • Analyze trace spans to detect agent type (RAG, orchestrator, chatbot, etc.)
  • Match detected patterns to best-fit built-in bundle
  • evalyn suggest-metrics --mode auto-bundle choosing bundle without user input

LLM Provider Support

• Gemini - Full support with auto-instrumentation
• OpenAI - Full support with auto-instrumentation
• Anthropic - Full support with auto-instrumentation
• xAI (Grok) - Full support with auto-instrumentation
• Ollama - Local model support (--provider ollama)

Framework Support

• LangChain - Automatic instrumentation
• LangGraph - Automatic instrumentation with node tracking
• Google ADK - Automatic instrumentation
• Claude Agent SDK - Automatic instrumentation

Storage & Data

• SQLite storage - Local-first, no cloud dependencies
• Prod/test separation - Separate databases for environments
• JSONL datasets - Human-readable, git-friendly format
• Checkpoint system - Resume interrupted evaluations

Testing & Quality

• Test coverage improvement - 1,063 tests across 30 test files
  • Analysis engine: trends, reports, core properties, insights
  • Model roundtrips: Span, FunctionCall, DatasetItem, Annotation, SpanMetricLink
  • SQLiteStorage: CRUD, ID resolution, annotations
  • CLI utilities: formatters, validation, config
  • CLI commands: analyze, compare, trend, list-runs, show-run, insights
  • Export formats: markdown, HTML, CSV builders
  • Metrics: HeuristicSuggester, subjective template validation, objective metrics
  • Tracing: instrumentation, streaming, provider instrumentors
• Realistic test fixtures - 10+ items, 3 metrics, mixed scores, failure reasons
• pytest-cov integration - Coverage reporting via --cov=evalyn_sdk
• Integration test unskip - Fixed 2 skipped integration tests

Last updated: 2026-03-25