/godmode:eval

Evaluate, benchmark, and regression-test AI/LLM systems. Covers evaluation framework design, benchmark creation, human evaluation protocols, automated evaluation (LLM-as-judge), regression testing, statistical significance, and continuous evaluation pipelines.

Usage

/godmode:eval                              # Full evaluation workflow
/godmode:eval --dataset                    # Create or manage evaluation dataset
/godmode:eval --judge                      # Set up LLM-as-judge evaluation
/godmode:eval --human                      # Set up human evaluation protocol
/godmode:eval --benchmark                  # Create reusable benchmark
/godmode:eval --regression                 # Set up or run regression tests
/godmode:eval --compare <a> <b>            # Compare two systems or versions
/godmode:eval --report                     # Generate report from existing results
/godmode:eval --ci                         # Generate CI/CD integration
/godmode:eval --calibrate                  # Calibrate LLM judge vs human scores
/godmode:eval --significance               # Run statistical significance analysis
/godmode:eval --history                    # Show evaluation trends
/godmode:eval --quick                      # Lightweight eval (golden set only)
/godmode:eval --full                       # Comprehensive eval (all categories)

What It Does

1. Discovers what AI system needs evaluation, quality dimensions, and evaluation budget
2. Designs evaluation dataset from golden sets, production logs, synthetic, adversarial, and regression examples
3. Selects evaluation framework (RAGAS, DeepEval, Promptfoo, LangSmith, Braintrust, custom)
4. Designs LLM-as-judge with scoring rubrics, calibration, and bias mitigation
5. Designs human evaluation protocol with annotator training, agreement measurement, and adjudication
6. Creates reusable benchmarks with categories, weighting, and baseline scores
7. Builds regression test suite from production failures that runs in CI/CD
8. Runs statistical significance tests (bootstrap, McNemar, Wilcoxon) with confidence intervals
9. Generates evaluation config, dataset, judge prompts, regression tests, and report

Output

• Evaluation config at evals/<system>/eval-config.yaml
• Dataset at evals/<system>/dataset/
• Judge prompts at evals/<system>/judges/
• Regression tests at evals/<system>/regression/
• Results at evals/<system>/results/
• Report at docs/evals/<system>-eval-report.md
• Commit: "eval: <system> — v<version>, <N> examples, correctness=<val>, faithfulness=<val>"

Next Step

After evaluation: /godmode:prompt to fix prompt issues, /godmode:rag to improve retrieval, /godmode:agent to fix agent issues, or /godmode:ship to deploy if metrics pass.

Examples

/godmode:eval Evaluate our customer support RAG chatbot
/godmode:eval --judge Compare GPT-4 vs Claude on summarization
/godmode:eval --regression Set up regression tests for our AI pipeline
/godmode:eval --benchmark Create a benchmark for our classification task
/godmode:eval --compare v2.0 v2.1 Is the new version better?
/godmode:eval --calibrate Calibrate our LLM judge against human ratings