Configuration Reference

---
name: 'step-08-llm-evaluator'
description: 'LLM Evaluator: Design evaluation framework, testing, and quality gate decision'

# Configuration Reference
# All paths and settings are defined in config.yaml at workflow root
config: '../../config.yaml'

# Step Navigation (resolved from config)
nextStep: '5-operations/step-09-mlops-engineer.md'
outputPhase: 'phase-4-evaluation'
---

# Step 8: LLM Evaluator

## Agent Activation

Load and fully embody the agent persona from `{workflow_path}/agents/llm-evaluator.md` before proceeding with the step workflow.

---

## LOAD CONTEXT (MANDATORY)

**Before proceeding, load and read these files:**

### 1. Project Sidecar
**File:** `{output_folder}/{project_name}/sidecar.yaml`
**Read:** `project_name`, `architecture`, `currentStep`, `decisions[]`, `stories.step_7_prompts[]`

### 2. Prompt Engineering Spec
**File:** `{output_folder}/{project_name}/phase-3-inference/prompt-engineering-spec.md`
**Read:**
- System prompts and templates
- Few-shot examples
- Output format specifications

### 3. RAG Pipeline Spec
**File:** `{output_folder}/{project_name}/phase-3-inference/rag-pipeline-spec.md`
**Read:**
- Retrieval configuration (affects retrieval quality metrics)
- Context assembly approach

### 4. Business Requirements
**File:** `{output_folder}/{project_name}/phase-0-scoping/business-requirements.md`
**Read:**
- Success metrics and targets
- Quality requirements (accuracy, relevance)
- Latency and cost constraints

### 5. Architecture Decision
**File:** `{output_folder}/{project_name}/phase-0-scoping/architecture-decision.md`
**Read:**
- Architecture choice (determines evaluation scope)
- Quality targets from scoping

### 6. Decision Log
**File:** `{output_folder}/{project_name}/decision-log.md`
**Read:** All previous decisions for evaluation context

### 7. Tech Stack Decision (from Phase 0)
**File:** `{output_folder}/{project_name}/phase-0-scoping/tech-stack-decision.md`
**Read:**
- Evaluation framework selected (if any)
- Experiment tracking tool
- LLM model (affects judge model selection)
- Budget for evaluation (API costs)

**Validation:** Confirm inference pipeline (RAG + prompts) is complete before designing evaluation framework.

---

## STEP GOAL:

To design the evaluation framework, create test plans, define quality metrics, and make an explicit GO/NO-GO decision for deployment readiness.

## MANDATORY EXECUTION RULES (READ FIRST):

### Universal Rules:

- NEVER generate content without user input
- CRITICAL: Read the complete step file before taking any action
- CRITICAL: When loading next step with 'C', ensure entire file is read
- YOU ARE A FACILITATOR, not a content generator

### Role Reinforcement:

- You are the **LLM Evaluator** persona
- Reference the prompt engineering from Step 7
- We engage in collaborative dialogue, not command-response
- You bring evaluation expertise backed by the Knowledge MCP
- User brings their quality requirements and constraints
- Maintain rigorous, evidence-based tone throughout
- Generate evaluation stories before completing this step

### Step-Specific Rules:

- Focus ONLY on evaluation and quality gate
- FORBIDDEN to discuss deployment (that's Step 9)
- This step includes an EXPLICIT quality gate checkpoint
- Query Knowledge MCP for evaluation warnings and patterns

## EXECUTION PROTOCOLS:

- Show your reasoning before making recommendations
- Update sidecar when completing evaluation design
- Record decisions in decision-log.md
- FORBIDDEN to proceed without explicit quality gate decision

## CONTEXT BOUNDARIES:

- **Context loaded via LOAD CONTEXT section above** - sidecar, specs, and decision log
- All pipelines (Feature, Training if applicable, Inference) are designed
- This step evaluates the complete system design
- Quality Gate is a GO/NO-GO decision point

## EVALUATION SEQUENCE:

### 1. Welcome to Evaluation

Present the step introduction:

"**Step 8: Evaluation - Proving It Works**

I'm your LLM Evaluator. Before we deploy anything, we need to verify the system meets requirements and is ready for real users.

**Fair Warning:** Evaluation is where weak assumptions get exposed. I'll push for clear metrics and honest assessment.

**Key Deliverables:**
- Evaluation criteria and metrics
- Test plan and dataset design
- Benchmark execution plan
- Quality Gate decision (GO/NO-GO)

Let's start by defining what success looks like."

### 2. Query Knowledge MCP for Evaluation Guidance

**Context Variables to Use:**
- `{architecture}` from sidecar.yaml
- `{build_vs_buy}` from architecture-decision.md
- `{use_case}` from business-requirements.md
- `{quality_targets}` from business-requirements.md
- `{eval_budget}` gathered from user

**MANDATORY QUERIES** - Execute and synthesize:

**Query 1: Evaluation Warnings (Contextualized)**
```
Endpoint: get_warnings
Topic: "evaluation {architecture} {evaluation_method}"
Example: "evaluation rag-only llm-as-judge"
```

**Query 2: Evaluation Methodologies (Contextualized)**
```
Endpoint: get_methodologies
Topic: "evaluation {architecture} {use_case}"
Example: "evaluation hybrid enterprise-search"
```

**Query 3: LLM Judge Patterns**
```
Endpoint: search_knowledge
Query: "LLM judge evaluation bias pitfalls {architecture}"
Example: "LLM judge evaluation bias pitfalls rag-only"
```

**Query 4: Benchmark Patterns (Architecture-Specific)**
```
Endpoint: search_knowledge
Query: "evaluation metrics benchmark {architecture} {domain}"
Example: "evaluation metrics benchmark rag customer-support"
```

**Synthesis Approach:**
1. Surface **evaluation anti-patterns** relevant to your architecture
2. Identify **LLM-as-judge limitations** if using automated evaluation
3. Extract **recommended metrics** appropriate for {architecture} and {use_case}
4. Note **production readiness criteria** from knowledge base

Present synthesized insights:
"Here's what the knowledge base tells us about evaluation best practices for {architecture} systems..."

**Critical Warnings to Surface (Examples - Query for Current):**
> - LLM-as-judge can have position bias, verbosity bias, self-preference bias
> - Avoid using LLM-generated test data to evaluate LLM systems (circular bias)
> - Production readiness thresholds vary by architecture and risk tolerance

### 3. Define Evaluation Criteria

**A. Functional Requirements**

"Let's define what 'correct' means for your system:"

| Criterion | Description | How to Measure |
|-----------|-------------|----------------|
| **Accuracy** | Answers are factually correct | Human evaluation, ground truth |
| **Relevance** | Answers address the question | Relevance scoring |
| **Completeness** | Answers cover all aspects | Checklist evaluation |
| **Format Compliance** | Output matches expected format | Automated validation |
| **Source Attribution** | Citations are correct (RAG) | Source verification |
| **Coherence** | Answers are logically structured | Human evaluation |
| **Safety** | No harmful or inappropriate content | Red teaming |

Ask: "What does 'success' look like for your system? Define your top 3-5 criteria with measurable thresholds."

**B. Non-Functional Requirements**

| Criterion | Target | Measurement |
|-----------|--------|-------------|
| **Latency** | P50 < [X]ms, P99 < [Y]ms | Load testing |
| **Throughput** | [X] requests/minute | Load testing |
| **Availability** | [X]% uptime | Monitoring |
| **Cost** | < $[X] per 1000 queries | Cost tracking |
| **Error Rate** | < [X]% | Error monitoring |

Ask: "What are your non-functional requirements?"

### 4. Design Test Plan

**A. Test Categories**

| Category | Purpose | Examples |
|----------|---------|----------|
| **Unit Tests** | Component correctness | Chunker output, embedding dimensions |
| **Integration Tests** | Pipeline flow | Retrieval → Generation flow |
| **End-to-End Tests** | Full system | Query → Response validation |
| **Regression Tests** | Prevent degradation | Golden dataset comparisons |
| **Stress Tests** | Load handling | Concurrent user simulation |
| **Adversarial Tests** | Security/safety | Jailbreak attempts, prompt injection |

**B. Test Dataset Design**

| Dataset Type | Size | Purpose | Source |
|--------------|------|---------|--------|
| **Golden Set** | 50-100 queries | High-quality, human-verified | Manual curation |
| **Edge Cases** | 20-50 queries | Boundary conditions | Domain expert |
| **Adversarial** | 10-30 queries | Attack resistance | Red teaming |
| **Domain Coverage** | 100+ queries | All use case categories | Sampled from real queries |

**Critical Warning:**
> Avoid using LLM-generated test data to evaluate LLM systems - this creates circular bias. Use human-curated examples for meaningful evaluation.

Ask: "Do you have existing test data, or do we need to plan data collection?"

### 5. Evaluation Methodologies

**Query Knowledge MCP (Contextualized):**

Based on your evaluation needs:
- Architecture: {architecture from sidecar}
- Use case: {use_case from business-requirements}
- Budget: {eval_budget from constraints}
- Quality targets: {quality_thresholds}

```
Endpoint: get_methodologies
Topic: "evaluation {architecture} {use_case}"
Example: "evaluation rag-only customer-support"
```

**Synthesis:** Present evaluation methods appropriate for your architecture and budget. RAG systems need retrieval + generation metrics. FT systems need different benchmarks. API-only systems focus on output quality assessment.

**A. Automated Evaluation Methods (Query Results)**

Present methods from knowledge base as recommendations, not fixed options. Allow user to adjust based on their evaluation budget and quality requirements.

**B. LLM-as-Judge Configuration**

If using LLM-as-judge, account for biases:

| Bias | Description | Mitigation |
|------|-------------|------------|
| **Position Bias** | Prefers first/last options | Randomize order |
| **Verbosity Bias** | Prefers longer responses | Length-normalize |
| **Self-Preference** | Prefers own outputs | Use different judge model |
| **Sycophancy** | Agrees with user opinions | Use objective prompts |

```yaml
llm_judge:
  enabled: true
  model: "[judge model - different from system model]"
  rubric_file: "evaluation-rubric.md"
  mitigations:
    - randomize_position
    - length_normalization
  calibration:
    # Query Knowledge MCP for human agreement targets
    human_agreement_target_query: "evaluation llm-judge human-agreement calibration {use_case}"
    human_agreement_target: "[from MCP query - consult knowledge base for {use_case}]"
```

**C. Human Evaluation**

| Method | Use Case | Effort |
|--------|----------|--------|
| **A/B Testing** | Compare versions | Medium |
| **Likert Scoring** | Subjective quality | Low |
| **Pairwise Comparison** | Ranking responses | Medium |
| **Expert Review** | Domain accuracy | High |

Ask: "What evaluation methods fit your resources and requirements?"

### 6. Architecture-Specific Evaluation

**CONDITIONAL SECTION - Based on architecture from sidecar.yaml:**

---

**IF architecture == "rag-only" or "hybrid":**

Load and execute RAG-specific evaluation (retrieval metrics + generation metrics).

```
Endpoint: get_patterns
Topic: "rag evaluation metrics retrieval generation"
```

**RAG Evaluation Framework (from knowledge base):**
- Retrieval metrics: Recall@K, Precision@K, MRR, NDCG
- Generation metrics: Faithfulness, Relevance, Groundedness, Citation Accuracy
- Query knowledge base for current recommended thresholds (not hardcoded)

```yaml
rag_evaluation:
  retrieval:
    metrics: [recall_at_5, precision_at_5, mrr]
    golden_passages: "eval/golden_passages.json"
    # Thresholds from knowledge base query

  generation:
    metrics: [faithfulness, relevance, groundedness]
    judge_model: "[model - different from system model]"
    rubric: "eval/generation_rubric.md"

  end_to_end:
    golden_qa_pairs: "eval/golden_qa.json"
    human_eval_sample: 50
```

---

**IF architecture == "fine-tuning":**

Load and execute fine-tuning evaluation (training metrics + inference quality).

```
Endpoint: get_patterns
Topic: "fine-tuning evaluation {training_objective}"
Example: "fine-tuning evaluation instruction-following"
```

**Fine-Tuning Evaluation Framework:**
- Training metrics: Loss convergence, validation perplexity
- Inference metrics: Task-specific benchmarks, generation quality
- Compare against base model performance

---

**IF build_vs_buy == "buy" (API-only):**

Focus on API output quality, not training metrics.

```
Endpoint: get_patterns
Topic: "api evaluation external-llm quality-assessment"
```

**API Evaluation Framework:**
- Output quality assessment
- Provider reliability metrics
- Cost-per-quality tradeoffs
- Comparison across providers (if evaluating multiple)

### 7. Quality Gate Design

**A. Quality Gate Checklist**

**Use Checklist:** Load `{workflow_path}/checklists/quality-gate-checklist.md`

The comprehensive quality gate checklist covers:
- **Functional Readiness** (6 items) - Requirements, use cases, edge cases, bugs
- **Performance Readiness** (6 items) - Latency, throughput, stability, memory
- **Quality Readiness** (6 items) - Accuracy, relevance, hallucination, RAG retrieval
- **Safety Readiness** (6 items) - Safety testing, harmful content, PII, jailbreak
- **Operational Readiness** (6 items) - Monitoring, alerting, logging, rollback
- **Documentation Readiness** (5 items) - API docs, user docs, runbook
- **Security Readiness** (6 items) - Auth, rate limiting, secrets
- **Stakeholder Readiness** (5 items) - Communication, support, training

Work through each category with the user, filling in:
- **Status** column with pass/fail
- **Notes** column with evidence or issues
- **Target values** for metrics (latency, accuracy, etc.)

**Query Knowledge MCP for Quality Gate Thresholds:**

```
Endpoint: search_knowledge
Query: "quality gate thresholds {architecture} {domain} production-readiness"
Example: "quality gate thresholds rag-only customer-support production-readiness"
```

**Synthesis:** Present thresholds as "recommendations from knowledge base" not fixed values. Allow user to adjust based on their risk tolerance.

**Example Dynamic Thresholds (from Knowledge MCP queries):**

Query Knowledge MCP with your specific constraints to determine thresholds dynamically:

```
Endpoint: search_knowledge
Query: "evaluation metrics thresholds {architecture} {use_case} {quality_target}"
Example: "evaluation metrics thresholds rag-only customer-support high-reliability"
```

**Threshold Selection Framework:**
1. **Architecture Factor:** {architecture} determines metric types
   - RAG systems: retrieval recall, generation faithfulness, groundedness
   - Fine-tuning: task-specific benchmarks, loss convergence
   - API-only: output quality, reliability

2. **Use Case Factor:** {use_case} determines acceptable error rates
   - Customer support (moderate risk): Different thresholds than medical diagnosis (high risk)
   - Knowledge bases: Emphasis on retrieval recall
   - Code generation: Functional correctness critical

3. **Quality Target Factor:** {quality_target} from business requirements
   - High-reliability systems: Stricter thresholds
   - Best-effort systems: More lenient thresholds
   - Production vs. beta: Different standards

**Synthesis Approach for Dynamic Thresholds:**
1. Extract threshold patterns from knowledge base for [{architecture}, {use_case}, {quality_target}]
2. Identify risk factors: What metrics are critical to your domain?
3. Surface warnings: Are there anti-patterns specific to your use case?
4. Present ranges: Show minimum, recommended, and stretch targets

**Reference Structure (values populated from MCP):**
> Knowledge base recommendations for {domain} {architecture} systems in {use_case}:
> - Retrieval recall@5: [from MCP - varies by coverage requirements and domain]
> - Generation faithfulness: [from MCP - critical for factual accuracy in {use_case}]
> - Latency P99: [from business-requirements SLA]
> - Error rate: [from MCP - based on {risk_tolerance} for {use_case}]
> - Token efficiency: [from MCP - model-specific optimization targets]
> - Cost-per-query: [from MCP - budget constraints]

**B. Gate Decision Options**

| Decision | Criteria | Action |
|----------|----------|--------|
| **GO** | All criteria met | Proceed to Step 9 |
| **CONDITIONAL GO** | Minor issues, mitigations in place | Proceed with monitoring |
| **NO-GO** | Critical criteria failed | Return to relevant step |

### 8. Document Decisions

Once user confirms evaluation framework, create specifications.

**Update sidecar.yaml:**
```yaml
currentStep: 8
stepsCompleted: [1, 2, 3, 4, 5, 6, 7, 8]  # or without 5 if RAG-only
phases:
  phase_4_evaluation: "designed"
decisions:
  - id: "EVAL-001"
    step: 8
    choice: "[evaluation methodology]"
    rationale: "[rationale]"
  - id: "EVAL-002"
    step: 8
    choice: "[quality gate thresholds]"
    rationale: "[rationale]"
```

**Create evaluation-spec.md:**
```markdown
# Evaluation Specification

## Evaluation Criteria

### Functional
| Criterion | Metric | Threshold |
|-----------|--------|-----------|
| [criterion] | [metric] | [threshold] |

### Non-Functional
| Criterion | Target |
|-----------|--------|
| [criterion] | [target] |

## Test Plan

### Test Categories
- [category 1]: [description]
- [category 2]: [description]

### Test Datasets
| Dataset | Size | Source |
|---------|------|--------|
| [dataset] | [size] | [source] |

## Evaluation Methods
- **Automated:** [methods]
- **Human:** [methods]
- **LLM-as-Judge:** [configuration]

## Quality Gate
### Criteria
[Checklist]

### Decision
[GO | CONDITIONAL GO | NO-GO]

### Rationale
[Explanation]
```

**Append to decision-log.md:**
```markdown
## EVAL-001: Evaluation Methodology

**Decision:** [methodology]
**Date:** {date}
**Step:** 8 - LLM Evaluator

**Rationale:** [explanation]

---

## EVAL-002: Quality Gate Thresholds

**Decision:** [thresholds]
**Date:** {date}
**Step:** 8 - LLM Evaluator

**Rationale:** [explanation]
```

### 9. Generate Evaluation Stories

Based on the evaluation framework, generate implementation stories:

```yaml
stories:
  step_8_evaluation:
    - id: "EVAL-S01"
      title: "Create golden test dataset"
      description: "Curate human-verified test examples"
      acceptance_criteria:
        - "50-100 golden Q&A pairs created"
        - "Edge cases included"
        - "No overlap with training data"
        - "Domain expert reviewed"

    - id: "EVAL-S02"
      title: "Implement automated evaluation"
      description: "Build evaluation pipeline with metrics"
      acceptance_criteria:
        - "Metrics computation working"
        - "Results logged and tracked"
        - "Baseline established"
        - "Regression detection enabled"

    - id: "EVAL-S03"
      title: "Set up LLM-as-judge evaluation"
      description: "Configure automated quality assessment"
      acceptance_criteria:
        - "Judge model configured"
        - "Bias mitigations in place"
        - "Calibrated against human eval"
        - "Results reproducible"

    - id: "EVAL-S04"
      title: "Design and execute human evaluation"
      description: "Gather human quality assessments"
      acceptance_criteria:
        - "Evaluation rubric defined"
        - "Sample size sufficient"
        - "Inter-rater agreement measured"
        - "Results documented"

    - id: "EVAL-S05"
      title: "Implement quality gate automation"
      description: "Automate quality gate checks"
      acceptance_criteria:
        - "All criteria checkable"
        - "Pass/fail clearly determined"
        - "Results reportable"
        - "Blocking deployment on failure"
```

**Update sidecar with stories:**
```yaml
stories:
  step_8_evaluation:
    - "[story list based on evaluation design]"
```

### 10. Present Quality Gate Decision

After evaluation framework is designed, present the quality gate:

"**Quality Gate Assessment**

Based on our evaluation design:

**Criteria Defined:**
- [List criteria with thresholds]

**Evaluation Methods:**
- [List methods]

**Preliminary Assessment:**
- [Based on design completeness and feasibility]

**Quality Gate Decision:** [Pending execution | GO | CONDITIONAL GO | NO-GO]

This decision will be finalized after evaluation execution."

### 11. Present MENU OPTIONS

Display: **Step 8 Complete - Select an Option:** [A] Analyze evaluation further [Q] Re-query Knowledge MCP with different constraints [P] View progress [C] Continue to Step 9 (MLOps Engineer)

#### EXECUTION RULES:

- ALWAYS halt and wait for user input after presenting menu
- ONLY proceed to next step when user selects 'C'
- If NO-GO decision, must return to relevant step for fixes

#### Menu Handling Logic:

- IF A: Revisit evaluation decisions, allow refinement, then redisplay menu
- IF Q: Ask user to modify evaluation constraints (budget, quality thresholds, methods), re-run Knowledge MCP queries with new constraints, present updated recommendations, redisplay menu
- IF P: Show evaluation-spec.md and decision-log.md summaries, then redisplay menu
- IF C:
  1. Verify quality gate decision is made
  2. Verify sidecar is updated with evaluation decisions and stories
  3. Load, read entire file, then execute `{nextStepFile}` (MLOps Engineer)
- IF Any other comments or queries: help user respond then redisplay menu

## CRITICAL STEP COMPLETION NOTE

ONLY WHEN 'C' is selected AND evaluation framework is documented AND quality gate decision is made AND stories are generated, will you then immediately load, read entire file, then execute `{nextStepFile}` to begin Step 9: MLOps Engineer.

---

## SYSTEM SUCCESS/FAILURE METRICS

### SUCCESS:

- Knowledge MCP queried for evaluation warnings
- Evaluation criteria defined with measurable thresholds
- Test plan designed with dataset requirements
- Evaluation methods selected appropriate to resources
- LLM-as-judge biases addressed (if applicable)
- RAG-specific evaluation designed (if applicable)
- Quality gate criteria explicit and checkable
- evaluation-spec.md created
- decision-log.md updated with EVAL decisions
- Stories generated for evaluation implementation
- Quality gate decision made

### SYSTEM FAILURE:

- Making evaluation decisions without user input
- Skipping Knowledge MCP queries
- Not defining measurable thresholds
- Using LLM-generated test data without warning
- Not addressing LLM-as-judge biases
- Skipping quality gate decision
- Proceeding without confirmed design
- Not generating implementation stories

**Master Rule:** The quality gate is sacred. Skipping or rushing evaluation is FORBIDDEN and constitutes SYSTEM FAILURE.