---

document_id: AIDHA-TASK-004 owner: Ingestion Engineering Lead status: Draft version: "0.6" last_updated: 2026-03-03 title: Claim Extraction Evaluation Matrix type: TASK docops_version: "2.0"

<!-- markdownlint-disable MD013 MD031 --> <!-- MEMINIT_METADATA_BLOCK -->

Document ID: AIDHA-TASK-004 Owner: Ingestion Engineering Lead Approvers: — Status: Draft Version: 0.6 Last Updated: 2026-03-03 Type: TASK

Claim Extraction Evaluation Matrix

Version History

Version	Date	Author	Change Summary	Reviewers	Status	Reference
0.1	2026-03-02	AI-assisted	Initial atomic task breakdown	—	Draft	AIDHA-TASK-003-ATOMIC
0.2	2026-03-03	AI-assisted	Add independent manual baseline + rubric/fixture best practices	—	Draft	AIDHA-TASK-003-ATOMIC
0.3	2026-03-03	AI-assisted	Add calibration loop, type contracts, CI smoke gating, and governance notes	—	Draft	AIDHA-TASK-003-ATOMIC
0.4	2026-03-03	AI-assisted	Tighten type contracts and aggregator/report specs	—	Draft	AIDHA-TASK-003-ATOMIC
0.5	2026-03-03	AI-assisted	Add extraction retry semantics, logging, token/chunk ablations, and JSON-mode prerequisites	—	Draft	AIDHA-TASK-003-ATOMIC
0.6	2026-03-03	AI-assisted	Add tier filtering and extraction-vs-judge cost breakdown	—	Draft	AIDHA-TASK-003-ATOMIC

Overview

This task defines a systematic evaluation framework for YouTube claim extraction quality. The framework runs a matrix of 5–10 YouTube videos × ~10 LLM models, then uses LLM-as-judge scoring to evaluate each extraction run across four quality dimensions:

Dimension	Definition
Completeness	Does the extraction capture all substantive claims present in the transcript? (Recall proxy)
Accuracy	Are extracted claims faithful to the source material without hallucination or distortion? (Precision proxy)
Topic Coverage	Do the claims proportionally cover the video's topic distribution and timeline? (Representativeness proxy)
Atomicity	Are claims single, indivisible assertions without redundancy? (Structure proxy)

Goals

• Establish a repeatable benchmark to compare extraction quality across models (and model versions).
• Detect regressions in extraction quality (and/or editorial second-pass filtering) before merge.
• Produce actionable diagnostics (missing claims, hallucinations, redundancy patterns, gap areas) so fixes are targeted.

Non-Goals

• Proving claims are true in the real world (fact-checking against external sources is out of scope).
• Building a perfect recall/precision ground truth for all corpus videos (we calibrate against a small Golden Set; the rest is proxy scoring).

Definitions (Operational)

• Claim: A standalone, falsifiable assertion attributable to the video content (not to outside knowledge).
• Substantive claim: Non-trivial statements that change a reader’s beliefs/actions (exclude greetings, ads, housekeeping, “subscribe”, etc.).
• Atomic claim: One indivisible assertion. If it contains “and/or” with multiple assertions, it should be split.
• Harness: The in-repo extraction pipeline + prompts + post-processing (including any editorial second pass).
• Manual baseline (no harness): A direct prompt-response extraction using an external UI (Gemini web / ChatGPT UI) to sanity-check what content is extractable independent of our harness and model wrappers.

Anchored Scoring Guidance (0–10)

These anchors are for humans and the judge prompt so scores are comparable over time.

• Completeness
  • 0: Misses almost all substantive claims; only captures a few obvious points.
  • 5: Captures main claims but misses many secondary claims and/or entire segments.
  • 10: Captures essentially all substantive claims; misses (at most) minor details.
• Accuracy
  • 0: Many hallucinations, distortions, wrong attributions, or invented numbers.
  • 5: Mostly faithful but with several distortions/over-generalizations.
  • 10: Faithful paraphrases with no hallucinations; qualifies uncertainty appropriately.
• Topic Coverage
  • 0: Mostly one topic while ignoring large parts of the video, or heavily front-loaded.
  • 5: Covers major topics but under-represents some segments/topics.
  • 10: Proportional coverage across the full timeline and topic distribution.
• Atomicity
  • 0: Mostly multi-claim sentences, duplicates, and merged ideas.
  • 5: Mixed: many atomic claims, but frequent redundancy or multi-claim items.
  • 10: Nearly all claims are atomic, non-redundant, and consistently formatted.

Expected Outputs (Artifacts)

• Cell-level extraction output: per (videoId, modelId, extractorVariant) a structured claim set plus prompt/response trace.
• Cell-level scoring output: per (videoId, modelId, extractorVariant, judgeModelId) a validated ClaimSetScore plus trace.
• Reports: markdown for human review; JSON for programmatic trend/CI gating.
• Manual baseline snapshots: captured prompts + responses for a small subset, plus a short comparison write-up of systematic deltas vs harness (especially editorial second pass).

Design Principles (per engineering-principles.md)

• Separation of concerns: extraction, scoring, and reporting are distinct modules
• Determinism: all tests reproducible without live API calls (cached transcripts + recorded LLM responses)
• Validate at the boundary: LLM judge responses validated via zod schema before aggregation
• Fail explicitly: scoring failures produce structured error context, not silent zeros
• Test-first: each module begins with a failing test before implementation
• Internal schemas (pre-v1.0): Matrix report/output schemas are internal and may change freely until a v1.0 contract is declared.

Constraints & Risks (Call Out Early)

• Copyright / licensing: do not commit full copyrighted transcripts to the public repo. Prefer:
  • Synthetic transcripts for CI tests.
  • Short excerpts only when clearly defensible.
  • Local-only caches for full real transcripts (gitignored, outside tests/fixtures/).
• Context window: judges cannot always see a 2hr transcript; plan for chunked / sampled scoring where needed.
• Judge bias: LLM-as-judge can drift; calibrate on Golden Set and use multi-judge consensus + variance flags.
• Cost: matrix runs can be expensive; dry-run planning and caching are mandatory.
• Budget ceiling: default ceiling for a “full matrix” run is $25 estimated cost. Runs above this should require manual approval. (--dry-run must print estimated cost before execution.)
• Prompt injection (transcript content): transcripts may contain adversarial text. Judge prompts should wrap transcript in clear delimiters and instruct the judge to treat transcript text as data, not instructions.
• Rollback/recovery: define how to invalidate a bad run/cell cache without deleting unrelated cache entries.

Dependency

This task depends on AIDHA-TASK-003-ATOMIC delivering (minimum):

• A stable ClaimCandidate shape and extraction pipeline outputs we can score deterministically.
• Prompt/version metadata needed for cache keys (promptVersion, extractorVersion, and editor version/variant identifiers).
• Editorial second-pass implementations (AIDHA-PLAN-004) that can be toggled for ablation as extractorVariantId values.

Non-blocking: any optional heuristic enrichment work in AIDHA-TASK-003-ATOMIC that does not change the evaluation harness contract.

Acceptance Criteria

1. No existing tests - validation defined in Phase 4: Validation & CI Integration and references Phase 4 test tasks (4.1-4.6).
2. Document-level artifacts defined:
   • Cell-level extraction output
   • Cell-level scoring output / ClaimSetScore
   • Aggregated Reports

---

Phase 1: Test Corpus & Infrastructure

Task 1.1: Curate evaluation video corpus

• Task: Create [packages/praecis/youtube/tests/fixtures/eval-matrix/corpus.json] defining 5–10 YouTube video entries with fields: videoId, url, title, channelName, durationMinutes, topicDomain, expectedClaimDensity (low/medium/high), language, captionSource (manual/auto/unknown), speakerStyle (solo/interview/panel/unknown), rationale
• Rationale: A diverse corpus spanning different content types (lecture, interview, panel, solo explainer), durations (15min–2hr+), and domains (nutrition, neuroscience, exercise physiology) prevents overfitting evaluation to a single video style. The existing test video h_1zlead9ZU (Huberman × Aragon, ~2hr, nutrition) is necessary but insufficient alone.
• Selection Criteria: At least 2 videos per domain category; at least 1 video <30min and 1 video >90min; at least 1 multi-speaker panel; no duplicate channels; at least 1 video with notably noisy captions (to test robustness to transcript quality)
• Regression Guard: Corpus file validated by schema test; minimum 5 entries enforced
• Completion Criteria: Corpus JSON passes schema validation; each entry includes rationale for inclusion; corpus covers ≥3 distinct topic domains
• Spec Example:

  {
    "videoId": "h_1zlead9ZU",
    "url": "https://www.youtube.com/watch?v=h_1zlead9ZU",
    "title": "Dr. Andrew Huberman: The Science of Nutrition...",
    "channelName": "Huberman Lab",
    "durationMinutes": 124,
    "topicDomain": "Nutrition",
    "expectedClaimDensity": "high",
    "rationale": "High-density scientific assertions; multi-speaker debate."
  }
  

Task 1.1b: Validate corpus schema (zod + test)

• Task: Create [packages/praecis/youtube/src/eval/corpus-schema.ts] exporting a zod schema for CorpusEntry
• Task: Create [packages/praecis/youtube/tests/eval/corpus-schema.test.ts] validating corpus.json against the schema
• Rationale: The corpus file is a core input contract; schema validation prevents accidental drift and confusing runtime failures.
• Regression Guard: Schema test runs in CI; invalid/missing fields fail fast with clear output
• Completion Criteria: corpus.json validation runs in CI and fails on invalid entries

Task 1.2: Ingest and cache transcripts for corpus videos

• Task: Create script [scripts/eval-matrix/ingest-corpus.sh] that runs pnpm -C packages/praecis/youtube cli ingest video <url> for each corpus entry and writes transcript JSON to a local-only cache directory (gitignored), e.g. out/eval-matrix/transcripts/<videoId>.json
• Task: Create a small committed fixture set [packages/praecis/youtube/tests/fixtures/eval-matrix/transcript-excerpts/] containing:
  • synthetic transcripts for deterministic unit/integration tests, and/or
  • short excerpt transcripts (seconds/minutes, not hours) when clearly defensible
• Rationale: Determinism matters, but committing full YouTube transcripts is likely a licensing/copyright risk for a public repo. Separate local evaluation corpora from committed CI fixtures so engineering discipline does not force risky content into git.
• Regression Guard: Script is idempotent; skips already-cached transcripts; validates transcript non-empty; fails loudly when cache dir is missing/unwritable
• Completion Criteria: Corpus ingestion populates local cache; CI tests run using only committed excerpts/synthetic fixtures; repo .gitignore prevents accidental transcript commits

Task 1.3: Define model registry

• Task: Create [packages/praecis/youtube/src/eval/model-registry.ts] exporting EvalModel[] with fields: id, provider, baseUrl, modelName, contextWindow, supportsJsonMode, costPer1kTokens, notes, tier (frontier|midtier|budget), availability (stable|experimental|free-tier)
• Rationale: Centralised model metadata enables cost estimation, capability gating (e.g., JSON mode), and reporting. Avoids stringly-typed model references scattered across scripts.
• Model Candidates: GPT-5, GPT-5-mini, Claude Opus 4, Claude Sonnet 4, Gemini 2.5 Pro, Gemini 2.5 Flash, Llama 4 Maverick, Llama 4 Scout, DeepSeek-R1, Qwen 3 235B
• Regression Guard: Registry validated by unit test; each entry requires non-empty id and provider
• Execution Guidance: Start with ≤4 models for the first baseline run (cost/benefit), then expand toward ~10 once the harness and judge calibration are stable.
• Selection Guidance: For the initial ≤4 models, prefer one ceiling (frontier), one likely production default (midtier), one cost floor (budget), and one long-context specialist. Keep “free-tier/experimental gateway” models optional because availability and behavior can change.
• Completion Criteria: Registry contains ≥8 models across ≥3 providers; TypeScript types exported; unit test validates schema

Task 1.4: Implement matrix runner orchestrator

• Task: Create [packages/praecis/youtube/src/eval/matrix-runner.ts] with runEvaluationMatrix(corpus: CorpusEntry[], models: EvalModel[], options: MatrixOptions): Promise<MatrixResult> that iterates video × model combinations, invokes extraction, and collects raw claim sets
• Rationale: Orchestrator separates extraction execution from scoring (separation of concerns). Supports partial runs, resume-on-failure, and parallel execution per model. It must also support pipeline variants so we can measure deltas (e.g., editorial second pass on/off).
• Variant Requirement: Matrix keys include an extractorVariantId (e.g., raw, editorial-pass-v1, editorial-pass-v2) so we can run ablations without changing model IDs.
• Contract Requirement: Define (and export) the core eval types so downstream modules are not guesswork:
  • ExtractorVariantId
  • VideoContext (explicitly what judge sees)
  • MatrixOptions
  • MatrixCell
• Spec Definition (Minimum)

  export interface VideoContext {
    videoId: string;
    title: string;
    channelName: string;
    description?: string;
    url?: string;
    durationMinutes?: number;
    topicDomain?: string;
  }
  
  export interface MatrixOptions {
    outputDir: string;
    resume: boolean;
    dryRun: boolean;
    variants: ExtractorVariantId[];
    judgeModels: string[];
    maxConcurrency: number;
    timeoutMs: number;
    // Optional evaluation-only overrides to avoid silent truncation/drops.
    extractionMaxTokens?: number;
    extractionMaxChunks?: number;
  }
  
  export interface MatrixCell {
    videoId: string;
    modelId: string;
    extractorVariantId: ExtractorVariantId;
    claimSet: ClaimCandidate[];
    scores?: ClaimSetScore[];
    consensusScore?: {
      mean: ClaimSetScore;
      variance: Partial<Record<ScoreDimension, number>>;
    };
    error?: { message: string; code?: string };
  }
  
  export type ScoreDimension =
    | "completeness"
    | "accuracy"
    | "topicCoverage"
    | "atomicity"
    | "overallScore";
  
  export interface MatrixResult {
    cells: MatrixCell[];
    metadata: {
      startedAt: string;
      completedAt?: string;
      config: MatrixOptions;
      failedCellCount: number;
    };
  }
  

• Failure Semantics: For any cell that still fails after extraction retries, record a structured error on that MatrixCell and continue the run (do not abort the entire matrix by default).
• Logging Requirement: Emit structured progress logs so long runs are readable:
  • Example: [cell 12/60] videoId=h_1zlead9ZU modelId=gpt-5-mini variant=raw status=ok durationMs=53210
• Regression Guard: Each run produces a deterministic output keyed by videoId + modelId + extractorVariantId + promptVersion; results cached to out/eval-matrix/runs/; raw prompt/response traces stored per cell for debugging
• Completion Criteria: Orchestrator completes a 2-video × 2-model × 2-variant matrix in <10 minutes using cached transcripts; outputs structured JSON per cell

Task 1.4b: Make extractor variants non-stringly-typed

• Task: Create [packages/praecis/youtube/src/eval/extractor-variants.ts] exporting ExtractorVariantId and a registry of supported variants (initial: raw, editorial-pass-v1, editorial-pass-v2)
• Rationale: Variants are a first-class experimental axis; a registry prevents silent typos and makes reports consistent.
• Regression Guard: Matrix runner validates requested variants against the registry before executing
• Completion Criteria: Invalid variant IDs fail fast with a clear error; CLI --variants flag (Task 3.4) is validated against the registry
  • Optional Variant: Add single-pass for models whose contextWindow can handle the full transcript and prompt overhead, to compare chunk-and-merge vs full-context extraction.

Task 1.5: Add matrix result caching layer

• Task: Create [packages/praecis/youtube/src/eval/matrix-cache.ts] implementing read/write for matrix cell results keyed by sha256(videoId + modelId + extractorVariantId + promptVersion + extractorVersion)
• Rationale: LLM extraction is expensive (~$0.01–0.50 per video×model cell). Caching prevents re-extraction when only scoring logic changes. Cache invalidation on prompt/extractor version change ensures freshness.
• Regression Guard: Cache miss triggers extraction; cache hit skips extraction and logs cache-hit; stale cache detected by version mismatch. If extractorVersion is unavailable, fall back to unknown-v0 (but emit a warning).
• Completion Criteria: Second run of identical matrix completes in <5 seconds; version bump triggers full re-extraction

Task 1.6: Create human-verified golden annotations

• Task: Create [packages/praecis/youtube/tests/fixtures/eval-matrix/golden-annotations.json] containing human-verified "ideal" claim sets for 2 representative videos from the corpus (one short, one long)
• Task: Define schema for annotations: videoId, idealClaims: { text: string, evidence?: { quote?: string, startMs?: number, endMs?: number } }[], rejectedClaims: { text: string, reason: string }[]
• Rationale: To trust the "LLM-as-Judge", we must calibrate it against human judgment. This "Golden Set" serves as the ground truth for validating the scoring engine itself (Task 2.2).
• Note: This Golden Set is distinct from the AIDHA-TASK-003-ATOMIC golden extraction fixtures (e.g., claims-golden.json). Those protect extraction outputs; this set calibrates the judge/scorer.
• Annotation Guidance: Create a short, co-located guideline (golden-annotations-guidelines.md) that defines what counts as a substantive claim, how to split atomic claims, and how to handle hedged language.
• Schema Change (Fixture Shape): GoldenAnnotation.idealClaims is now an object array (not string[]) to allow optional evidence metadata. Any existing golden-annotations.json fixtures and any parsing/validation code must migrate.
  • Before:

    {
      "videoId": "abc",
      "idealClaims": ["Claim 1", "Claim 2"],
      "rejectedClaims": []
    }
    

  • After:

    {
      "videoId": "abc",
      "idealClaims": [
        { "text": "Claim 1" },
        {
          "text": "Claim 2",
          "evidence": { "startMs": 123000, "endMs": 129000 }
        }
      ],
      "rejectedClaims": []
    }
    

• Regression Guard: JSON validated by schema; contains at least 2 videos
• Spec Definition:

  interface GoldenAnnotation {
    videoId: string;
    idealClaims: {
      text: string; // The perfect set of claims a human would extract
      evidence?: { quote?: string; startMs?: number; endMs?: number };
    }[];
    rejectedClaims: {
      text: string;
      reason: "hallucination" | "redundant" | "fragment" | "topic-drift";
    }[];
  }
  

• Completion Criteria: Fixture file exists and contains manually curated claims for the selected videos

Task 1.7: Capture independent manual baseline (no harness)

• Task: Create [docs/55-testing/eval-matrix/manual-baseline-no-harness.md] describing a manual procedure to extract claims directly via external UIs (Gemini web and/or ChatGPT UI) from a small subset of transcripts
• Task: Create [packages/praecis/youtube/tests/fixtures/eval-matrix/manual-baseline/] containing captured prompt/response snapshots for at least:
  • 2 videos (or 1 video with 2 distant segments: early and late)
  • 2 different external UIs/models (e.g., Gemini web and ChatGPT)
  • 2 extraction instructions (“high recall” vs “high precision”) to expose editorial-like filtering behavior
• Rationale: This bypasses our harness entirely and answers a key question: “Are we consistently excluding valuable content because of our prompts/post-processing/editorial second pass, independent of model selection and wrappers?”
• Baseline Protocol (Minimum)
  • Use the same committed transcript excerpt (from Task 1.2) as input (paste into UI), with an explicit “do not use outside knowledge” instruction.
  • Use the lowest-temperature / most-deterministic setting available (or note when the UI does not expose it).
  • Use an identical prompt template per “mode” (high recall / high precision), committed alongside the snapshots (e.g., manual-baseline/prompt-template-high-recall.md and manual-baseline/prompt-template-high-precision.md).
  • Use a predictable snapshot naming convention, e.g. {videoId}-{provider}-{mode}.md.
  • Request atomic claims; ask for a short “what I excluded and why” section.
  • Record: date/time, UI/provider, model name if visible, any toggles (temperature/verbosity), prompt text, raw response.
• Comparison Questions
  • Which claim categories are present in the manual baseline but missing in harness output?
  • Do missing categories correlate with editorial second pass filtering rules (e.g., “speculative”, “hedged”, “anecdotal”, “mechanism explanations”)?
  • Are the same categories missing across different external UIs/models (systematic harness issue) or do they vary by model (model capability issue)?
  • For models with very large context windows: does single-context “unfettered” extraction identify cross-segment claims that chunked extraction misses?
• Regression Guard: Snapshots stored alongside the excerpt input text used; snapshots are stable and referenced from the write-up
• Governance: If any snapshot content is committed and derived from third-party transcript text, register it in AIDHA-GOV-005.
• Comparison Method: This comparison is a one-time manual human review step. The write-up should be committed in the same PR as the snapshots it analyzes.
• Completion Criteria: Manual baseline doc exists; at least 4 prompt/response snapshots captured; write-up identifies at least 3 concrete “systematic miss” patterns or explicitly states none found

Task 1.8: Evaluate editorial second pass via ablation (raw vs filtered)

• Task: Define an extractorVariantId for “raw” (no editorial pass) and “editorial-pass-v1” (current), and include both in the evaluation matrix runs for at least 2 videos × 3 models
• Rationale: This isolates whether the editorial second pass is trading away completeness/topic coverage disproportionately relative to accuracy/atomicity gains.
• Cross-Reference: editorial-pass-v1/editorial-pass-v2 should map to the editorial ranking variants described in AIDHA-PLAN-004, so ablations are interpretable.
• Additivity Check (Silent Loss Risks): For at least 1 dense video, include a sensitivity check to detect silent claim loss due to:
  • maxTokens truncation (e.g., run extraction with default vs higher extractionMaxTokens and compare deltas)
  • maxChunks capping (set extractionMaxChunks high enough to cover full duration for long videos; document any production caps and expected completeness impact)
• Regression Guard: Variant IDs are first-class in cache keys and reports (no accidental overwrites)
• Completion Criteria: Report includes a “variant delta” section showing score deltas and qualitative deltas (missingClaims/hallucinations changes) between raw vs editorial-pass-v1

---

Phase 2: LLM-as-Judge Scoring Engine

Task 2.1: Define scoring rubric schema

• Task: Create [packages/praecis/youtube/src/eval/scoring-rubric.ts] exporting zod schemas for ClaimSetScore with fields: completeness: number (0–10), accuracy: number (0–10), topicCoverage: number (0–10), atomicity: number (0–10), overallScore: number (0–10), reasoning: string, missingClaims: { text: string }[], hallucinations: { text: string }[], redundancies: { text: string }[], gapAreas: { area: string }[], plus judgeMeta: { judgeModelId: string, judgePromptVersion: string }
• Rationale: Structured scoring with explicit sub-dimensions prevents vague "good/bad" judgments. Zod validation at the boundary (engineering-principles.md §5) catches malformed judge responses before they corrupt aggregation.
• Definition: overallScore is the arithmetic mean of { completeness, accuracy, topicCoverage, atomicity } (unweighted). The scoring executor should recompute it from the four dimensions to prevent drift.
• Cross-Reference: Reuse ClaimCandidate from the extraction pipeline (packages/praecis/youtube/src/extract/types.ts). Do not redefine claim shapes in the eval module.
• Schema Change (Judge Output Shape): ClaimSetScore.missingClaims, hallucinations, redundancies, and gapAreas are now arrays of objects (not string[]). Update any parsers/aggregators and invalidate any cached judge outputs created under older schema versions.
• Regression Guard: Schema rejects scores outside 0–10 range; reasoning required non-empty; arrays may be empty but must be present
• Spec Definition:

  export const ClaimSetScoreSchema = z.object({
    completeness: z.number().min(0).max(10),
    accuracy: z.number().min(0).max(10),
    topicCoverage: z.number().min(0).max(10),
    atomicity: z.number().min(0).max(10),
    overallScore: z.number().min(0).max(10),
    reasoning: z.string().min(10),
    missingClaims: z.array(z.object({ text: z.string().min(1) })),
    hallucinations: z.array(z.object({ text: z.string().min(1) })),
    redundancies: z.array(z.object({ text: z.string().min(1) })),
    gapAreas: z.array(z.object({ area: z.string().min(1) })),
    judgeMeta: z.object({
      judgeModelId: z.string().min(1),
      judgePromptVersion: z.string().min(1),
    }),
  });
  

• Completion Criteria: Schema validates example scores; rejects out-of-range values; unit test covers edge cases (0, 10, missing fields)

Task 2.2: Implement judge prompt template

• Task: Create [packages/praecis/youtube/src/eval/prompts/judge-claim-quality.ts] exporting buildJudgePrompt(transcript: string, claims: ClaimCandidate[], videoContext: VideoContext): { system: string; user: string } that instructs the judge model to evaluate the four dimensions
• Rationale: The judge prompt is the most critical component. It must be calibrated against the human-verified Golden Set (Task 1.6) to ensure it penalizes what a human would penalize.
• Prompt Design:
  • System: "You are an expert evaluator of information extraction quality..."
  • Include the full transcript (or representative chunks) as ground truth
  • Include the extracted claim set to evaluate
  • Include 2 calibration examples derived from Task 1.6 (Golden Annotations)
  • Request structured JSON output matching ClaimSetScore schema
• Best Practice Requirements
  • Blind the judge to the extraction model/provider name (avoid “model X produced these”).
  • Randomize claim order (or explicitly instruct the judge not to assume ordering implies importance).
  • For long transcripts: support chunked scoring (e.g., score per excerpt/segment then aggregate) to avoid context-window truncation.
  • Mitigate transcript prompt-injection: wrap transcript content in explicit delimiters (e.g., <TRANSCRIPT>...</TRANSCRIPT>) and instruct the judge to ignore any instructions found inside.
• Prompt Structure Spec:
  • System: "You are an expert evaluator of information extraction quality... Output JSON only."
  • User:
    • TRANSCRIPT_CONTEXT: (Title, Channel, Description)
    • TRANSCRIPT_TEXT: (The text to evaluate against)
    • CANDIDATE_CLAIMS: (The JSON list of claims to score)
    • CALIBRATION_EXAMPLES: (Array of { claims: [], score: {}, reasoning: "" })
• Regression Guard: Prompt template tested for presence of all four dimension names; calibration examples included; output format instruction present
• Completion Criteria: Judge prompt produces parseable ClaimSetScore JSON from ≥2 different judge models; inter-rater agreement >0.7 on calibration examples

Task 2.2b: Calibrate judge prompt against Golden Set (iteration loop)

• Task: Define a calibration protocol that runs the judge over Golden Set examples and compares the resulting scores to expected human judgments (from Task 1.6)
• Rationale: “Inter-rater agreement >0.7” is only meaningful if we have an explicit loop to iterate prompts (and/or judge model choice) until that target is met or we decide it is infeasible.
• Acceptance Criteria: Calibration rounds are recorded (prompt version, judge model, deltas). Calibration is considered “pass” when agreement exceeds threshold for all four dimensions on the Golden Set examples, or a documented decision is made to adjust the threshold and why.

Task 2.2c: Specify chunked scoring strategy (when transcript exceeds contextWindow)

• Task: Specify and implement a chunking strategy for judge scoring:
  • Partition transcript into segments by token budget (with small overlap).
  • Score per segment, then aggregate to a single ClaimSetScore.
  • Define how missingClaims/hallucinations/redundancies union/dedup across segments.
• Rationale: Without a specified strategy, “support chunked scoring” becomes an untestable aspiration and long-form videos will behave inconsistently.
• Gating Rule: Use full-transcript scoring when model.contextWindow comfortably exceeds transcriptTokenCount + promptOverhead. Otherwise, chunk deterministically.
• Regression Guard: A unit test should cover chunking determinism: same transcript yields same segment boundaries and aggregate score.

Task 2.3: Implement scoring executor

• Task: Create [packages/praecis/youtube/src/eval/scoring-executor.ts] with scoreClaimSet(judgeClient: LlmClient, judgeModel: string, transcript: string, claims: ClaimCandidate[], videoContext: VideoContext): Promise<Result<ClaimSetScore>> that sends the judge prompt, validates the response, and retries once on parse failure
• Rationale: Scoring execution is separated from prompt construction (SRP) and from matrix orchestration. Retry with parse-error feedback (per task-003 Task 2.10 pattern) improves judge response quality.
• Implementation Note: Reuse the existing LlmClient abstraction from the ingestion/extraction stack where possible; judge model selection is per-call via judgeModel. If judges require different provider config or keys, model registry metadata should drive client routing rather than ad-hoc conditionals.
• Judge Token Budget: Configure a judge-specific maxTokens high enough to avoid truncation of structured diagnostics arrays (recommend starting range 4000–8000, then tune). Expose as AIDHA_EVAL_JUDGE_MAX_TOKENS.
• JSON Mode Prerequisite: If supportsJsonMode is true for a model, ensure the client can actually request JSON-only output (e.g., OpenAI-compatible response_format). If the current LlmClient cannot pass this through, add it as a prerequisite task before relying on JSON mode in eval.
• Regression Guard: Parse failures logged with raw response for debugging; retry includes validation error in follow-up prompt; timeout configurable via AIDHA_EVAL_JUDGE_TIMEOUT_MS
• Completion Criteria: Executor returns validated ClaimSetScore or structured error; retry success rate >80% on intentionally malformed responses

Task 2.4 (Optional Enhancement): Implement multi-judge consensus scoring

• Task: Create [packages/praecis/youtube/src/eval/consensus-scorer.ts] with scoreWithConsensus(...) that runs ≥2 judge models (configurable) and computes mean scores with inter-rater variance
• Rationale: Single-judge scoring is unreliable due to model-specific biases. Multi-judge consensus with variance reporting surfaces disagreements that indicate ambiguous extraction quality. Engineering-principles.md §8: "Optimise after measuring" — variance data guides judge selection.
• KISS Default: Single-judge scoring should be the default path initially; add consensus after Task 2.2b calibration is stable.
• Consensus Method: Mean of dimension scores; flag cells where any dimension variance >2.0 for manual review
• Regression Guard: Minimum 2 judges required; single-judge fallback emits warning; variance computed per dimension
• Completion Criteria: Consensus scorer produces mean + variance for all four dimensions; high-variance cells flagged in output

Task 2.5: Add judge response caching

• Task: Extend [packages/praecis/youtube/src/eval/matrix-cache.ts] to cache judge scores keyed by sha256(videoId + extractionModelId + judgeModelId + claimSetHash + judgePromptVersion)
• Rationale: Judge calls are as expensive as extraction calls. Caching prevents re-scoring when only reporting changes. Separate cache key from extraction cache ensures judge prompt changes trigger re-scoring without re-extraction.
• Regression Guard: Judge prompt version included in cache key; stale scores invalidated on prompt change
• Completion Criteria: Re-running scoring on cached extractions completes in <10 seconds; prompt version bump triggers re-scoring

---

Phase 3: Reporting & Visualisation

Task 3.1: Implement matrix result aggregator

• Task: Create [packages/praecis/youtube/src/eval/matrix-aggregator.ts] with aggregateMatrixResults(cells: MatrixCell[]): MatrixReport that computes per-model averages, per-video averages, overall rankings, and dimension-specific leaderboards
• Rationale: Raw cell scores are not actionable without aggregation. Per-model averages reveal which models extract best; per-video averages reveal which content types are hardest; dimension leaderboards show model strengths (e.g., "Model X is most accurate but least complete").
• Aggregation Metrics: mean, median, min, max, stddev per dimension per model; rank ordering by overall score; cost-efficiency ratio (score / cost)
• Regression Guard: Aggregator handles missing cells (partial matrix runs) gracefully; empty matrix returns structured error
• Spec Definition:

  type ScoreDimension =
    | "completeness"
    | "accuracy"
    | "topicCoverage"
    | "atomicity"
    | "overallScore";
  
  type StatName = "mean" | "median" | "min" | "max" | "stddev";
  
  type DimensionStats = Record<ScoreDimension, Record<StatName, number>>;
  
  interface MatrixReport {
    summary: { bestModel: string; worstModel: string; hardestVideo: string };
    modelStats: Record<string, { dimensions: DimensionStats; estimatedCostUsd?: number }>;
    videoStats: Record<string, { dimensions: DimensionStats }>;
    leaderboards: Record<ScoreDimension, { modelId: string; score: number }[]>;
  }
  

• Completion Criteria: Aggregator produces valid report from a 3×3 matrix; rankings are deterministic (tiebreaker by model name)

Task 3.2: Generate markdown comparison report

• Task: Create [packages/praecis/youtube/src/eval/report-markdown.ts] with renderMatrixReport(report: MatrixReport): string producing a markdown document with: summary table, per-model scorecards, per-video heatmap (using emoji indicators), dimension radar descriptions, cost analysis, and recommendations
• Rationale: Markdown output integrates with the existing dossier/docs workflow and is reviewable in PRs. Structured format enables both human review and automated trend detection.
• Output Structure:
  • Executive summary (best model, worst model, hardest video)
  • Model × Video score matrix table
  • Per-dimension leaderboard tables
  • Variant delta section (raw vs editorial-pass-v1)
  • Manual baseline delta notes (links to Task 1.7 artifacts)
  • Cost-efficiency analysis
  • Failure analysis (cells scoring <4 on any dimension)
• Regression Guard: Report renderer tested with mock data; output validated as parseable markdown
• Completion Criteria: Report renders cleanly in MkDocs preview (pnpm docs:serve); all sections populated from mock matrix data

Task 3.3: Generate JSON export for programmatic analysis

• Task: Create [packages/praecis/youtube/src/eval/report-json.ts] with exportMatrixJson(report: MatrixReport, options: { pretty?: boolean }): string producing machine-readable JSON with full cell-level detail
• Rationale: JSON export enables downstream tooling (dashboards, trend analysis, CI integration) without coupling to markdown rendering. Engineering-principles.md §2: separation of concerns between human-readable and machine-readable outputs.
• Regression Guard: JSON output validated against zod schema; round-trip test (export → parse → re-export) produces identical output
• Completion Criteria: JSON export includes all cell scores, aggregations, and metadata; file size <1MB for a 10×10 matrix

Task 3.4: Add CLI command for evaluation matrix

• Task: Add eval matrix subcommand to [packages/praecis/youtube/src/cli.ts] with flags: --corpus <path>, --models <comma-separated>, --tier <frontier|midtier|budget>, --judge-models <comma-separated>, --variants <comma-separated>, --output-dir <path>, --format <md|json|both>, --resume (skip cached cells), --dry-run (show matrix plan without execution), --max-concurrency <n>, --invalidate-run <runId>
• Rationale: CLI integration enables both interactive use and CI automation. --dry-run prevents accidental expensive runs; --resume enables incremental matrix completion.
• Regression Guard: CLI help text updated; --dry-run produces no LLM calls; unknown flags produce clear error
• Completion Criteria: pnpm -C packages/praecis/youtube cli eval matrix --dry-run --corpus <path> outputs planned matrix dimensions without API calls; --help documents all flags

Task 3.5: Implement run/cell invalidation (rollback tool)

• Task: Implement --invalidate-run <runId> to remove only the cache entries for a specific run (extraction + judge caches), without deleting unrelated caches
• Rationale: When a run is clearly corrupted (bad prompt version, partial failure, wrong config), we need a safe recovery mechanism that does not require manual filesystem surgery.
• Regression Guard: Invalidation prints exactly what it will delete (dry-run supported) and refuses to delete paths outside the expected cache roots
• Completion Criteria: A run can be invalidated and then re-run cleanly with fresh outputs, while other runs remain cached

---

Phase 4: Validation & CI Integration

Task 4.1: Create unit tests for scoring rubric

• Task: Create [packages/praecis/youtube/tests/eval/scoring-rubric.test.ts] testing zod schema validation for valid scores, boundary values, out-of-range rejection, missing required fields, and empty arrays
• Rationale: Test-first (engineering-principles.md §4). Scoring schema is the contract between judge LLM output and aggregation logic — schema bugs silently corrupt all downstream metrics.
• Regression Guard: Tests run in CI without LLM dependency; pure schema validation
• Completion Criteria: ≥10 test cases covering valid, invalid, and edge-case score payloads

Task 4.2: Create unit tests for judge prompt template

• Task: Create [packages/praecis/youtube/tests/eval/judge-prompt.test.ts] asserting: prompt contains all four dimension names, includes calibration examples, requests JSON output, includes transcript content, includes claim set
• Rationale: Prompt contract tests (per task-003 Phase 2 pattern) prevent silent prompt regression that degrades judge quality without visible code changes.
• Regression Guard: Tests are deterministic; no LLM calls; string assertion only
• Completion Criteria: Tests fail if any dimension name removed from prompt; tests fail if calibration examples removed

Task 4.3: Create integration test for matrix runner with mock LLM

• Task: Create [packages/praecis/youtube/tests/eval/matrix-runner.test.ts] running a 2-video × 2-model matrix with mock LLM client returning recorded responses, asserting: all cells populated, scores within valid range, cache populated, report generated
• Rationale: Integration test validates the full pipeline (extraction → scoring → aggregation → reporting) without live API calls. Engineering-principles.md §4: "Test pyramid — use integration tests for boundaries."
• Regression Guard: Mock LLM responses recorded from actual runs; deterministic replay
• Completion Criteria: Test completes in <30 seconds; all matrix cells contain valid scores; report markdown renders without errors

Task 4.4: Add CI quality gate for extraction regression

• Task: Create [packages/praecis/youtube/tests/eval/quality-gate.spec.ts] that loads the latest matrix report JSON and asserts either:
  • absolute minimums (initial bootstrap), and/or
  • no-regression vs a pinned baseline report (preferred once baseline exists)
• Rationale: Absolute thresholds tend to be brittle (corpus changes, judge drift). A pinned baseline with “no regression beyond delta” is usually the more stable CI signal.
• Regression Guard: Gate reads from cached report; does not trigger new extraction; thresholds configurable via environment variables
• CI-Safe Mode Requirement: CI must not run a full live matrix. Use a 2-video × 2-model smoke matrix with recorded/mock responses (Task 4.3) and compare against a pinned baseline report fixture (e.g., packages/praecis/youtube/tests/fixtures/eval-matrix/baseline-report.json) with a default tolerance (e.g., no dimension drops >1.0 point).
• Completion Criteria: CI fails if quality thresholds breached; baseline fixture path and tolerance are documented in test file comments

Task 4.5: Add cost tracking and budget alerting

• Task: Extend matrix runner to track token usage per cell and emit a cost summary in the report with fields: extractionTokens, judgeTokens, extractionCostUsd, judgeCostUsd, estimatedCostUsd, costPerCell, costPerModel, costPerVideo
• Rationale: A 10×10 matrix with judge scoring could cost $5–50+ per run. Cost visibility prevents budget surprises and enables cost-optimised model selection. Engineering-principles.md §5: "Think failure-first" — cost overrun is a failure mode.
• Regression Guard: Cost estimation uses conservative token-to-cost ratios from model registry; actual costs logged alongside estimates
• Completion Criteria: Cost summary appears in both markdown and JSON reports; total estimated cost displayed before execution in --dry-run mode

Task 4.6: Keep AIDHA-TESTING-001 current

• Task: When Phase 4 tests are added, update AIDHA-TESTING-001 to register the new eval tests in the packages/praecis/youtube map and refresh the baseline counts
• Rationale: The test suite map is how reviewers keep coverage coherent across a growing repo; new eval tests should be discoverable.
• Completion Criteria: AIDHA-TESTING-001 lists the new eval tests; baseline counts refreshed; pnpm docs:build succeeds.

---

Critical Path (Execution Order)

This is the minimum dependency order to avoid rework and undefined types/contracts.

flowchart TD
  A[1.1 Corpus JSON] --> B[1.1b Corpus schema + test]
  B --> C[1.2 Transcript local cache + committed excerpts]
  A --> D[1.3 Model registry]
  A --> E[1.4 Matrix runner + core eval types]
  E --> F[1.4b Extractor variant registry]
  F --> Z[1.8 Editorial ablation]
  E --> G[1.5 Extraction cache]
  C --> E
  C --> H[1.6 Golden annotations + guidelines]
  H --> I[2.1 Scoring rubric schema]
  I --> J[2.2 Judge prompt template]
  J --> K[2.2b Judge calibration loop]
  J --> L[2.2c Chunked scoring strategy]
  K --> M[2.3 Scoring executor]
  M --> N[2.5 Judge score caching]
  M --> O[2.4 Consensus scorer (optional)]
  E --> P[3.1 Aggregator]
  P --> Q[3.2 Markdown report]
  P --> R[3.3 JSON export]
  E --> S[3.4 CLI wiring]
  S --> Y[3.5 Invalidate-run rollback tool]
  I --> T[4.1 Rubric unit tests]
  J --> U[4.2 Prompt unit tests]
  E --> V[4.3 Mocked matrix integration test]
  R --> W[4.4 CI quality gate (pinned baseline)]
  S --> X[4.5 Cost tracking + dry-run estimate]