13-02-PLAN

---
phase: 13-retrieval-evaluation-framework
plan: 02
type: execute
wave: 1
depends_on: []
files_modified:
  - src/main/java/dev/alexandria/search/eval/EvaluationExporter.java
  - src/main/java/dev/alexandria/search/eval/GoldenSetEntry.java
  - src/main/java/dev/alexandria/search/eval/EvaluationResult.java
  - src/main/resources/eval/golden-set.json
  - src/main/resources/application.yml
  - src/test/java/dev/alexandria/search/eval/EvaluationExporterTest.java
autonomous: true
requirements:
  - EVAL-02
  - EVAL-05

must_haves:
  truths:
    - "A golden-set.json file contains 100 annotated queries with graded relevance judgments"
    - "Each query has a type (factual, conceptual, code_lookup, troubleshooting) matching the required distribution"
    - "Each query has relevance judgments with chunk identifiers and grades 0-2"
    - "EvaluationExporter writes aggregate CSV with global and per-type metrics"
    - "EvaluationExporter writes detailed CSV with per-query chunk results"
    - "CSV filenames contain ISO timestamp and configurable label"
    - "Output directory is configurable via application properties"
  artifacts:
    - path: "src/main/resources/eval/golden-set.json"
      provides: "100 annotated queries for retrieval evaluation"
      min_lines: 200
    - path: "src/main/java/dev/alexandria/search/eval/GoldenSetEntry.java"
      provides: "Record for parsing golden set entries"
      contains: "record GoldenSetEntry"
    - path: "src/main/java/dev/alexandria/search/eval/EvaluationResult.java"
      provides: "Record for per-query evaluation results"
      contains: "record EvaluationResult"
    - path: "src/main/java/dev/alexandria/search/eval/EvaluationExporter.java"
      provides: "CSV export service for evaluation results"
      contains: "class EvaluationExporter"
    - path: "src/test/java/dev/alexandria/search/eval/EvaluationExporterTest.java"
      provides: "Unit tests for CSV export"
      min_lines: 50
  key_links:
    - from: "src/main/java/dev/alexandria/search/eval/EvaluationExporter.java"
      to: "src/main/resources/application.yml"
      via: "@Value for output directory and default label"
      pattern: "alexandria\\.eval"
    - from: "src/main/resources/eval/golden-set.json"
      to: "src/main/java/dev/alexandria/search/eval/GoldenSetEntry.java"
      via: "Jackson deserialization"
      pattern: "GoldenSetEntry"
---

<objective>
Create the golden set of 100 annotated queries and the CSV export service for evaluation results.

Purpose: The golden set provides the ground truth for measuring retrieval quality. The exporter enables tracking metric trends across pipeline changes (phases 14-18). Both are independent of the metrics computation (plan 01).

Output: golden-set.json with 100 queries, GoldenSetEntry/EvaluationResult records, EvaluationExporter service, application config.
</objective>

<execution_context>
@./.claude/get-shit-done/workflows/execute-plan.md
@./.claude/get-shit-done/templates/summary.md
</execution_context>

<context>
@.planning/PROJECT.md
@.planning/ROADMAP.md
@.planning/STATE.md
@.planning/phases/13-retrieval-evaluation-framework/13-CONTEXT.md
@src/main/resources/application.yml
</context>

<tasks>

<task type="auto">
  <name>Task 1: Create golden set data model and 100 annotated queries</name>
  <files>
    src/main/java/dev/alexandria/search/eval/GoldenSetEntry.java
    src/main/resources/eval/golden-set.json
  </files>
  <action>
    Create `GoldenSetEntry` record in `dev.alexandria.search.eval`:
    ```java
    public record GoldenSetEntry(
        String query,
        QueryType queryType,
        List<RelevanceJudgment> judgments
    ) {}
    ```
    Note: `RelevanceJudgment` and `QueryType` are created in plan 01. If plan 02 executes first, create minimal placeholder versions that plan 01 will flesh out. Use the same package `dev.alexandria.search.eval`.

    Create `src/main/resources/eval/golden-set.json` with 100 queries about Spring Boot documentation.

    **Distribution (per user decision):**
    - ~30 code lookup queries (e.g., "How to create a REST controller with Spring Boot", "Spring Boot @Transactional annotation example")
    - ~30 factual queries (e.g., "What is the default embedded server in Spring Boot", "Spring Boot auto-configuration order")
    - ~25 conceptual queries (e.g., "How does Spring Boot auto-configuration work", "Difference between @Component and @Service")
    - ~15 troubleshooting queries (e.g., "Spring Boot application fails to start with port already in use", "How to fix circular dependency in Spring Boot")

    **JSON structure:**
    ```json
    [
      {
        "query": "How to create a REST controller in Spring Boot",
        "queryType": "CODE_LOOKUP",
        "judgments": [
          { "chunkId": "spring-boot/web/rest-controllers#creating-rest-controller", "grade": 2 },
          { "chunkId": "spring-boot/web/rest-controllers#request-mapping", "grade": 1 },
          { "chunkId": "spring-boot/web/rest-controllers#response-body", "grade": 1 }
        ]
      }
    ]
    ```

    **ChunkId format:** Use semantic path-style identifiers that map to Spring Boot documentation sections. These are logical identifiers — the actual matching against real chunks happens in the integration test (plan 03) which maps these to real chunk content via source_url + section_path metadata.

    **Relevance grades (per user decision):**
    - 0 = not relevant (do NOT include grade-0 entries in the JSON — absence means 0)
    - 1 = partially relevant (related topic but not the primary answer)
    - 2 = highly relevant (directly answers the query)

    Each query should have 2-5 relevant judgments (grade 1 or 2). Aim for realistic relevance — not every query will have 5 highly relevant chunks.

    **Quality criteria for queries:**
    - Queries should be natural language, as a developer would type them
    - Vary specificity: some broad ("Spring Boot configuration"), some narrow ("@ConditionalOnProperty annotation syntax")
    - Include queries with technical terms (annotations, class names) and queries with natural language descriptions
  </action>
  <verify>
    - File `src/main/resources/eval/golden-set.json` exists and is valid JSON
    - JSON contains exactly 100 entries
    - Distribution: approximately 30 CODE_LOOKUP, 30 FACTUAL, 25 CONCEPTUAL, 15 TROUBLESHOOTING
    - Each entry has query, queryType, and at least 1 judgment with grade 1 or 2
    - `./gradlew compileJava` passes
  </verify>
  <done>
    golden-set.json contains 100 diverse, well-structured queries with graded relevance judgments covering all 4 query types in the specified distribution
  </done>
</task>

<task type="auto">
  <name>Task 2: Create CSV export service with application configuration</name>
  <files>
    src/main/java/dev/alexandria/search/eval/EvaluationResult.java
    src/main/java/dev/alexandria/search/eval/EvaluationExporter.java
    src/main/resources/application.yml
    src/test/java/dev/alexandria/search/eval/EvaluationExporterTest.java
  </files>
  <action>
    **EvaluationResult record:**
    ```java
    public record EvaluationResult(
        String query,
        QueryType queryType,
        List<ChunkResult> chunkResults,
        double recallAt5, double recallAt10, double recallAt20,
        double precisionAt5, double precisionAt10, double precisionAt20,
        double mrr,
        double ndcgAt5, double ndcgAt10, double ndcgAt20,
        double averagePrecision,
        double hitRateAt5, double hitRateAt10, double hitRateAt20
    ) {
        public record ChunkResult(String chunkId, double score, int rank, int relevanceGrade) {}
    }
    ```

    **EvaluationExporter** Spring @Service:
    - Constructor-injected with `@Value("${alexandria.eval.output-dir:${user.home}/.alexandria/eval}")` for output directory
    - `export(List<EvaluationResult> results, String label)` method that writes BOTH CSVs:

    **Aggregate CSV** (`eval-aggregate-{timestamp}-{label}.csv`):
    - Columns: query_type, count, recall_at_5, recall_at_10, recall_at_20, precision_at_5, precision_at_10, precision_at_20, mrr, ndcg_at_5, ndcg_at_10, ndcg_at_20, map, hit_rate_at_5, hit_rate_at_10, hit_rate_at_20
    - One row per QueryType (FACTUAL, CONCEPTUAL, CODE_LOOKUP, TROUBLESHOOTING) with averages
    - One GLOBAL row with overall averages across all queries
    - Values formatted to 4 decimal places

    **Detailed CSV** (`eval-detailed-{timestamp}-{label}.csv`):
    - Columns: query, query_type, chunk_id, score, rank, relevance_grade, recall_at_10, mrr, ndcg_at_10
    - One row per query+chunk combination (per user decision: "query + chunk_id + score + rang + jugement de pertinence")
    - Include per-query metrics on the first chunk row for each query

    **Timestamp format:** ISO local datetime with hyphens replacing colons (e.g., `2026-02-21T14-30-00`), per user decision.

    **Directory creation:** Create output directory if it does not exist.

    **Application config additions** to `application.yml`:
    ```yaml
    alexandria:
      eval:
        output-dir: ${ALEXANDRIA_EVAL_DIR:${user.home}/.alexandria/eval}
        thresholds:
          recall-at-10: 0.70
          mrr: 0.60
    ```

    **Unit tests** for EvaluationExporter:
    - Test aggregate CSV output format with known EvaluationResult data (write to temp directory)
    - Test detailed CSV output format
    - Test filename contains timestamp and label
    - Test directory creation when it does not exist
    - Use `@TempDir` for isolated filesystem testing — no mocks needed for pure I/O verification
  </action>
  <verify>
    - `./gradlew test --tests "dev.alexandria.search.eval.EvaluationExporterTest"` passes
    - `./gradlew spotlessApply && ./gradlew compileJava` passes
    - Application config has `alexandria.eval.output-dir` and `alexandria.eval.thresholds` properties
  </verify>
  <done>
    EvaluationExporter writes correctly formatted aggregate and detailed CSVs to a configurable directory with timestamped filenames, and application.yml has configurable thresholds and output directory
  </done>
</task>

</tasks>

<verification>
- golden-set.json contains exactly 100 entries with correct type distribution
- EvaluationExporter unit tests pass
- Application properties include eval configuration
- `./gradlew spotlessApply && ./gradlew compileJava` passes clean
</verification>

<success_criteria>
- 100 annotated queries exist in golden-set.json with graded relevance judgments
- Query type distribution matches: ~30 code, ~30 factual, ~25 conceptual, ~15 troubleshooting
- CSV export produces two files (aggregate + detailed) with correct format
- Output directory and thresholds are configurable via application.yml
</success_criteria>

<output>
After completion, create `.planning/phases/13-retrieval-evaluation-framework/13-02-SUMMARY.md`
</output>