Evaluation Scripts

Create Golden Test Set

The create_test_set.py script helps you interactively build a golden test dataset for evaluating the retrieval system.

Quick Start

python scripts/create_test_set.py

How It Works

1. Shows random items from your collection with full details
2. Helps you write queries for each item with natural language
3. Automatically records ground truth (item IDs, categories, etc.)
4. Saves to data/eval/test_queries.json

Interactive Example

🔍 Golden Test Set Builder

Options:
  1. Browse random items from all categories
  2. Browse items from a specific category

How many items to browse? [10]: 5

════════════════════════════════════════
ITEM 1
════════════════════════════════════════
Category: Food
Headline: Tofuya Ukai dining spot beneath Tokyo Tower
Summary: Beautiful restaurant beneath Tokyo Tower...

Create a query for this item?
(y)es, (n)o, (m)ulti-item, (q)uit [y]: y

Query: What restaurants are in Tokyo?
Query Type: 1 (location_search)
Reference answer (optional): Tofuya Ukai

✓ Created query q001

Features

Query Creation Modes

• Single-item (y) - One query → one item
• Multi-item (m) - One query → multiple items (e.g., "all Tokyo restaurants")
• Skip (n) - Skip current item
• Quit (q) - Save and exit

Query Types

1. location_search - "restaurants in Tokyo"
2. category_search - "beauty products"
3. specific_question - "what perfume brands..."
4. object_content - "images with text"
5. complex_multi_part - "Japanese food and shopping"

Output Format

Creates data/eval/test_queries.json:

{
  "queries": [
    {
      "id": "q001",
      "query": "What restaurants are in Tokyo?",
      "type": "location_search",
      "ground_truth_items": ["item-id-1", "item-id-2"],
      "expected_category": "Food",
      "min_expected_results": 2,
      "reference_answer": "Tofuya Ukai"
    }
  ]
}

Best Practices

✅ Do:

• Create 20-30 diverse queries
• Use natural language
• Include various query types
• Add reference answers for key queries

❌ Don't:

• Create only one query type
• Use overly specific/technical queries
• Skip categories you want to evaluate

Keyboard Shortcuts

• y - Create query
• n - Skip item
• m - Multi-item query
• q - Quit
• Ctrl+C - Cancel current input

Next Steps

After creating queries:

# Review the test set
cat data/eval/test_queries.json | python -m json.tool

# Run evaluation (see below)
python scripts/evaluate_retrieval.py

Evaluate Retrieval Quality

The evaluate_retrieval.py script measures search quality using standard Information Retrieval metrics.

Quick Start

# Standard run against golden database (uses subdomain routing)
python scripts/evaluate_retrieval.py

# Verbose output with detailed progress
python scripts/evaluate_retrieval.py --verbose

Prerequisites

1. Start the API server on port 8000:

   uvicorn main:app --port 8000
   

   The evaluation script automatically uses golden.localhost subdomain routing to access the golden database.

2. Ensure you have the evaluation dataset:

   • data/eval/retrieval_evaluation_dataset.json (50 test queries)

How It Works

1. Validates API connection - Finds running API server (tries 8000, 8001, 8080, 3000)
2. Routes to golden database - Uses golden.localhost subdomain routing (via Host header)
3. Verifies item count - Ensures you're testing against the golden DB (55 items)
4. Runs all queries - Executes 50 test queries against the search endpoint
5. Calculates metrics - Computes Precision@K, Recall@K, MRR, NDCG@K
6. Generates reports - Creates markdown and JSON reports with detailed results

Metrics Calculated

For each K value (1, 3, 5, 10):

• Precision@K - Of the top K results, what fraction are relevant?
• Recall@K - Of all relevant items, what fraction appear in top K?
• NDCG@K - Normalized Discounted Cumulative Gain (accounts for ranking quality)
• MRR - Mean Reciprocal Rank (average of 1/rank of first relevant result)

Command-Line Options

python scripts/evaluate_retrieval.py [OPTIONS]

Options:
  --port PORT                API port (default: 8000)
  --base-url URL            Full base URL (overrides port)
  --use-golden-subdomain    Use golden.localhost routing (default: True)
  --no-golden-subdomain     Disable golden routing (test against production)
  --dataset PATH            Evaluation dataset path
  --output-dir PATH         Report output directory (default: data/eval/reports)
  --top-k VALUES            K values for metrics (default: 1,3,5,10)
  --expected-items N        Expected DB items (default: 55)
  --skip-item-check         Skip item count validation
  --verbose                 Show detailed progress

Examples

# Run against golden database (default)
python scripts/evaluate_retrieval.py

# Run against production database
python scripts/evaluate_retrieval.py --no-golden-subdomain --skip-item-check

# Custom K values
python scripts/evaluate_retrieval.py --top-k 1,5,10,20

# Remote server
python scripts/evaluate_retrieval.py --base-url http://192.168.1.100:8000

# Custom output location
python scripts/evaluate_retrieval.py --output-dir ./my_reports

Output Reports

Creates two timestamped files in data/eval/reports/:

1. eval_YYYYMMDD_HHMMSS_report.md - Human-readable markdown report

   • Summary metrics table
   • Breakdown by query type
   • Detailed results for all 50 queries

2. eval_YYYYMMDD_HHMMSS_report.json - Machine-readable JSON

   • Full metric data
   • Individual query results
   • Timing statistics
   • Configuration details

Sample Output

============================================================
Retrieval Evaluation Script
============================================================
✓ API endpoint: http://localhost:8000
✓ Item count validated: 55 items

Loaded dataset: 50 queries

Evaluating 50 queries...
  Progress: 50/50 (100%)
Completed in 8.43s

✓ Reports generated:
  - data/eval/reports/eval_20241214_153022_report.json
  - data/eval/reports/eval_20241214_153022_report.md

============================================================
Evaluation complete!
============================================================

Safety Features

The script prevents accidental evaluation against production:

• Golden subdomain routing - Automatically routes to golden database by default
• Item count validation - Checks that DB has exactly 55 items (golden dataset size)
• Port auto-discovery - Tries common ports if default fails
• Clear warnings - Shows warning if item count doesn't match expected

Use --no-golden-subdomain --skip-item-check to run against production instead.