Golden Dataset Guidelines

What Is a Golden Dataset?

A golden dataset is a curated collection of examples with known-correct answers that you use to:

1. Benchmark — Measure system performance objectively
2. Regression test — Detect when changes break things
3. Compare — Evaluate different approaches
4. Calibrate — Tune evaluation metrics

It's called "golden" because it represents the ground truth — the standard against which everything is measured.

---

Structure of Each Example

# Example entry in golden_dataset.yaml

- id: "query_001"                    # Unique identifier
  
  # The query
  query: "How do I export my data to CSV?"
  
  # Metadata
  category: "how-to"                 # Query type
  difficulty: "simple"               # simple, medium, hard
  created_at: "2025-01-15"
  source: "support_tickets"          # Where this query came from
  
  # Ground truth for RETRIEVAL evaluation
  relevant_documents:
    - doc_id: "user_guide_export"
      chunk_ids: ["chunk_127", "chunk_128"]
      relevance: "high"              # high, medium, low
    - doc_id: "api_reference_export"
      chunk_ids: ["chunk_456"]
      relevance: "medium"
  
  # Ground truth for GENERATION evaluation
  reference_answer: |
    To export your data to CSV:
    1. Go to Settings > Data > Export
    2. Select "CSV" as the format
    3. Choose the date range for your data
    4. Click "Export"
    
    Your file will download automatically. Large exports may take a few minutes.
  
  # Specific requirements
  must_include:                      # Answer MUST mention these
    - "Settings"
    - "Export"
    - "CSV"
  must_not_include:                  # Answer must NOT mention these
    - "pricing"
    - "subscription"
  
  # Expected behavior
  expected_behavior: "direct_answer"  # direct_answer, clarify, decline, escalate
  
  # Notes
  notes: "Steps verified against v2.3. UI changed in v2.4 - update needed."

---

Required Categories

Include examples from each category:

1. Simple Factual (Minimum: 4)

Questions with clear, single answers.

- id: "factual_001"
  query: "What file formats are supported for import?"
  category: "simple_factual"
  difficulty: "simple"
  expected_behavior: "direct_answer"

Why include: Baseline. If the system can't handle these, something is fundamentally broken.

2. How-To (Minimum: 4)

Step-by-step procedural questions.

- id: "howto_001"
  query: "How do I reset my password?"
  category: "how_to"
  difficulty: "simple"
  expected_behavior: "direct_answer"

Why include: Common user need. Tests ability to convey procedures clearly.

3. Troubleshooting (Minimum: 4)

Error resolution and debugging.

- id: "trouble_001"
  query: "Why am I getting error 403 when I try to access the API?"
  category: "troubleshooting"
  difficulty: "medium"
  expected_behavior: "direct_answer"

Why include: High value. These are the queries that currently become support tickets.

4. Comparison (Minimum: 2)

Questions requiring synthesis across multiple sources.

- id: "compare_001"
  query: "What's the difference between the Basic and Pro plans?"
  category: "comparison"
  difficulty: "medium"
  expected_behavior: "direct_answer"

Why include: Tests ability to synthesize. Requires pulling from multiple chunks.

5. Complex/Multi-Step (Minimum: 2)

Nuanced questions requiring deep understanding.

- id: "complex_001"
  query: "How do I set up SSO with Okta for my team of 50 users?"
  category: "complex"
  difficulty: "hard"
  expected_behavior: "direct_answer"

Why include: Stress test. If the system handles these, it's robust.

6. Out of Scope (Minimum: 2)

Questions the system should NOT answer directly.

- id: "oos_001"
  query: "What's your company's annual revenue?"
  category: "out_of_scope"
  difficulty: "simple"
  expected_behavior: "decline"
  reference_answer: "I can help with product documentation, but I don't have information about company financials."

Why include: Tests guardrails. System should gracefully decline, not hallucinate.

7. Ambiguous (Minimum: 2)

Unclear questions needing clarification.

- id: "ambig_001"
  query: "How do I do the thing with the data?"
  category: "ambiguous"
  difficulty: "medium"
  expected_behavior: "clarify"
  reference_answer: "I'd be happy to help with your data. Could you tell me more about what you're trying to do? For example, are you looking to export, import, or analyze your data?"

Why include: Tests edge handling. System should ask for clarification, not guess.

---

Building Your Dataset

Step 1: Gather Real Queries

Best sources:

• Support ticket history (anonymized)
• Search logs
• User feedback
• Sales/customer success input
• Your own testing

Don't invent queries. Real queries are messier and more valuable.

Step 2: Categorize and Deduplicate

Group similar queries. Keep one representative example per concept.

Step 3: Identify Ground Truth

For each query:

1. Find the documents that contain the answer
2. Note specific chunks if using chunked retrieval
3. Write or identify the ideal answer

This is labor-intensive. It's also the most valuable part.

Step 4: Define Requirements

For each query:

• What MUST the answer include?
• What must it NOT include?
• What's the expected behavior?

Step 5: Validate

Have someone else review:

• Are the relevant docs actually relevant?
• Is the reference answer actually correct?
• Are the requirements reasonable?

Step 6: Version and Maintain

• Track when examples were created/updated
• Note which product version they apply to
• Update when docs change

---

Quality Checklist

For each example:

• Query is realistic (from real users or realistic scenario)
• Category is correctly assigned
• Difficulty is accurately assessed
• Relevant documents are correctly identified
• All relevant chunks are listed
• Reference answer is accurate and complete
• Must-include items are necessary (not just nice-to-have)
• Must-not-include items are genuinely problematic
• Expected behavior is appropriate
• Notes include any caveats or version info

For the dataset as a whole:

• All categories are represented
• Difficulty distribution is reasonable
• No duplicate or near-duplicate queries
• Examples cover the main documentation areas
• Edge cases are included

---

Common Mistakes

1. Dataset Is Too Easy

If all examples are simple factual queries, you won't catch problems with harder cases.

Fix: Ensure balanced category distribution.

2. Reference Answers Are Wrong

If your ground truth is incorrect, your evaluation is meaningless.

Fix: Have domain experts validate reference answers.

3. Relevant Docs Are Incomplete

Missing relevant documents causes false negatives in retrieval evaluation.

Fix: Thorough review of all possible source documents.

4. Dataset Doesn't Match Production

If test queries don't represent real usage, you're optimizing for the wrong thing.

Fix: Use real queries whenever possible.

5. Dataset Is Static

Docs change. Products change. Static datasets become stale.

Fix: Regular review and update schedule.

---

Example Golden Dataset

Here's a minimal starter dataset (20 examples):

# golden_dataset.yaml

version: "1.0"
created_at: "2025-01-15"
product_version: "2.3"

examples:

  # ============ SIMPLE FACTUAL (4) ============
  
  - id: "factual_001"
    query: "What file formats can I import?"
    category: "simple_factual"
    difficulty: "simple"
    relevant_documents:
      - doc_id: "import_guide"
        chunk_ids: ["import_001"]
        relevance: "high"
    reference_answer: "You can import CSV, JSON, XML, and Excel (.xlsx) files."
    must_include: ["CSV", "JSON"]
    expected_behavior: "direct_answer"
  
  - id: "factual_002"
    query: "What is the maximum file size for uploads?"
    category: "simple_factual"
    difficulty: "simple"
    relevant_documents:
      - doc_id: "limits_guide"
        chunk_ids: ["limits_003"]
        relevance: "high"
    reference_answer: "The maximum file size for uploads is 100MB."
    must_include: ["100MB"]
    expected_behavior: "direct_answer"
  
  - id: "factual_003"
    query: "Which browsers are supported?"
    category: "simple_factual"
    difficulty: "simple"
    relevant_documents:
      - doc_id: "requirements"
        chunk_ids: ["req_002"]
        relevance: "high"
    reference_answer: "We support Chrome, Firefox, Safari, and Edge (latest two versions of each)."
    must_include: ["Chrome", "Firefox"]
    expected_behavior: "direct_answer"
  
  - id: "factual_004"
    query: "What's the API rate limit?"
    category: "simple_factual"
    difficulty: "simple"
    relevant_documents:
      - doc_id: "api_reference"
        chunk_ids: ["api_limits"]
        relevance: "high"
    reference_answer: "The API rate limit is 100 requests per minute per API key."
    must_include: ["100", "minute"]
    expected_behavior: "direct_answer"

  # ============ HOW-TO (4) ============
  
  - id: "howto_001"
    query: "How do I export my data to CSV?"
    category: "how_to"
    difficulty: "simple"
    relevant_documents:
      - doc_id: "export_guide"
        chunk_ids: ["export_001", "export_002"]
        relevance: "high"
    reference_answer: |
      To export your data to CSV:
      1. Go to Settings > Data > Export
      2. Select "CSV" as the format
      3. Choose the date range
      4. Click "Export"
    must_include: ["Settings", "Export", "CSV"]
    expected_behavior: "direct_answer"
  
  # ... (continue with remaining examples)
  
  # ============ TROUBLESHOOTING (4) ============
  
  - id: "trouble_001"
    query: "I'm getting error 403 when calling the API"
    category: "troubleshooting"
    difficulty: "medium"
    relevant_documents:
      - doc_id: "api_errors"
        chunk_ids: ["error_403"]
        relevance: "high"
      - doc_id: "api_auth"
        chunk_ids: ["auth_001"]
        relevance: "medium"
    reference_answer: |
      Error 403 means your API key doesn't have permission for this endpoint. Check that:
      1. Your API key is valid and not expired
      2. Your plan includes access to this endpoint
      3. You're using the correct API key for this environment
    must_include: ["permission", "API key"]
    expected_behavior: "direct_answer"
  
  # ... (continue)
  
  # ============ OUT OF SCOPE (2) ============
  
  - id: "oos_001"
    query: "What's your company's stock price?"
    category: "out_of_scope"
    difficulty: "simple"
    relevant_documents: []
    reference_answer: "I can help with product documentation, but I don't have information about company financials or stock prices."
    expected_behavior: "decline"
  
  # ... (continue)
  
  # ============ AMBIGUOUS (2) ============
  
  - id: "ambig_001"
    query: "How do I connect it?"
    category: "ambiguous"
    difficulty: "medium"
    relevant_documents: []
    reference_answer: "I'd be happy to help you connect. Could you tell me what you're trying to connect? For example, are you connecting to the API, a database integration, or a third-party service?"
    expected_behavior: "clarify"

---

Maintenance Schedule

Activity	Frequency	Owner
Review for accuracy	Monthly	[Name]
Add new examples	After major releases	[Name]
Remove outdated examples	After major releases	[Name]
Validate reference answers	Quarterly	[Name]
Check category balance	Quarterly	[Name]

---

Next Steps

1. Create your golden_dataset.yaml following this structure
2. Start with 20 examples (minimum for meaningful evaluation)
3. Validate with domain experts
4. Expand to 50-100 examples as you learn what matters

---

Template version: 1.0