Golden Set — Medical-Retrieval Probes

This document describes the construction protocol for the medical-retrieval golden set used in experiment E3 (§10.4 of the paper).

Status: protocol drafted; probe set to be generated against the live ingested corpus once make services is up and the DDAs in examples/ and the PDFs in PDFs/ have been ingested.

Goal

Measure how well the hybrid retrieval pipeline (FAISS + graph context + optional ontology re-rank) returns the right context for the assistant to ground a proactive guidance turn. This is not a Q&A benchmark: probes are framed as context requests, not as user questions.

Probe shape

Each probe has the form:

{
  "id": "probe-001",
  "scenario": "short natural-language description of the in-conversation moment",
  "probe_query": "the actual retrieval query the assistant would issue",
  "patient_context": {
    "active_conditions": [...],
    "medications": [...],
    "allergies": [...],
    "recent_symptoms": [...]
  },
  "expected_relevant_chunks": ["chunk_id_1", "chunk_id_2", ...],
  "expected_relevant_entities": ["entity_id_1", "entity_id_2", ...],
  "rationale": "why these are the right chunks/entities for this scenario",
  "tags": ["contraindication", "guidance", "monitoring", ...]
}

Categories of probes (target ~30–50 total)

We aim for balanced coverage across the kinds of context the assistant must fetch in a real conversation. The categories below are calibrated to the proactive-guidance framing — none of them are framed as direct questions from the patient.

Category	n	Description
C1 — Symptom contextualisation	~8	Patient mentions a new symptom; retrieve related conditions, common precursors, red-flag indicators.
C2 — Medication safety lookup	~8	Active prescription + new symptom or new prescription; retrieve interaction guidance, contraindications, monitoring requirements.
C3 — Condition-trajectory grounding	~6	Existing diagnosis; retrieve typical progression, monitoring intervals, expected next steps.
C4 — Procedure / device context	~5	Patient has a recent procedure or device; retrieve aftercare, warning signs, follow-up windows.
C5 — Allergy / sensitivity check	~5	New medication / food / environmental exposure; retrieve allergy profile and known sensitivities.
C6 — Population-level context	~4	Demographic / risk-factor based; retrieve relevant guidelines (e.g. age-band screening reminders).
C7 — Adherence / lifestyle nudge context	~4	Retrieve evidence to support a proactive lifestyle nudge consistent with the patient's profile.

Construction protocol

1. Seed. After ingesting examples/ and PDFs/, list all (Chunk, ExtractedEntity, BusinessConcept) triples and group them by ontology class.
2. Draft scenarios. For each category, hand-craft scenarios that the ingested corpus can plausibly support. Avoid scenarios that depend on knowledge not in the corpus.
3. Probe query phrasing. Phrase each probe as a context-fetching string, not a question. Example: ✅ "context for: new-onset epigastric pain in a patient on long-term NSAIDs" rather than ❌ "What causes epigastric pain?".
4. Annotate ground truth. For each probe, identify the chunks and KG entities that the authors believe are correct context. Record the rationale.
5. Author review. Two-author cross-check on at least 30% of probes for inter-rater agreement. Disagreements resolved by discussion; surviving ambiguities tagged disputed and excluded from headline numbers.
6. Clinician sanity check (recommended, not required for this paper). Flag any probe that would benefit from clinician review for the future conversation-level evaluation (§12 / F4).

What is explicitly out of scope

• Free-form patient questions. That is a Q&A benchmark; we do not build one because SynapseFlow is not a Q&A system.
• Multi-turn dialogue traces. Reserved for future work (§12 / F4).
• Clinical correctness of any generated response. This golden set evaluates retrieval, not generation.

Scoring (cross-reference to §10.4)

For each probe and each condition (C1, C2, C3 in §10.4):

• recall@k over expected_relevant_chunks ∪ expected_relevant_entities
• precision@k over the same union
• MRR over the union
• (optional) nDCG@10 with binary relevance

Aggregate: macro-average across probes within a category, then across categories.

Reproducibility

The final probe set will be checked in as golden_set_rag.json once generated. The seed corpus is what is in examples/ and PDFs/ at the commit referenced from paper/README.md ("the exact branch used for the results in this paper").