MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes
FreeFirst public benchmark for detecting and correcting medical errors in clinical notes
About MEDEC: A Benchmark for Medical Error Detection and Correction in Clinical Notes
MEDEC is the first publicly available benchmark for medical error detection and correction in clinical notes. It covers five error types: Diagnosis, Management, Treatment, Pharmacotherapy, and Causal Organism. The dataset comprises 3,848 clinical texts, including 488 real clinical notes from three US hospital systems that were not previously seen by any large language model. MEDEC was used in the MEDIQA-CORR shared task and has been evaluated on recent LLMs such as o1-preview, GPT-4, Claude 3.5 Sonnet, and Gemini 2.0 Flash. A comparative study with medical doctors shows that while LLMs perform well, they are still outperformed by human experts, highlighting the benchmark's challenge and importance for improving clinical text validation.
Key Features
Pros & Cons
- First publicly available benchmark for medical error detection and correction in clinical notes
- Uses real clinical notes from multiple hospital systems, ensuring ecological validity
- Covers a diverse set of clinically relevant error types
- Includes human expert performance comparison, providing a strong baseline
- Challenging benchmark that highlights gaps between LLMs and human doctors
- LLMs still significantly underperform compared to medical doctors in error detection and correction
- Limited to five predefined error categories, may not capture all clinical errors
- Evaluation metrics may have limitations, as discussed in the paper
- Dataset size (3,848 texts) may be insufficient for some training purposes