MMedBench
Freea benchmark that evaluates large language models' ability to answer medical questions across multiple languages.
FreeFree tier
Inputs: textOutputs: text
About MMedBench
MMedBench is a multilingual medical benchmark designed to evaluate large language models (LLMs) on answering multiple-choice medical questions across six languages. It comprises 45,000 training samples and 8,518 test samples, each paired with a correct answer and a high-quality rationale. The benchmark includes a public leaderboard comparing models such as GPT-4, Llama 3, and MMed-Llama 3 on both accuracy and rationale quality (measured by BLEU-1). The dataset and submission instructions are available on GitHub, and a detailed paper is referenced for further methodology.
Key Features
Multilingual medical multiple-choice dataset covering 6 languages
Includes 45,000 training samples and 8,518 test samples
Each question provides a correct answer and a high-quality rationale
Public leaderboard tracking model accuracy and rationale quality (BLEU-1)
Supports model submission via GitHub repository
Accompanied by a research paper detailing methodology
Open-source dataset and code available
Pros & Cons
Pros
- Covers six diverse languages, enabling cross-lingual medical evaluation
- Provides high-quality rationales for each answer, allowing evaluation of explanation quality
- Large dataset with separate train and test splits (45k/8.5k)
- Public leaderboard with results from leading models (GPT-4, Llama, Mistral, etc.)
- Open-source and freely available for research use
- Supported by a detailed paper for reproducibility
Cons
- Limited to multiple-choice format, not evaluating open-ended generation
- Only covers six languages, missing many other medically important languages
- Leaderboard evaluation metrics (accuracy and BLEU-1) may not capture clinical usefulness
- No built-in evaluation service or API; users must run evaluations themselves
- Rationale quality metric (BLEU-1) may not correlate with clinical accuracy
Best For
Evaluating large language models on multilingual medical question answeringBenchmarking model performance on medical knowledge across languagesResearch on rationale generation in medical QATraining and fine-tuning models for multilingual medical tasksComparing open-source and proprietary models on a standardized medical benchmark
FAQ
What is MMedBench?
MMedBench is a benchmark for evaluating large language models on multilingual medical multiple-choice questions. It includes 45k training and 8.5k test samples across 6 languages, each with a correct answer and rationale.
How many languages does MMedBench cover?
MMedBench covers 6 languages. The specific languages are not listed on the homepage, but the dataset is designed for multilingual medical QA.
How can I submit my model to the leaderboard?
To submit a model, please follow the instructions in the GitHub repository linked on the website.
Is the dataset freely available?
Yes, the MMedBench dataset is open-source and can be downloaded from the GitHub repository. It is free to use for research.