PubMedQA
Freea biomedical question-answering benchmark designed for answering research-related questions using PubMed abstracts.
FreeFree tier
About PubMedQA
PubMedQA is a biomedical question-answering benchmark designed for answering research questions with yes/no/maybe using corresponding PubMed abstracts. It consists of 1,000 expert-labeled, 61,200 unlabeled, and 211,300 artificially generated QA instances. The dataset is used to evaluate biomedical QA models, with a public leaderboard ranking models such as GPT-4, Med-PaLM 2, Meditron, Palmyra-Med, and others. The task requires reasoning over scientific literature, and the benchmark includes a leaderboard with accuracy and macro-F1 scores. The dataset and code are open source and available on GitHub.
Key Features
1,000 expert-labeled QA instances from PubMed
61,200 unlabeled and 211,300 artificially generated instances
Yes/no/maybe answer format
Public leaderboard with model accuracy rankings
Open source dataset and code on GitHub
Detailed paper citation and submission instructions
Pros & Cons
Pros
- Expert-labeled data ensures high-quality ground truth
- Large scale with multiple subsets for robust evaluation
- Public leaderboard facilitates model comparison and reproducibility
- Open source and freely available for academic and commercial use
- Widely adopted, with many state-of-the-art models reported
Cons
- Limited to yes/no/maybe answers, not free-form or explanatory
- Uses only PubMed abstracts, not full-text articles
- Benchmark may be approaching saturation, with top models exceeding human performance
- Primarily evaluates language models, not clinical deployment readiness
Best For
Benchmarking biomedical question-answering modelsEvaluating large language models on scientific reasoningResearch in natural language processing for medicineDeveloping and testing biomedical AI systems
FAQ
What task format does PubMedQA use?
PubMedQA requires answering research questions with yes/no/maybe based on corresponding PubMed abstracts.
How can I download the dataset?
The dataset can be downloaded from the PubMedQA GitHub repository: https://github.com/pubmedqa/pubmedqa
How do I submit my model to the leaderboard?
Follow the submission instructions provided in the GitHub repository.
How should I cite PubMedQA?
Use the citation: Jin et al., 'PubMedQA: A Dataset for Biomedical Research Question Answering,' EMNLP-IJCNLP 2019.