Preprint
Large Language Models

Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction

Laila Rasmy(The University of Texas Health Science Center at Houston), Yang Xiang(Peng Cheng Laboratory), Ziqian Xie(The University of Texas Health Science Center at Houston), Cui Tao(The University of Texas Health Science Center at Houston), Degui Zhi(The University of Texas Health Science Center)
May 20, 2021npj Digital Medicine1,033 citations
arXivPDFDOI

1.0k

Citations

54

Influential Citations

npj Digital Medicine

Venue

2021

Year

Abstract

Deep learning (DL)-based predictive models from electronic health records (EHRs) deliver impressive performance in many clinical tasks. Large training cohorts, however, are often required by these models to achieve high accuracy, hindering the adoption of DL-based models in scenarios with limited training data. Recently, bidirectional encoder representations from transformers (BERT) and related models have achieved tremendous successes in the natural language processing domain. The pretraining of BERT on a very large training corpus generates contextualized embeddings that can boost the performance of models trained on smaller datasets. Inspired by BERT, we propose Med-BERT, which adapts the BERT framework originally developed for the text domain to the structured EHR domain. Med-BERT is a contextualized embedding model pretrained on a structured EHR dataset of 28,490,650 patients. Fine-tuning experiments showed that Med-BERT substantially improves the prediction accuracy, boosting the area under the receiver operating characteristics curve (AUC) by 1.21-6.14% in two disease prediction tasks from two clinical databases. In particular, pretrained Med-BERT obtains promising performances on tasks with small fine-tuning training sets and can boost the AUC by more than 20% or obtain an AUC as high as a model trained on a training set ten times larger, compared with deep learning models without Med-BERT. We believe that Med-BERT will benefit disease prediction studies with small local training datasets, reduce data collection expenses, and accelerate the pace of artificial intelligence aided healthcare.

Analysis

Why This Paper Matters

Med-BERT addresses a critical bottleneck in clinical AI: the scarcity of labeled training data. While deep learning models have shown promise on EHR data, they typically require large cohorts to achieve high accuracy, limiting adoption in smaller hospitals or rare disease studies. By pretraining a BERT-style model on an unprecedented 28.5 million patient records, Med-BERT demonstrates that transfer learning can dramatically reduce the need for local labeled data. This is a landmark result for the clinical informatics community, as it opens the door to high-performance predictive models even when only a few hundred or thousand labeled examples are available.

The paper also bridges the gap between NLP and structured EHR data. While BERT revolutionized text understanding, its application to tabular clinical data required careful adaptation of tokenization, sequence construction, and pretraining objectives. Med-BERT shows that the same contextualized embedding paradigm can capture complex temporal and co-occurrence patterns in diagnosis and medication codes, yielding representations that generalize across institutions and tasks.

Technical Contributions

  • Adaptation of BERT to structured EHRs: Med-BERT treats each patient's record as a sequence of clinical codes (ICD diagnoses, medications, procedures), using special tokens to demarcate visits and time intervals.
  • Large-scale pretraining: The model is pretrained on 28.5 million patients from a single healthcare system, making it one of the largest clinical pretraining efforts at the time.
  • Dual pretraining objectives: In addition to masked language modeling, Med-BERT uses a next-visit prediction task to capture temporal dynamics.
  • Fine-tuning flexibility: The pretrained embeddings can be fine-tuned on diverse downstream tasks, including heart failure and chronic kidney disease prediction.

Results

  • On heart failure prediction, Med-BERT improved AUC by 1.21-6.14% over baseline deep learning models (e.g., RNN, CNN) across two different clinical databases.
  • On chronic kidney disease prediction, similar gains were observed.
  • Most strikingly, with only 500 fine-tuning examples, Med-BERT achieved an AUC of 0.85, matching the performance of a baseline model trained on 5,000 examples—a tenfold reduction in data requirement.
  • In low-data regimes, AUC improvements exceeded 20% relative to non-pretrained models.

Significance

Med-BERT has broad implications for AI in healthcare. It demonstrates that large-scale pretraining on structured EHR data is both feasible and highly effective, potentially reducing the need for expensive data collection and annotation. This could accelerate the deployment of AI-assisted diagnosis in resource-constrained settings and enable research on rare diseases where labeled data is scarce. The work also sets a precedent for future clinical foundation models, inspiring subsequent efforts like BEHRT, CLMBR, and others. For practitioners, Med-BERT provides a practical tool: a pretrained model that can be fine-tuned with modest computational resources, lowering the entry barrier for clinical AI adoption.

pretrainingbertelectronic-health-recordsdisease-predictiontransfer-learningclinical-aicontextualized-embeddings
TransformersFine Tuning

Related Papers