Harvard Study Shows AI Tops ER Doctors in Diagnoses
Researchers from Harvard Medical School and Beth Israel Deaconess Medical Center have released a study that tests large language models in medical scenarios. The work, published this week in the journal Science, includes analysis of actual emergency room cases. In those instances, one AI model provided diagnoses that matched or exceeded the accuracy of human doctors.
The team, made up of physicians and computer scientists, ran multiple experiments. They compared OpenAI's o1 and 4o models against human physicians. The goal was to gauge performance across different medical tasks.
Experiment with Real ER Patients
A key part of the research involved 76 patients who visited the Beth Israel emergency room. Two attending physicians reviewed each case and offered diagnoses. Separately, OpenAI's o1 and 4o models generated their own diagnoses based on the same records.
Two additional attending physicians then evaluated all these diagnoses. They did not know which came from AI and which from humans. This blind setup ensured fair comparison.
The study reported that at every stage of diagnosis, o1 either matched or slightly outperformed the two original physicians. The 4o model also performed well. Differences stood out most at the initial triage stage. There, limited patient information creates high pressure for quick, correct calls.
Strong Triage Performance
During triage, the o1 model achieved an exact or very close diagnosis in 67% of cases. One physician reached that level in 55% of instances. The other physician succeeded in 50%.
Importantly, the AI received no special preparation. Models saw only the text from electronic medical records available at each decision point. This matched exactly what physicians used.
Arjun Manrai, who leads an AI lab at Harvard Medical School and served as a lead author, commented in a Harvard press release. "We tested the AI model against virtually every benchmark, and it eclipsed both prior models and our physician baselines," he said.
Harvard Medical School traces its roots to 1782. It stands as one of the world's leading institutions for medical education and research. Beth Israel Deaconess Medical Center, a major teaching hospital in Boston, affiliates closely with Harvard. It handles thousands of emergency visits yearly and supports advanced clinical studies.
Stay ahead of the AI curve
The most important updates, news, and content — delivered weekly.
No spam. Unsubscribe anytime.
OpenAI, the developer of o1 and 4o, focuses on artificial general intelligence. The company released these models as advanced tools for complex reasoning tasks. O1 emphasizes step-by-step thinking, while 4o handles multimodal inputs including text and images.
Cautions and Next Steps
The study makes no claim that AI can handle life-or-death choices in emergency rooms yet. Instead, results point to an urgent need for prospective trials. Those would test the technology directly in patient care environments.
Researchers limited their work to text-based inputs. They noted that foundation models struggle more with images or other non-text data, based on prior studies.
Adam Rodman, a Beth Israel physician and lead author, spoke to the Guardian. He pointed out the lack of a formal framework for AI accountability in diagnoses. Patients, he said, prefer human guidance for life-or-death and tough treatment choices.
This research builds on growing interest in AI for healthcare. Institutions like Harvard continue to explore how such tools might assist clinicians without replacing them.
Broader Implications for Medical AI
The findings come amid rapid advances in AI capabilities. Models like those from OpenAI now tackle specialized domains such as medicine. Yet, deployment requires rigorous validation.
Beth Israel Deaconess, founded in 1916 through mergers, ranks among top U.S. hospitals for research. It pioneered electronic health records and now integrates AI in trials.
Manrai's AI lab at Harvard examines computational methods for health data analysis. Rodman's clinical role complements this with practical emergency medicine insights.
Overall, the study underscores AI's promise in high-stakes diagnostics. It also reinforces the value of human expertise and the push for controlled real-world testing.