From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond

This research paper systematically investigates run-time steering strategies for large language models (LLMs) applied to medical challenge problems. It builds on the Medprompt approach—which uses chain-of-thought reasoning and ensembling to achieve state-of-the-art medical performance with a general LLM like GPT-4—and compares it with OpenAI's o1-preview model, a reasoning-native model that performs run-time inference before generating responses. The study evaluates o1-preview across diverse medical benchmarks, finding that it outperforms GPT-4 with Medprompt even without prompting, and that few-shot prompting actually hinders o1's performance. It also analyzes cost-performance trade-offs, revealing a Pareto frontier where GPT-4o offers affordability while o1-preview achieves top performance at higher cost. The paper notes near-saturation on existing medical benchmarks and discusses the need for more challenging benchmarks, reflecting on general directions for inference-time computation.