simple-evals
FreeEval tools by OpenAI.
FreeFree tier
About simple-evals
Simple-evals is a lightweight open-source library developed by OpenAI for evaluating language model performance. It provides reference implementations of standard benchmarks including MMLU, GPQA, MATH, HumanEval, MGSM, DROP, SimpleQA, and more. Originally created to transparently publish accuracy numbers alongside new OpenAI models, the repository also hosts reference implementations for HealthBench, BrowseComp, and SimpleQA. Note: As of July 2025, the library is deprecated and will no longer receive updates for new models or benchmark results.
Key Features
Lightweight library for evaluating language models
Reference implementations for multiple benchmarks: MMLU, GPQA, MATH, HumanEval, MGSM, DROP, SimpleQA, HealthBench, BrowseComp
Provides transparency for OpenAI model accuracy numbers
Open-source code under the openai GitHub organization
Supports evaluation with standard prompting methods
Pros & Cons
Pros
- Open-source and free to use
- Lightweight and easy to set up
- Provides standardized, reproducible evaluation scripts
- Includes reference results for many popular benchmarks
- Directly from OpenAI, ensuring consistency with their reported numbers
Cons
- Deprecated as of July 2025; no longer updated for new models or benchmarks
- Limited to the benchmarks and method implementations provided
- Requires access to OpenAI API keys or compatible models
- Does not support custom dataset creation out-of-the-box
Best For
Benchmarking large language models across standard datasetsReproducing OpenAI's reported evaluation scoresTesting model performance on a variety of tasks including math, code, reasoning, and reading comprehensionAcademic research on LLM evaluation
FAQ
What is simple-evals?
Simple-evals is a lightweight open-source library for evaluating language models, created by OpenAI to transparently publish accuracy numbers alongside their latest models.
Is simple-evals still maintained?
No, as of July 2025 the repository is deprecated and will no longer be updated for new models or benchmark results. It continues to host reference implementations for HealthBench, BrowseComp, and SimpleQA.
What benchmarks are included?
The library includes implementations for MMLU, GPQA, MATH, HumanEval, MGSM, DROP, SimpleQA, HealthBench, BrowseComp, and multilingual MMLU.
Do I need an API key to use simple-evals?
Yes, simple-evals requires API access to evaluate models (e.g., OpenAI API keys) or access to model outputs that follow the expected input format.