OLMO-eval
Freea repository for evaluating open language models.
FreeFree tier
About OLMO-eval
OLMo-eval is a unified workbench for evaluating language models throughout the model development loop. It provides a registry of benchmark tasks and composable suites, supporting inference via vLLM, LiteLLM for commercial APIs, and a mock provider for dry runs. The project features a harness abstraction that separates execution policy from task definition, enabling both standard and tool-augmented evaluations without modification. It also supports multi-turn agentic evaluation with tool calling and sandboxed environments, LLM-as-judge scoring, aggregate and instance-level prediction storage, and inspection tooling for viewing instances, prompts, and model responses.
Key Features
Registry of benchmark tasks and composable suites with named variants for few-shot settings, formatting, and scoring
Support for inference via vLLM, LiteLLM for commercial APIs, and a mock provider for dry runs and debugging
Harness abstraction that separates execution policy from task definition, enabling baseline and tool-augmented evaluations without modification
Multi-turn agentic evaluation with tool calling, scaffolds, and sandboxed environments via Docker, Podman, or Modal
LLM-as-judge scoring with auxiliary providers, including locally served judge models
Aggregate and instance-level prediction storage
Inspection tooling for viewing instances, formatted prompts, token arrays, and model responses
Pros & Cons
Pros
- Open source and freely available on GitHub
- Flexible inference provider support: vLLM, LiteLLM, and mock
- Supports both standard and agentic evaluation with tool augmentation
- Harness abstraction allows easy swapping of tasks and execution policies
- Includes inspection tools for detailed analysis of prompts and model responses
Cons
- Requires technical setup with Python and uv package manager
- vLLM inference provider is Linux-only (not available on macOS natively)
- Primarily aimed at developers and researchers, not a turn-key product for non-technical users
Best For
Evaluating open language models during developmentRunning standard benchmark evaluations (e.g., MMLU, GSM8K, HumanEval) with few-shot variantsPerforming dry runs and debugging with mock inference providerConducting multi-turn agentic evaluations with tool calling and sandboxed environmentsUsing LLM-as-a-judge for scoring model outputs
FAQ
What inference providers does OLMo-eval support?
OLMo-eval supports inference via vLLM, LiteLLM for commercial APIs, and a built-in mock provider for dry runs and debugging.
Can OLMo-eval evaluate agentic behavior?
Yes, it supports multi-turn agentic evaluation with tool calling, scaffolds, and sandboxed environments using Docker, Podman, or Modal.