Neura Market
LM Evaluation Harness logo

LM Evaluation Harness

Free

EleutherAI's unified LLM evaluation framework

FreeFree tier
Inputs: text
Type
Open Source
Company
EleutherAI
Visit Website GitHub (40 stars)

About LM Evaluation Harness

The LM Evaluation Harness is an open-source framework developed by EleutherAI for few-shot evaluation of autoregressive language models. It provides a unified interface to evaluate models on hundreds of standard benchmarks, including tasks from the Open LLM Leaderboard. The framework supports config-based task creation with YAML files, Jinja2 prompt design, output post-processing, and answer extraction. It integrates with multiple model backends such as HuggingFace (transformers), vLLM, SGLang, and API-based models, and also includes a prototype for multimodal (text+image) evaluation. Recent updates have introduced a CLI with subcommands (run, ls, validate), lighter installation options, and support for model steering and chain-of-thought trace stripping.

Key Features

Config-based task creation and configuration with YAML support
Support for multiple model backends: HuggingFace, vLLM, SGLang, and API models
Jinja2 prompt design for easy modification of prompts
Output post-processing and answer extraction
CLI with subcommands: run, ls, validate
Multimodal (text+image) evaluation prototype
Integration with Open LLM Leaderboard tasks

Pros & Cons

Pros
  • Open-source and free to use
  • Unified framework supporting a wide range of models and tasks
  • Active development with regular updates and community contributions
  • Config-based task creation simplifies custom evaluation
  • Supports both local and API-based model backends
  • Includes multimodal evaluation capabilities (prototype)
Cons
  • Primarily focused on autoregressive language models; limited support for other architectures
  • Multimodal evaluation is still in prototype stage
  • Requires technical expertise to install, configure, and run
  • Dependency on external model backends may require significant compute resources
  • Documentation is primarily on GitHub; may be scattered

Best For

Evaluate autoregressive language models on standard benchmarksRun few-shot evaluations for leaderboard submissionsBenchmark model performance across multiple tasksTest model reasoning using chain-of-thought tasks (e.g., BIG-Bench-Hard)Develop custom evaluation tasks using YAML configuration

FAQ

What model backends are supported?
The LM Evaluation Harness supports HuggingFace (transformers), vLLM, SGLang, and API-based models. It also has prototype support for multimodal models.
Is the LM Evaluation Harness free?
Yes, it is open-source and free to use. It is hosted on GitHub under the EleutherAI organization.
How can I install it?
Installation is done via pip. The base package no longer includes transformers/torch; you can install model backends separately, e.g., pip install lm_eval[hf] for HuggingFace models, pip install lm_eval[vllm] for vLLM support.