LM Evaluation Harness Reference

Overview

The LM Evaluation Harness evaluates large language models across 60+ academic benchmarks using standardized prompts and metrics. It is the industry standard used by EleutherAI, HuggingFace, and major labs.

When to Use

• Benchmarking model quality for academic papers
• Comparing model quality across standard tasks
• Tracking training progress over time
• Reporting standardized metrics for reproducibility
• When reproducible evaluation is required

Prerequisites

• Python environment with pip
• NVIDIA GPU with CUDA 11.8+ (CPU possible but very slow)
• VRAM: 16GB for 7B model (bf16), 8GB (8-bit); 28GB for 13B (bf16), 14GB (8-bit); 70B requires multi-GPU or quantization
• Install lm-eval:

pip install lm-eval

• For vLLM:

pip install vllm

Capabilities

The harness evaluates LLMs on 60+ academic benchmarks including MMLU, HumanEval, GSM8K, TruthfulQA, HellaSwag, ARC, WinoGrande, and MBPP. It supports HuggingFace models, the vLLM backend, and API models (OpenAI, Anthropic, etc.). It supports quantized models (4-bit/8-bit) and CPU offloading, custom checkpoints and tokenizers, few-shot evaluation with configurable number of shots, batch size auto-detection, logging individual predictions, multi-GPU strategies (data parallel, tensor parallel), custom task creation, and code-executing tasks (HumanEval, MBPP) with an explicit confirmation flag.

Parameters

--model (required): Backend to use for evaluation (hf, vllm, or API).

--model_args (required): Arguments for the model backend, e.g., pretrained=model-name, dtype=bfloat16, load_in_4bit=True, tokenizer=..., tensor_parallel_size=N, gpu_memory_utilization=0.8.

--tasks (required): Comma-separated list of benchmark tasks (e.g., mmlu,gsm8k,hellaswag).

--num_fewshot (optional): Number of few-shot examples to use (default 5; 0 for zero-shot).

--batch_size (optional): Batch size for evaluation; auto for auto-detection, or integer (e.g., 8).

--device (optional): Device to use (e.g., cuda:0).

--output_path (optional): Path to save results JSON file.

--log_samples (optional): Flag to save individual predictions.

--confirm_run_unsafe_code (required for code tasks): Required flag to run code-executing tasks (HumanEval, MBPP).

Standard Benchmark Evaluation

Benchmark Evaluation:

• Step 1: Choose benchmark suite
• Step 2: Configure model
• Step 3: Run evaluation
• Step 4: Analyze results

Step 1: Choose Benchmark Suite

Choose core reasoning benchmarks (MMLU, GSM8K, HellaSwag, TruthfulQA, ARC) and code benchmarks (HumanEval, MBPP). The recommended standard suite is:

--tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge

Step 2: Configure Model

Specify the HuggingFace model with pretrained=..., optionally set dtype, load_in_4bit/load_in_8bit, custom tokenizer path, device (cuda:0), and batch_size (auto or integer).

Step 3: Run Evaluation

Use the lm_eval command with --model hf, --model_args, --tasks, --num_fewshot, --batch_size, --output_path, and --log_samples.

Evaluate any HuggingFace model:

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag \
  --device cuda:0 \
  --batch_size 8

View available tasks:

lm-eval ls tasks

Quantized model:

lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,load_in_4bit=True \
  --tasks mmlu \
  --device cuda:0

Custom checkpoint:

lm_eval --model hf \
  --model_args pretrained=/path/to/my-model,tokenizer=/path/to/tokenizer \
  --tasks mmlu \
  --device cuda:0

Full MMLU with 5-shot:

# Full MMLU evaluation (57 subjects)
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu \
  --num_fewshot 5 \  # 5-shot evaluation (standard)
  --batch_size 8 \
  --output_path results/ \
  --log_samples  # Save individual predictions

# Multiple benchmarks at once
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu,gsm8k,hellaswag,truthfulqa,arc_challenge \
  --num_fewshot 5 \
  --batch_size 8 \
  --output_path results/llama2-7b-eval.json

Step 4: Analyze Results

JSON output contains results per task with primary metric (acc, exact_match, acc_norm) and stderr, plus config.

{
  "results": {
    "mmlu": {
      "acc": 0.459,
      "acc_stderr": 0.004
    },
    "gsm8k": {
      "exact_match": 0.142,
      "exact_match_stderr": 0.006
    },
    "hellaswag": {
      "acc_norm": 0.765,
      "acc_norm_stderr": 0.004
    }
  },
  "config": {
    "model": "hf",
    "model_args": "pretrained=meta-llama/Llama-2-7b-hf",
    "num_fewshot": 5
  }
}

Track Training Progress

Training Progress Tracking:

• Step 1: Set up periodic evaluation
• Step 2: Choose quick benchmarks
• Step 3: Automate evaluation
• Step 4: Plot learning curves

Step 1: Set Up Periodic Evaluation

Create a script that takes a checkpoint directory and step number, and runs lm_eval on that checkpoint.

#!/bin/bash
# eval_checkpoint.sh

CHECKPOINT_DIR=$1
STEP=$2

lm_eval --model hf \
  --model_args pretrained=$CHECKPOINT_DIR/checkpoint-$STEP \
  --tasks gsm8k,hellaswag \
  --num_fewshot 0 \  # 0-shot for speed
  --batch_size 16 \
  --output_path results/step-$STEP.json

Step 2: Choose Quick Benchmarks

For frequent evaluation, use HellaSwag (~10 min), GSM8K (~5 min), and PIQA (~2 min). Avoid MMLU (~2 hours) and HumanEval (requires code execution).

Step 3: Automate Evaluation

Integrate into the training loop by saving a checkpoint every N steps and running the eval script.

# In training loop
if step % eval_interval == 0:
    model.save_pretrained(f"checkpoints/step-{step}")

    # Run evaluation
    os.system(f"./eval_checkpoint.sh checkpoints step-{step}")

Or use a PyTorch Lightning callback:

from pytorch_lightning import Callback

class EvalHarnessCallback(Callback):
    def on_validation_epoch_end(self, trainer, pl_module):
        step = trainer.global_step
        checkpoint_path = f"checkpoints/step-{step}"

        # Save checkpoint
        trainer.save_checkpoint(checkpoint_path)

        # Run lm-eval
        os.system(f"lm_eval --model hf --model_args pretrained={checkpoint_path} ...")

Step 4: Plot Learning Curves

Load all result JSONs, extract step and metric, and plot with matplotlib.

import json
import matplotlib.pyplot as plt

# Load all results
steps = []
mmlu_scores = []

for file in sorted(glob.glob("results/step-*.json")):
    with open(file) as f:
        data = json.load(f)
        step = int(file.split("-")[1].split(".")[0])
        steps.append(step)
        mmlu_scores.append(data["results"]["mmlu"]["acc"])

# Plot
plt.plot(steps, mmlu_scores)
plt.xlabel("Training Step")
plt.ylabel("MMLU Accuracy")
plt.title("Training Progress")
plt.savefig("training_curve.png")

Compare Multiple Models

Model Comparison:

• Step 1: Define model list
• Step 2: Run evaluations
• Step 3: Generate comparison table

Step 1: Define Model List

Create a text file with HuggingFace model names, one per line.

# models.txt
meta-llama/Llama-2-7b-hf
meta-llama/Llama-2-13b-hf
mistralai/Mistral-7B-v0.1
microsoft/phi-2

Step 2: Run Evaluations

Loop over models, run lm_eval with the same tasks and settings, and save results to separate files named by model.

#!/bin/bash
# eval_all_models.sh

TASKS="mmlu,gsm8k,hellaswag,truthfulqa"

while read model; do
    echo "Evaluating $model"

    # Extract model name for output file
    model_name=$(echo $model | sed 's/\//-/g')

    lm_eval --model hf \
      --model_args pretrained=$model,dtype=bfloat16 \
      --tasks $TASKS \
      --num_fewshot 5 \
      --batch_size auto \
      --output_path results/$model_name.json

done < models.txt

Step 3: Generate Comparison Table

Load all results, extract the primary metric per task, create a pandas DataFrame, and output as a markdown table.

import json
import pandas as pd

models = [
    "meta-llama-Llama-2-7b-hf",
    "meta-llama-Llama-2-13b-hf",
    "mistralai-Mistral-7B-v0.1",
    "microsoft-phi-2"
]

tasks = ["mmlu", "gsm8k", "hellaswag", "truthfulqa"]

results = []
for model in models:
    with open(f"results/{model}.json") as f:
        data = json.load(f)
        row = {"Model": model.replace("-", "/")}
        for task in tasks:
            # Get primary metric for each task
            metrics = data["results"][task]
            if "acc" in metrics:
                row[task.upper()] = f"{metrics['acc']:.3f}"
            elif "exact_match" in metrics:
                row[task.upper()] = f"{metrics['exact_match']:.3f}"
        results.append(row)

df = pd.DataFrame(results)
print(df.to_markdown(index=False))

Example output:

Model	MMLU	GSM8K	HELLASWAG	TRUTHFULQA
meta-llama/Llama-2-7b	0.459	0.142	0.765	0.391
meta-llama/Llama-2-13b	0.549	0.287	0.801	0.430
mistralai/Mistral-7B	0.626	0.395	0.812	0.428
microsoft/phi-2	0.560	0.613	0.682	0.447

Evaluate with vLLM (Faster Inference)

vLLM Evaluation:

• Step 1: Install vLLM
• Step 2: Configure vLLM backend
• Step 3: Run evaluation

Step 1: Install vLLM

pip install vllm

Step 2: Configure vLLM Backend

Use --model vllm and set model_args with pretrained, tensor_parallel_size, dtype=auto, and gpu_memory_utilization=0.8.

lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=1,dtype=auto,gpu_memory_utilization=0.8 \
  --tasks mmlu \
  --batch_size auto

Step 3: Run Evaluation

vLLM is 5-10x faster than standard HuggingFace (e.g., MMLU on 7B: ~15-20 min vs ~2 hours).

# Standard HF: ~2 hours for MMLU on 7B model
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-2-7b-hf \
  --tasks mmlu \
  --batch_size 8

# vLLM: ~15-20 minutes for MMLU on 7B model
lm_eval --model vllm \
  --model_args pretrained=meta-llama/Llama-2-7b-hf,tensor_parallel_size=2 \
  --tasks mmlu \
  --batch_size auto

For multi-GPU setups:

lm_eval --model vllm \
  --model_args pretrained=model-name,tensor_parallel_size=2

Constraints and Caveats

• Code-executing tasks (HumanEval, MBPP) require the --confirm_run_unsafe_code flag; without it, the task is refused.
• MMLU evaluation takes ~2 hours on a 7B model (single A100); avoid for frequent training progress tracking.
• HumanEval requires a code execution environment.
• Different results than reported may occur due to incorrect fewshot count, task name, or model/tokenizer mismatch.
• Out of memory errors can be mitigated by reducing batch size, using quantization, or enabling CPU offloading.
• Evaluation on CPU is very slow.
• 70B model requires multi-GPU or quantization.

Failure Modes

Evaluation Too Slow

Use the vLLM backend, reduce fewshot examples:

--num_fewshot 0  # Instead of 5

Or evaluate a subset of MMLU:

--tasks mmlu_stem  # Only STEM subjects

Out of Memory

Reduce batch size:

--batch_size 1  # Or --batch_size auto

Use quantization:

--model_args pretrained=model-name,load_in_8bit=True

Enable CPU offloading:

--model_args pretrained=model-name,device_map=auto,offload_folder=offload

Different Results Than Reported

Check the fewshot count:

--num_fewshot 5  # Most papers use 5-shot

Check the exact task name:

--tasks mmlu  # Not mmlu_direct or mmlu_fewshot

Ensure model and tokenizer match:

--model_args pretrained=model-name,tokenizer=same-model-name

HumanEval Not Executing Code

Missing the --confirm_run_unsafe_code flag:

lm_eval --model hf \
  --model_args pretrained=model-name \
  --tasks humaneval \
  --confirm_run_unsafe_code  # Required to run tasks that execute generated code