lm-evaluation-harness (EleutherAI)

The standard for academic-style benchmarking

lm-evaluation-harness (EleutherAI, open-source) is the de-facto framework for running standard NLP benchmarks against language models. If you have ever read "model X scored Y on MMLU," that number was almost certainly produced by lm-eval-harness or a fork.

What it covers

200+ benchmark tasks out of the box: MMLU, HellaSwag, ARC, GSM8K, HumanEval, TruthfulQA, BIG-Bench, and many more.
Provider-agnostic — runs on Hugging Face models, vLLM, OpenAI / Anthropic APIs, local llama.cpp, MLX, and more.
Reproducibility — exact prompts, exact metric definitions, exact post-processing. Numbers from lm-eval-harness are comparable across teams.

What it does NOT do

This is not a product-eval framework. It does not help you measure whether your specific RAG pipeline answers your specific users' questions. It tells you whether the underlying model has certain general capabilities. Use it to compare base models or to validate that fine-tuning did not degrade general capabilities — not to ship a feature.

Principle: lm-eval-harness is for comparing models. promptfoo / DeepEval / Braintrust are for comparing systems. Both have their place; do not substitute one for the other.

Code

Run MMLU on a Hugging Face model·bash

# pip install lm-eval[hf]
lm_eval --model hf \
  --model_args pretrained=meta-llama/Llama-3.2-3B-Instruct \
  --tasks mmlu \
  --device cuda \
  --batch_size 8 \
  --output_path results/llama-3.2-3b-mmlu.json

Multiple benchmarks at once·bash

lm_eval --model hf \
  --model_args pretrained=Qwen/Qwen2.5-7B-Instruct \
  --tasks mmlu,gsm8k,hellaswag,truthfulqa_mc2 \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path results/qwen2.5-7b-bench.json

Custom task via YAML·yaml

# tasks/my_task.yaml
task: my_qa_task
dataset_path: json
dataset_name: null
dataset_kwargs:
  data_files: ./data/my_qa.jsonl
output_type: generate_until
doc_to_text: 'Question: {{question}}\nAnswer:'
doc_to_target: '{{answer}}'
metric_list:
  - metric: exact_match
    aggregation: mean
    higher_is_better: true

lm-evaluation-harness (EleutherAI)

The standard for academic-style benchmarking

What it covers

What it does NOT do

Code

External links

Exercise

Progress

Comments 0