Evaluation Metrics and the evaluate Library

~24 min · training, eval, metrics

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The compute_metrics signature

Trainer calls compute_metrics(pred) at every eval_steps. pred.predictions is the raw logits/predictions; pred.label_ids is the gold. You return a dict; Trainer logs and tracks it.

The evaluate library

HF's evaluate library wraps standard metrics: accuracy, F1, BLEU, ROUGE, METEOR, BERTScore, perplexity, exact_match, and dozens more. Each is loaded from the Hub like a dataset: evaluate.load("accuracy"). The metric object exposes compute() for a one-shot eval and add_batch() for streaming aggregation.

Pick metrics that match your loss surface

Accuracy is for balanced classification. F1 is the right call on imbalanced classes. BLEU/ROUGE for translation/summarization but they're correlated weakly with human judgment for short responses — pair with semantic-similarity metrics (BERTScore, embedding cosine) for chat eval.

Code

compute_metrics with evaluate·python

import evaluate
import numpy as np

f1 = evaluate.load("f1")
acc = evaluate.load("accuracy")

def metrics(pred):
    preds = pred.predictions.argmax(-1)
    return {
        "accuracy": acc.compute(predictions=preds, references=pred.label_ids)["accuracy"],
        "f1_macro": f1.compute(predictions=preds, references=pred.label_ids, average="macro")["f1"],
    }

Streaming aggregation for big eval sets·python

import evaluate
acc = evaluate.load("accuracy")

# Imagine a 100k eval set; we don't want to materialize all preds at once.
for batch in eval_loader:
    with torch.no_grad():
        preds = model(**batch).logits.argmax(-1)
    acc.add_batch(predictions=preds.cpu().numpy(), references=batch["labels"].cpu().numpy())

print(acc.compute())

External links

Exercise

Add compute_metrics to your IMDB run from lesson 0. Track accuracy, F1 macro, F1 weighted. Verify TensorBoard shows all three. Inspect: which is the highest at the end? Which fluctuates most? Which is most predictive of human-perceived quality on a few sample reviews?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousData Collators: Padding and Beyond Next →Mixed Precision, Gradient Accumulation, and Memory

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.