Metrics and Evaluation

Loss is the gradient signal; metrics are the report card

You train against the loss. You evaluate against task-specific metrics. They're not the same number, and the gap matters.

Classification metrics

Accuracy — fraction correctly predicted. Easy to interpret, useless when classes are imbalanced.
Precision / Recall / F1 — per-class metrics that decompose accuracy into "of what I predicted positive, how much was actually positive (precision)" and "of all the actual positives, how many did I catch (recall)". F1 is their harmonic mean.
Top-K accuracy — fraction where the true class is in your top K predictions. Standard for ImageNet (top-1, top-5).
ROC AUC — area under the receiver operating characteristic. For binary classifiers; threshold-independent.

Regression metrics

MAE / RMSE — mean absolute error and root mean squared error. Both in the units of your target.
R² — fraction of variance explained.

Compute metrics on validation only, in eval mode, with inference_mode

You don't want metric computation polluting training-mode behavior (BatchNorm, Dropout) or building autograd graphs you don't need.

torchmetrics — when you don't want to roll your own

The torchmetrics package (separate install) has tested implementations of every common metric, including streaming variants that update incrementally over batches. For anything beyond accuracy, lean on it.

Code

Accuracy and per-class precision/recall — by hand·python

import torch

def accuracy(logits, targets):
    return (logits.argmax(-1) == targets).float().mean().item()

def precision_recall_f1(logits, targets, num_classes):
    preds = logits.argmax(-1)
    out = {}
    for c in range(num_classes):
        tp = ((preds == c) & (targets == c)).sum().item()
        fp = ((preds == c) & (targets != c)).sum().item()
        fn = ((preds != c) & (targets == c)).sum().item()
        prec = tp / (tp + fp) if (tp + fp) else 0.0
        rec  = tp / (tp + fn) if (tp + fn) else 0.0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) else 0.0
        out[c] = {'precision': prec, 'recall': rec, 'f1': f1}
    return out

logits = torch.randn(64, 5)
targets = torch.randint(0, 5, (64,))
print(f"acc: {accuracy(logits, targets):.3f}")
print(precision_recall_f1(logits, targets, 5))

The full evaluation function·python

import torch
import torch.nn as nn

def evaluate(model, val_loader, criterion, device):
    model.eval()
    total_loss = 0.0
    total_correct = 0
    total_samples = 0

    with torch.inference_mode():
        for x, y in val_loader:
            x, y = x.to(device), y.to(device)
            out = model(x)
            loss = criterion(out, y)

            total_loss += loss.item() * x.size(0)
            total_correct += (out.argmax(-1) == y).sum().item()
            total_samples += x.size(0)

    return total_loss / total_samples, total_correct / total_samples

# Usage
val_loss, val_acc = evaluate(model, val_loader, criterion, device)
print(f"val_loss={val_loss:.4f}  val_acc={val_acc:.4f}")

torchmetrics — when you want it tested for you·python

# pip install torchmetrics
import torch
from torchmetrics.classification import (
    MulticlassAccuracy, MulticlassF1Score, MulticlassConfusionMatrix
)

n_classes = 10
acc = MulticlassAccuracy(num_classes=n_classes, top_k=1)
acc5 = MulticlassAccuracy(num_classes=n_classes, top_k=5)
f1 = MulticlassF1Score(num_classes=n_classes, average='macro')
cm = MulticlassConfusionMatrix(num_classes=n_classes)

# Accumulate across batches
for x, y in val_loader:
    logits = model(x)
    acc.update(logits, y)
    acc5.update(logits, y)
    f1.update(logits, y)
    cm.update(logits, y)

print(f"top-1: {acc.compute():.4f}")
print(f"top-5: {acc5.compute():.4f}")
print(f"macro-F1: {f1.compute():.4f}")
print("Confusion matrix:")
print(cm.compute())

Loss is the gradient signal; metrics are the report card

Classification metrics

Regression metrics

Compute metrics on validation only, in eval mode, with inference_mode

torchmetrics — when you don't want to roll your own

Code

External links

Exercise

Progress

Comments 0