Metrics During Training

The metrics you must always log

Bare minimum: training loss per step, validation loss per epoch, validation accuracy (or task-appropriate metric) per epoch, learning rate per step. With these four, you can debug almost anything.

Recommended additions: gradient norm per step (catches exploding gradients), per-class accuracy (catches imbalance), throughput in examples/sec (catches data-loading regressions).

Tip: Plot the metrics. Numbers in a log file are forensic; plots are diagnostic. matplotlib in a Jupyter cell is fine for a one-off; W&B or TensorBoard for serious work.

Epoch averages vs running averages

Training loss is best plotted as a smoothed running average — single-step loss is too noisy to read. 0.99 * old + 0.01 * new is a fine default. Validation loss is once per epoch and you plot the raw value.

Don't trust accuracy alone

Accuracy hides imbalance, calibration, and edge-case failures. Always look at: precision/recall per class for classification; MAE/RMSE/quantile errors for regression; BLEU/ROUGE/exact-match for generation; the confusion matrix on the validation set after each epoch.

Principle: If you only watch one number, you'll get surprised by all the others. The first time a model 'looks fine on accuracy' but is broken on a minority class, it costs you a week of trust with the team.

Code

Per-step logging with a smoothed train loss·python

smooth_loss = None
for step, (xb, yb) in enumerate(train_loader):
    opt.zero_grad()
    loss = loss_fn(model(xb.to(device)), yb.to(device))
    loss.backward(); opt.step()

    val = loss.item()
    smooth_loss = val if smooth_loss is None else 0.99 * smooth_loss + 0.01 * val
    if step % 50 == 0:
        print(f"step {step:6d}  loss {val:.4f}  smooth {smooth_loss:.4f}  lr {opt.param_groups[0]['lr']:.2e}")

Per-class accuracy·python

import torch

@torch.inference_mode()
def per_class_accuracy(model, loader, num_classes, device):
    model.eval()
    correct = torch.zeros(num_classes); total = torch.zeros(num_classes)
    for xb, yb in loader:
        xb, yb = xb.to(device), yb.to(device)
        preds = model(xb).argmax(dim=-1)
        for c in range(num_classes):
            mask = (yb == c)
            correct[c] += ((preds == yb) & mask).sum().cpu()
            total[c]   += mask.sum().cpu()
    return (correct / total.clamp(min=1)).tolist()

The metrics you must always log

Epoch averages vs running averages

Don't trust accuracy alone

Code

External links

Exercise

Progress

Comments 0