Loss Functions

Match the loss to the task

The loss is the contract between your data and your optimizer. Pick the wrong one and the model will learn the wrong thing — even if accuracy looks superficially fine.

Multi-class classification — nn.CrossEntropyLoss. Takes raw logits, expects integer class labels.
Multi-label classification — nn.BCEWithLogitsLoss. One sigmoid per label, BCE per label.
Binary classification — nn.BCEWithLogitsLoss with a single output.
Regression (squared error) — nn.MSELoss. Sensitive to outliers.
Regression (robust) — nn.SmoothL1Loss or nn.HuberLoss.
Ranking / similarity — triplet loss, contrastive loss, InfoNCE.
Sequence generation — token-level cross-entropy with optional label smoothing.

Tip: If accuracy on a validation split looks fine but the model behaves weirdly on edge cases, suspect the loss before suspecting the model. The loss is what training optimizes; accuracy is just one report on it.

Cross-entropy is the workhorse

Cross-entropy loss for a single example with logits z and true class c is -log(softmax(z)[c]). It pushes the logit of the right class up and the logits of the wrong classes down, and the gradient is beautifully clean: softmax(z) - one_hot(c).

For balanced classes, plain cross-entropy works. For imbalanced classes, you can pass weight= per class, or use focal loss (down-weights easy examples). For very large vocabularies (LLMs), you can sample subsets of negatives instead of computing the full softmax.

Principle: Read the docstring of every loss you use. The 'reduction' argument (mean, sum, none) is the source of half the silent metric bugs in junior code.

Class weights, label smoothing, and friends

For imbalanced data, pass weight=class_weights. For overconfident models, use label_smoothing=0.1 to soften targets. For multi-label tasks, BCE per label gives you the right shape. These knobs sit on top of the basic loss and rarely change the choice, only the calibration.

Code

Loss functions for the four common shapes·python

import torch, torch.nn as nn

logits_mc = torch.randn(8, 10)
labels_mc = torch.randint(0, 10, (8,))
print(nn.CrossEntropyLoss()(logits_mc, labels_mc).item())

weights = torch.tensor([2.0]*5 + [1.0]*5)
loss_mc_balanced = nn.CrossEntropyLoss(weight=weights, label_smoothing=0.1)
print(loss_mc_balanced(logits_mc, labels_mc).item())

logits_bc = torch.randn(8, 1)
labels_bc = torch.randint(0, 2, (8, 1)).float()
print(nn.BCEWithLogitsLoss()(logits_bc, labels_bc).item())

preds  = torch.randn(8, 1)
target = torch.randn(8, 1)
print(nn.MSELoss()(preds, target).item())
print(nn.SmoothL1Loss()(preds, target).item())

Match the loss to the task

Cross-entropy is the workhorse

Class weights, label smoothing, and friends

Code

External links

Exercise

Progress

Comments 0