Loss Functions: The Rulers We Optimize

A Loss Encodes What Counts as Error

A loss function turns a prediction and target into an optimization signal. Its choice expresses assumptions about noise, error cost, class structure, and what tradeoffs the model should make.

Three Common Losses

Loss	Form	Typical use
Mean squared error	$\frac{1}{n} \sum (y - \overset{y}{^})^{2}$	Regression when large residuals should receive quadratic weight; connected to Gaussian-noise likelihood.
Mean absolute error	$\frac{1}{n} \sum ∣ y - \overset{y}{^} ∣$	Regression requiring more robustness to large residuals; connected to Laplace-noise likelihood.
Cross-entropy	$- \sum_{i} y_{i} lo g p_{i}$	Classification or next-token prediction with probabilistic targets.

Squaring Changes the Influence of Residuals

Under MSE, a residual of 10 contributes 100 while a residual of 1 contributes 1, so large residuals can dominate the fit. MAE grows linearly and is less sensitive to extreme residuals, although it has a nondifferentiable point at zero and answers a different statistical question.

Cross-entropy rewards probability assigned to the observed class, but minimizing it does not guarantee calibrated probabilities. Calibration depends on data, model specification, regularization, distribution shift, and sometimes post-hoc methods such as temperature scaling.

The loss tells the optimizer what to improve. Choose it for the task and assumptions; then evaluate separate properties such as calibration with separate diagnostics.

Code

MSE vs MAE on outlier-prone data·python

import numpy as np

y_true = np.array([1.0, 2.0, 3.0, 4.0, 100.0])      # last is an outlier
y_pred = np.array([1.1, 1.9, 3.2, 3.8, 5.0])

mse = np.mean((y_true - y_pred) ** 2)
mae = np.mean(np.abs(y_true - y_pred))
print(f"MSE: {mse:.3f}")    # huge — outlier dominates
print(f"MAE: {mae:.3f}")    # large but not as catastrophic

Cross-entropy for classification·python

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.5]])       # raw scores
target = torch.tensor([0])                      # correct class index

# Cross-entropy loss — built for classification
ce = F.cross_entropy(logits, target)
print(ce.item())                                # ~0.42

Exercise

Take the outlier example. Compute the slope/intercept of a regression line minimizing MSE vs MAE. (Use scipy.optimize.minimize or numpy.linalg.lstsq for MSE, and scipy.stats.linregress won't help — write a custom loss for MAE). Notice how MSE pulls the line toward the outlier; MAE doesn't.

Hint

MSE has a closed-form (least squares). MAE doesn't — use scipy's optimizer to minimize the absolute-error loss.

A Loss Encodes What Counts as Error

Three Common Losses

Squaring Changes the Influence of Residuals

Code

External links

Exercise

Progress

Comments 0