Loss Functions: The Rulers We Optimize

Loss = "How Wrong Are We?"

Every learning algorithm needs a number that says how badly the model is doing. That number is called loss, and the choice of loss function is one of the most important decisions in ML — it shapes what the model considers "wrong."

Three Workhorses

Loss	Formula	Use when
Mean Squared Error (MSE)	$\frac{1}{n} \sum (y - \overset{y}{^})^{2}$	Regression. Penalizes big errors heavily.
Mean Absolute Error (MAE)	$\frac{1}{n} \sum ∣ y - \overset{y}{^} ∣$	Regression with outliers. Treats all errors linearly.
Cross-Entropy	$- \sum y_{i} lo g \overset{y}{^}_{i}$	Classification. Probabilistically grounded.

MSE vs MAE — Why the Squaring Matters

MSE squares the error, so a residual of 10 contributes 100 while a residual of 1 contributes only 1. The model learns to desperately avoid big errors, even at the cost of being slightly worse on small ones. Good when small errors are forgivable but big ones are catastrophic. Bad when you have outliers (the squaring makes outliers dominate the loss).

MAE treats every error linearly. A residual of 10 contributes 10. The model is more tolerant of outliers but cares less about small refinements.

The loss function tells the model what to care about. Pick it intentionally — it's not a default. MSE for "no big surprises." MAE for "outliers exist and that's OK." Cross-entropy for "this is classification, calibrate probabilities."

Code

MSE vs MAE on outlier-prone data·python

import numpy as np

y_true = np.array([1.0, 2.0, 3.0, 4.0, 100.0])      # last is an outlier
y_pred = np.array([1.1, 1.9, 3.2, 3.8, 5.0])

mse = np.mean((y_true - y_pred) ** 2)
mae = np.mean(np.abs(y_true - y_pred))
print(f"MSE: {mse:.3f}")    # huge — outlier dominates
print(f"MAE: {mae:.3f}")    # large but not as catastrophic

Cross-entropy for classification·python

import torch
import torch.nn.functional as F

logits = torch.tensor([[2.0, 1.0, 0.5]])       # raw scores
target = torch.tensor([0])                      # correct class index

# Cross-entropy loss — built for classification
ce = F.cross_entropy(logits, target)
print(ce.item())                                # ~0.42

Exercise

Take the outlier example. Compute the slope/intercept of a regression line minimizing MSE vs MAE. (Use scipy.optimize.minimize or numpy.linalg.lstsq for MSE, and scipy.stats.linregress won't help — write a custom loss for MAE). Notice how MSE pulls the line toward the outlier; MAE doesn't.

Hint

MSE has a closed-form (least squares). MAE doesn't — use scipy's optimizer to minimize the absolute-error loss.

Loss Functions: The Rulers We Optimize

Loss = "How Wrong Are We?"

Three Workhorses

MSE vs MAE — Why the Squaring Matters

Code

External links

Exercise

Progress

Comments 0