Imbalanced Data — Sampler, Class Weights, Focal Loss

Three orthogonal strategies, often combined

Most real-world classification problems have skewed class distributions: 99% normal transactions, 1% fraud. Train naively and the model learns "always predict normal" — perfect accuracy, zero utility. Three tools, in increasing aggressiveness:

Weighted sampling — change which samples the DataLoader picks. Oversample rare classes; the model sees them more often.
Class weights in the loss — keep sampling proportional but make rare-class mistakes count more. nn.CrossEntropyLoss(weight=...).
Focal loss — for extreme imbalance (1:1000+), modify the loss to down-weight easy examples. Originally from RetinaNet.

These compose freely. Start with class weights (easiest, fewest hyperparameters); add weighted sampling if class weights aren't enough; reach for focal loss for truly extreme cases.

The trap to avoid

If you oversample rare classes AND apply class weights, you double-weight them. Pick one or carefully tune both. Same logic applies to focal loss — its α parameter is itself a class weight.

Code

WeightedRandomSampler — oversample rare classes·python

import torch
from torch.utils.data import DataLoader, WeightedRandomSampler

# Suppose 1000 class-A samples, 100 class-B samples, 50 class-C samples
labels = torch.cat([
    torch.zeros(1000, dtype=torch.long),
    torch.ones(100, dtype=torch.long),
    torch.full((50,), 2, dtype=torch.long),
])

class_counts = torch.tensor([1000., 100., 50.])
weights_per_class = 1.0 / class_counts
sample_weights = weights_per_class[labels]

sampler = WeightedRandomSampler(
    weights=sample_weights,
    num_samples=len(sample_weights),
    replacement=True,                  # required for over-sampling rare classes
)

loader = DataLoader(dataset, batch_size=32, sampler=sampler)
# Each batch now sees roughly equal counts of A, B, C

Class-weighted CrossEntropyLoss·python

import torch
import torch.nn as nn

# Inverse-frequency weighting: rare classes get bigger weight
class_counts = torch.tensor([1000., 100., 50.])
weights = 1.0 / class_counts
weights = weights / weights.sum() * len(class_counts)  # normalize to mean 1

criterion = nn.CrossEntropyLoss(weight=weights.to(device))

# Same training loop as usual; the loss value is now class-aware

Focal loss — for extreme imbalance·python

import torch
import torch.nn as nn
import torch.nn.functional as F

class FocalLoss(nn.Module):
    """Down-weights well-classified (easy) examples."""
    def __init__(self, alpha=1.0, gamma=2.0, reduction='mean'):
        super().__init__()
        self.alpha = alpha
        self.gamma = gamma
        self.reduction = reduction

    def forward(self, logits, targets):
        ce = F.cross_entropy(logits, targets, reduction='none')
        pt = torch.exp(-ce)            # prob of correct class
        focal = self.alpha * (1 - pt) ** self.gamma * ce
        if self.reduction == 'mean':
            return focal.mean()
        if self.reduction == 'sum':
            return focal.sum()
        return focal

# Use just like CrossEntropyLoss
criterion = FocalLoss(alpha=0.25, gamma=2.0)

Imbalanced Data — Sampler, Class Weights, Focal Loss

Three orthogonal strategies, often combined

The trap to avoid

Code

External links

Exercise

Progress

Comments 0