Dropout

What dropout does

During training, randomly set a fraction p of activations to zero (and scale the rest by 1/(1-p) to keep the expected value the same). During evaluation, dropout is identity. The net effect is that the model can't rely on any single neuron — it has to spread information across the population. That's a powerful implicit regularizer.

Common rates: p=0.1 for transformers, p=0.2-0.5 for fully-connected layers in CNNs, p=0.0 in convolutional layers (use SpatialDropout2d if you really want it).

Tip: Apply dropout AFTER the activation, not before. nn.Sequential(nn.Linear(...), nn.ReLU(), nn.Dropout(0.2), nn.Linear(...)). The other way around is a classic mistake — it zeros activations before they have a chance to be non-linear.

When dropout helps and when it hurts

Dropout was a major win for the dense MLP era and for the original transformer. With BatchNorm and modern data augmentation, it's a smaller win — and in some vision pipelines, dropout interacts badly with BatchNorm and hurts. Default for transformers; tune for CNNs; off for tabular models that already have other regularization.

Variants worth knowing

SpatialDropout2d drops entire feature maps in CNNs (useful when adjacent channels are correlated). DropPath / Stochastic Depth drops entire residual blocks at random — used in ConvNeXt and ViT for very deep networks. Word/Token dropout drops entire tokens in NLP, sometimes used as a data-augmentation alternative.

Principle: Dropout is a default knob to try, not a default to leave on. Measure with and without on your task; sometimes p=0 wins because BatchNorm and augmentation already cover the gap.

Code

Dropout in the right position·python

import torch.nn as nn

mlp = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(p=0.2),       # after ReLU
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(p=0.2),
    nn.Linear(128, 10),
)
# Eval mode disables dropout automatically.
mlp.train(); print(mlp(x))   # noisy outputs (dropout active)
mlp.eval();  print(mlp(x))   # deterministic outputs

What dropout does

When dropout helps and when it hurts

Variants worth knowing

Code

External links

Exercise

Progress

Comments 0