Dropout and Regularization (and Why It's Out of Fashion)

~8 min · dropout, regularization

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Dropout randomly zeroes a fraction of activations during training, forcing the network to learn redundant representations rather than rely on any single unit. The original Transformer used dropout 0.1 (base) and 0.3 (big), applied at attention weights, sublayer outputs, and embeddings.

Modern large pretrained models often use no dropout at all during pretraining, or very small values (0.0–0.05). The reasoning is regime-dependent. Pre-2020 models trained for many epochs over relatively small datasets — the risk of overfitting was real, dropout helped. Modern frontier models train for ~1 epoch over trillions of tokens — each example is seen once, overfitting is structurally impossible, and dropout just slows down learning by introducing noise.

Where dropout is still used in 2026: fine-tuning small datasets on smaller bases, distillation runs, dropout-based methods (e.g., MC Dropout) for uncertainty estimation. For pretraining at scale, the field has moved on.

Code

When you might still want dropout·python

# Pretraining at trillion-token scale: skip dropout entirely.
# Fine-tuning a small classifier on 10k examples: dropout = 0.1 still helps.
# LoRA adapter on a large base: dropout in the adapter layers is common (0.05-0.1).

class FineTuneHead(nn.Module):
    def __init__(self, d_model, n_classes, dropout=0.1):
        super().__init__()
        self.dropout = nn.Dropout(dropout)
        self.head = nn.Linear(d_model, n_classes)
    def forward(self, x):
        return self.head(self.dropout(x))

External links

nanoGPT — note the dropout=0.0 default

Exercise

Train two copies of the same small Transformer on a 10M-token corpus, dropout=0.0 vs dropout=0.1. Which converges faster? Which generalizes better to held-out data? Now repeat with a 100K-token corpus. Does the answer change?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousThe Four Numbers: d_model, d_ff, n_heads, n_layers Next →The Output Head: From Hidden States to Logits

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.