Learning Rate Schedules

The learning rate is the most important hyperparameter

If your learning rate is too small, training crawls. If it is too large, training diverges. The right value depends on the optimizer, the model, the batch size, the dataset, and the stage of training. There is no universal rule, but there are patterns that work.

For most modern training, the standard recipe is warmup + cosine decay: linearly ramp the learning rate from 0 to lr_max over the first few hundred to few thousand steps, then decay smoothly to ~0 over the rest of training. This combination handles the unstable early training and the diminishing-returns late training cleanly.

Tip: If you train transformers without warmup, you will see the loss either explode or stall in the first hundred steps. Warmup is non-optional for attention-based models above a certain size.

Common schedules in your toolbox

Constant — fine for short training and prototyping; rarely the final choice.
Step decay — multiply lr by 0.1 every N epochs. Classic for ResNet-style training; less common today.
Cosine annealing — smooth decay shaped like the second half of a cosine. Works almost everywhere.
Cosine with warm restarts — useful for very long training runs where you want a periodic refresh.
OneCycle — Smith's recipe: ramp up, then ramp down, with peak in the middle. Strong for short fine-tuning runs.

How to find a good `lr_max`

The "LR finder" trick (Smith, 2017): train for a few hundred steps with the lr increasing exponentially from 1e-7 to 10. Plot loss vs lr. Pick the lr value where the loss is descending fastest, then divide by 2–4. This usually beats blind hyperparameter search.

Principle: Pick the schedule first based on training duration; pick lr_max with the LR finder; then leave both alone and tune everything else. Adjusting the schedule mid-training is a recipe for confusion.

Code

Warmup + cosine schedule·python

import torch, math
from torch.optim.lr_scheduler import LambdaLR

def warmup_cosine(optimizer, warmup_steps, total_steps, min_ratio=0.1):
    def fn(step):
        if step < warmup_steps:
            return step / max(1, warmup_steps)
        progress = (step - warmup_steps) / max(1, total_steps - warmup_steps)
        cosine = 0.5 * (1 + math.cos(math.pi * progress))
        return min_ratio + (1 - min_ratio) * cosine
    return LambdaLR(optimizer, fn)

opt = torch.optim.AdamW(model.parameters(), lr=3e-4)
sch = warmup_cosine(opt, warmup_steps=500, total_steps=20_000)

for step in range(20_000):
    opt.zero_grad()
    loss = compute_loss(model, batch())
    loss.backward(); opt.step(); sch.step()

The learning rate is the most important hyperparameter

Common schedules in your toolbox

How to find a good `lr_max`

Code

External links

Exercise

Progress

Comments 0

The learning rate is the most important hyperparameter

Common schedules in your toolbox

How to find a good lr_max

Code

External links

Exercise

Progress

How to find a good `lr_max`