Learning Rate & Scheduling

~20 min · learning-rate, scheduler, cosine, warmup

Level 0Observer

0 XP0/43 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

The single most important hyperparameter

Learning rate controls how much weights change with each update. Too high → unstable, loss spikes. Too low → slow convergence or stuck.

Method	Range	Good starting point
Full fine-tuning	1e-6 to 5e-5	2e-5
LoRA / QLoRA	1e-5 to 5e-4	2e-4
OpenAI managed	0.1–10 (multiplier)	"auto" (~1.0)

Scheduling — never use a constant LR

Cosine — gradually decreases following a cosine curve. Most popular for fine-tuning.
Linear — decreases linearly from peak to zero.
Warmup — start tiny, ramp up to peak over first N% of steps, then decay.

The standard recipe: cosine schedule with 10% warmup ratio. Works well for almost every fine-tuning project.

Code

Cosine schedule with warmup·python

from trl import SFTConfig

args = SFTConfig(
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_ratio=0.1,           # warm up over first 10% of total steps
    # OR: warmup_steps=100,    # exactly 100 warmup steps
)

External links

Exercise

Run two short LoRA training runs on the same dataset: one at LR=2e-4 with cosine+warmup, one at LR=2e-5 (full FT default). Compare the loss curves. The 2e-5 LoRA run should learn much more slowly. Internalize how dramatic the LR difference is.

Progress

Progress is local-only — sign in to sync across devices.

← 🤗 Hugging Face StackQuiz · 5 questions Next →Batch Size & Gradient Accumulation

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.