Gradient Clipping — Taming Exploding Gradients

When backward overshoots, clip it

Some training setups produce gradients with huge norms — RNNs over long sequences, very deep networks without proper normalization, GANs, transformer training without warmup. Huge gradients × any learning rate produce huge updates that destabilize training (NaN losses, sudden divergence).

Gradient clipping bounds the magnitude of gradients before the optimizer step. Two common variants:

Clip by global norm — compute the L2 norm of all gradients combined, scale them down if it exceeds a threshold. Most common in modern training (Transformers, LLM fine-tuning, etc.).
Clip by value — clamp each gradient element to [-c, c]. Cruder, but useful when you have outlier elements rather than overall large norms.

Where it goes in the loop

Always between backward() and optimizer.step(). Clipping before backward doesn't make sense (no gradients yet); clipping after step is too late.

A typical max_norm value is 1.0 for transformer training, sometimes 5.0 for RNNs. The right value is empirical — log the actual gradient norm during training and you'll see whether you're hitting the cap often.

Code

Clip by global norm — the standard recipe·python

import torch
import torch.nn as nn

model = nn.Sequential(nn.Linear(100, 50), nn.ReLU(), nn.Linear(50, 10))
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
loss_fn = nn.CrossEntropyLoss()

x = torch.randn(32, 100)
y = torch.randint(0, 10, (32,))

optimizer.zero_grad()
loss = loss_fn(model(x), y)
loss.backward()

# Clip before step — returns the original (pre-clip) total norm
total_norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0)
print(f"pre-clip norm: {total_norm:.4f}")

optimizer.step()

Clip by value — cruder, sometimes useful·python

import torch
import torch.nn as nn

model = nn.Linear(10, 2)
loss = nn.functional.mse_loss(model(torch.randn(4, 10)), torch.randn(4, 2))
loss.backward()

# Clamps each element of every gradient tensor to [-0.5, 0.5]
torch.nn.utils.clip_grad_value_(model.parameters(), clip_value=0.5)

Logging gradient norms — diagnostic, no cap·python

import torch
import torch.nn as nn

model = nn.Linear(10, 2)
loss = nn.functional.mse_loss(model(torch.randn(4, 10)), torch.randn(4, 2))
loss.backward()

# Compute the global norm WITHOUT clipping (pass max_norm=inf)
norm = torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=float('inf'))
print(f"global grad norm: {norm:.4f}")

# In a training loop, log this every N steps. If it consistently exceeds
# your max_norm, your clip threshold is doing real work — that's a signal
# to investigate (lr too high? warmup needed? mixed-precision underflow?).

Gradient Clipping — Taming Exploding Gradients

When backward overshoots, clip it

Where it goes in the loop

Code

External links

Exercise

Progress

Comments 0