When the Chain Rule Bites: Vanishing & Exploding Gradients

The Hidden Cost of Deep

By the chain rule, the gradient at the input of a 100-layer network is the product of 100 local derivatives. If those derivatives are systematically less than 1 (like the sigmoid's max derivative of 0.25), the product shrinks rapidly: $0.2 5^{100} \approx 6 \times 1 0^{- 61}$ . The gradient becomes effectively zero — the network won't learn the early layers at all. This is the vanishing gradient problem.

The opposite also happens: if local derivatives are greater than 1, the product blows up. Exploding gradients destabilize training, sometimes producing NaNs.

The Fixes

ReLU instead of sigmoid/tanh — derivative is 1 (for positive inputs) instead of fractional. Doesn't shrink the chain product.
Residual connections (ResNet) — add the input back, so gradients can flow around layers via the skip path.
Batch normalization — keeps activations in a sane range so derivatives don't drift.
Gradient clipping — cap the gradient norm to a threshold to prevent explosions.
Better initialization (Xavier, He) — start with weights that don't immediately cause vanish/explode.

Why deep nets didn't work for decades. Sigmoid and tanh activations dominated early neural networks; their derivatives are bounded by 0.25 and 1 respectively. Stack many layers and gradients vanished. The 2012 ImageNet revolution wasn't about new ideas — it was about ReLU, GPU compute, and bigger data finally letting depth pay off.

Track Reward

You now read calculus the way it shows up in AI: derivatives are slopes that tell training where to go; integration is occasional; the chain rule is the spine of every neural network. Vanishing and exploding gradients are the chain rule biting. Next: backprop, the disciplined application of all of this.

Code

Watching gradients vanish·python

import torch

# Vanishing gradient demo with sigmoids
x = torch.tensor(1.0, requires_grad=True)
y = x
for _ in range(20):
    y = torch.sigmoid(y)         # 20 sigmoids in a row
y.backward()
print(f"dy/dx after 20 sigmoids: {x.grad.item():.2e}")
# A very tiny number — gradient has nearly vanished

When the Chain Rule Bites: Vanishing & Exploding Gradients

The Hidden Cost of Deep

The Fixes

Track Reward

Code

External links

Exercise

Progress

Comments 0