When the Chain Rule Bites: Vanishing & Exploding Gradients

How Depth Changes Gradient Signals

The chain rule multiplies local derivatives—or, for vector-valued layers, Jacobians—along paths through a computation graph. If relevant magnitudes repeatedly fall below 1, gradients reaching early layers can shrink; if they repeatedly exceed 1, gradients can grow and destabilize training. Sigmoid saturation is a classic source of shrinking derivatives because its derivative is at most 0.25.

Real networks are more complex than a product of scalar derivatives. Weight matrices, activation distributions, Jacobian directions, and singular values all interact. The core intuition remains: repeated composition can attenuate or amplify learning signals.

A Design Toolkit, Not One Antidote

Activations: ReLU avoids positive-side saturation but has zero derivative for negative inputs and can create dead units. GELU and SiLU are also widely used.
Residual connections: identity paths give gradients routes around several transformations.
Initialization: Xavier and He schemes aim to keep activation and gradient scales reasonable at the start of training.
Normalization: BatchNorm or LayerNorm can improve scale and optimization geometry, but neither guarantees that vanishing gradients disappear.
Gradient clipping: limits exploding updates; it does not restore a vanished signal.

The 2012 transition had no single cause. ReLU, GPU computation, large labeled datasets, convolutional architecture, initialization, optimization, and regularization worked together. Reducing the result to “ReLU revived deep learning” hides the interaction.

Track Reward

Derivatives describe local change, and the chain rule carries that change through a computation graph. In a deep model, signals can weaken or explode. Backpropagation reuses the local derivatives efficiently; architecture and optimization determine whether those derivatives remain useful.

Code

Watching gradients vanish·python

import torch

# Vanishing gradient demo with sigmoids
x = torch.tensor(1.0, requires_grad=True)
y = x
for _ in range(20):
    y = torch.sigmoid(y)         # 20 sigmoids in a row
y.backward()
print(f"dy/dx after 20 sigmoids: {x.grad.item():.2e}")
# A very tiny number — gradient has nearly vanished

Exercise

Repeat the demo with torch.relu, first from a positive input and then from a negative input. Compare the gradients with the sigmoid chain.

Hint

A positive ReLU chain can preserve a derivative of 1, while a negative input produces a derivative of 0 immediately. This demonstrates both the benefit and the dead-unit failure mode.

How Depth Changes Gradient Signals

A Design Toolkit, Not One Antidote

Track Reward

Code

External links

Exercise

Progress

Comments 0