The Chain Rule: How Derivatives Compose

The Most Important Calculus Rule for AI

You almost never see a function in isolation in ML. Most are compositions: $f (g (x))$ where one function feeds into another. To differentiate $f (g (x))$ , you need the chain rule:

$\frac{d}{d x} f (g (x)) = f^{'} (g (x)) \cdot g^{'} (x)$

In words: "the derivative of the outer function (evaluated at the inner) times the derivative of the inner function." Multiply, don't add. The slopes compose multiplicatively as you nest functions.

Why It's the Backbone of Backprop

A neural network is a massive composition of functions: input → linear layer → activation → linear layer → activation → ... → loss. To compute how a loss changes with respect to a weight in the very first layer, you have to chain derivatives across every layer in between. The chain rule is the bookkeeping that makes this possible.

Backpropagation (next track) is just the chain rule applied systematically, one layer at a time, with intermediate values cached during the forward pass.

Quick Examples

$\frac{d}{d x} sin (x^{2}) = cos (x^{2}) \cdot 2 x$
$\frac{d}{d x} e^{3 x} = e^{3 x} \cdot 3$
$\frac{d}{d x} lo g (σ (x)) = \frac{1}{σ ( x )} \cdot σ^{'} (x)$ where $σ^{'} (x) = σ (x) (1 - σ (x))$ for the sigmoid.

The chain rule says derivatives multiply across compositions. Stack 100 layers, the gradient at the input is a product of 100 local derivatives. This is why deep networks suffer from vanishing/exploding gradients — and why ReLU, batch norm, and residual connections were invented.

import torch x = torch.tensor(2.0, requires_grad=True) # y = sin(x^2) — a composition y = torch.sin(x ** 2) y.backward() # By chain rule: dy/dx = cos(x^2) * 2x = cos(4) * 4 ≈ -2.614 print(x.grad.item()) # -2.614...

The Chain Rule: How Derivatives Compose

The Most Important Calculus Rule for AI

Why It's the Backbone of Backprop

Quick Examples

Code

External links

Exercise

Progress

Comments 0