The Derivative Is the Slope at a Point

Zooming In

Take a curve. Pick a point on it. Now zoom in. Zoom more. Eventually, the curve looks like a straight line — its tangent line. The slope of that tangent is the derivative at that point.

Formally: $f^{'} (x) = lim_{h \to 0} \frac{f ( x + h ) - f ( x )}{h}$ . The "rise over run" as the run shrinks to nothing.

Why It's the Most Useful Quantity in AI

Loss landscapes are curves (well, surfaces in many dimensions, but locally curves along any direction). At a given point in weight space, the derivative of the loss with respect to a weight tells you: "how much will the loss change if I nudge this weight a tiny bit?"

That's the steering wheel of training. Without it, you'd be wandering randomly. With it, you can confidently take the smallest, most useful step.

Common Derivative Shapes

Function	Derivative
$f (x) = c$ (constant)	$0$
$f (x) = x$	$1$
$f (x) = x^{2}$	$2 x$
$f (x) = e^{x}$	$e^{x}$ (yes, itself)
$f (x) = lo g (x)$	$1/ x$
$f (x) = sin (x)$	$cos (x)$

You don't need to memorize this — autograd does the work. But knowing the shape helps you sanity-check unexpected results.

The derivative is the slope at a point. In AI, it's the answer to "which direction should I nudge this parameter to reduce the loss?"

Code

Numerical and symbolic, agreeing·python

import numpy as np
import torch

# Numerical derivative — approximate by zooming in
def numerical_derivative(f, x, h=1e-6):
    return (f(x + h) - f(x)) / h

f = lambda x: x ** 2
print(numerical_derivative(f, 3.0))   # ~6.0  (should be 2*3 = 6)

# Symbolic via autograd
x = torch.tensor(3.0, requires_grad=True)
(x ** 2).backward()
print(x.grad.item())                   # 6.0

Exercise

Write a Python function that computes the numerical derivative of any function f at any point x using (f(x+h) - f(x-h)) / (2h) (the symmetric form, more accurate). Test with f(x) = sin(x) at x = 0 — should be ≈ 1 (since cos(0) = 1).

Hint

Symmetric form is more accurate than forward differences for smooth functions. h = 1e-5 or so works well; smaller can introduce float precision noise.

The Derivative Is the Slope at a Point

Zooming In

Why It's the Most Useful Quantity in AI

Common Derivative Shapes

Code

External links

Exercise

Progress

Comments 0