Local Change and Accumulated Change
Knowing how far a car traveled does not immediately tell you its speed now; you need the rate of change of position. Conversely, knowing its speed over time lets you accumulate distance. Differentiation and integration connect those questions.
- Differentiation measures how sensitively an output changes when an input changes locally.
- Integration accumulates a rate or density over an interval.
In learning, describes how the loss changes when parameter moves slightly. Collecting these partial derivatives produces a gradient vector that an optimizer can use.
What Automatic Differentiation Does—and Does Not Do
Frameworks such as PyTorch record a computation graph and apply the chain rule, avoiding long symbolic derivations by hand. They do not decide whether the objective is appropriate, whether a nondifferentiable operation broke the graph, or whether the resulting gradient is a useful learning signal.
You still need to reason about sign and scale. If loss rises instead of falling, inspect the update sign; if gradients are zero, inspect saturation, dead activations, detachment, and graph structure.