Activation Functions: The Spark of Non-Linearity

Why Linear Alone Can't Win

Stack 100 linear layers without anything between them and you get... one linear layer. The composition of linear functions is still linear. A 100-layer purely-linear network has the exact same expressive power as a single linear regression. Useless waste of compute.

Activation functions introduce non-linearity. They're the kink in the line that lets the network bend reality into useful shapes. Without them, no curves, no boundaries, no understanding of cats vs dogs.

The Activation Zoo

Activation	Formula	When to use
ReLU	$max (0, x)$	Default in modern deep nets. Cheap, doesn't vanish for positive inputs.
Sigmoid	$1/ (1 + e^{- x})$	Output layer for binary classification. Squeezes to (0, 1) — interpretable as probability.
Tanh	$(e^{x} - e^{- x}) / (e^{x} + e^{- x})$	Squeezes to (-1, 1). Used in older RNNs.
Softmax	$e^{x_{i}} / \sum e^{x_{j}}$	Output layer for multi-class classification. Squeezes to a probability distribution.
GeLU / SiLU	smooth ReLU variants	Modern Transformers, LLMs.

Why ReLU Took Over

Sigmoid's max derivative is 0.25. Stack many and gradients vanish (we saw this in the Calculus track). ReLU's derivative is 1 for positive inputs — gradients flow through unchanged. Plus it's just max(0, x), which is cheap on hardware. The 2012 deep learning revival was largely "ReLU + GPUs + ImageNet."

Activations turn linear stacks into universal function approximators. ReLU is the modern default; pick others only when you have a specific reason.

import torch import torch.nn.functional as F x = torch.linspace(-5, 5, 11) print("input :", x) print("relu :", F.relu(x)) print("sigmoid:", torch.sigmoid(x)) print("tanh :", torch.tanh(x)) print("gelu :", F.gelu(x))

Exercise

Plot ReLU, sigmoid, and GeLU on the same axes for x in [-5, 5]. Notice how ReLU is straight-line at zero, sigmoid is smooth and saturates, GeLU is a smoother ReLU. Pick the right one when you write nn.Linear(...) followed by F.???(x).

Hint

Each activation has a personality. ReLU = sharp gate. Sigmoid = smooth gate that saturates. GeLU = ReLU's polite older cousin used in modern transformers.

Activation Functions: The Spark of Non-Linearity

Why Linear Alone Can't Win

The Activation Zoo

Why ReLU Took Over

Code

External links

Exercise

Progress

Comments 0