Activation Functions: The Spark of Non-Linearity

Why Affine Layers Need Nonlinearity

A composition of affine maps is still one affine map. Adding an elementwise nonlinear activation between layers lets a network represent curved boundaries and richer functions. The usefulness comes from the composition, not from an activation in isolation.

Common Activation Choices

Activation	Range or behavior	Typical role
ReLU	$max (0, x)$	Cheap and common in convolutional and feed-forward networks; zero gradient on the negative side can create dead units.
GELU	Smooth, input-dependent gate	Common in Transformer feed-forward blocks.
SiLU	$x σ (x)$	Common in modern vision and language architectures.
Sigmoid	(0,1)	Binary probabilities or gates when paired with an appropriate objective; it saturates at large magnitudes.
Tanh	(-1,1)	Centered gates and recurrent-state transformations; it also saturates.
Softmax	A normalized vector	Converts a vector of logits into a categorical distribution; it is not an elementwise hidden-layer activation.

Probability-shaped outputs are not automatically calibrated. The activation defines a mathematical range or normalization, while calibration is an empirical property of the trained system.

No Single Modern Default

ReLU helped make deep convolutional networks practical because it is cheap and avoids positive-side saturation. GELU and SiLU are now common in Transformers and other architectures. The 2012 ImageNet breakthrough combined architecture, GPUs, data, ReLU, optimization, augmentation, and regularization.

Choose an activation as part of an architecture. Expressivity, gradient flow, hardware cost, output semantics, and empirical behavior all matter.

import torch import torch.nn.functional as F x = torch.linspace(-5, 5, 11) print("input :", x) print("relu :", F.relu(x)) print("sigmoid:", torch.sigmoid(x)) print("tanh :", torch.tanh(x)) print("gelu :", F.gelu(x))

Exercise

Plot ReLU, sigmoid, and GeLU on the same axes for x in [-5, 5]. Notice how ReLU is straight-line at zero, sigmoid is smooth and saturates, GeLU is a smoother ReLU. Pick the right one when you write nn.Linear(...) followed by F.???(x).

Hint

Each activation has a personality. ReLU = sharp gate. Sigmoid = smooth gate that saturates. GeLU = ReLU's polite older cousin used in modern transformers.

Why Affine Layers Need Nonlinearity

Common Activation Choices

No Single Modern Default

Code

External links

Exercise

Progress

Comments 0