Activation Functions

The activation menu, with reasons

ReLU — max(0, x). Default for hidden layers in CNNs and most MLPs since 2012. Cheap, sparse, gradients are 1 or 0 — easy to backprop. Failure mode: dead neurons (a unit stuck at zero forever).

GELU — Gaussian Error Linear Unit, smoother than ReLU near zero. Default in transformers (BERT, GPT, ViT). Slightly more expensive but trains better at scale. x * Φ(x).

SiLU / Swish — x * sigmoid(x). Used in EfficientNet, LLaMA, modern image-generation models. Smooth like GELU, slightly cheaper to compute.

Sigmoid — squashes to (0,1). Fine as the output activation for binary classification (paired with BCE loss). Bad as a hidden activation in deep networks — gradients saturate, training stalls.

Tanh — squashes to (-1, 1), zero-centered. Used inside RNNs (LSTMs, GRUs). Same saturation problem as sigmoid for deep stacks.

Softmax — turns a vector of logits into a probability distribution. Almost always the last step before cross-entropy loss. PyTorch's CrossEntropyLoss already includes a numerically stable softmax — do not stack them.

Tip: Default to ReLU (or GELU in a transformer). Use sigmoid only as a binary output. Use softmax only as the final classifier. This three-line rule covers 90% of architectures.

Why activations decide trainability

Activations control the gradient flow. ReLU passes gradients of 1 wherever it is active (and 0 elsewhere), which keeps deep stacks trainable. Sigmoid and tanh saturate at the tails, killing gradients. Choosing the wrong activation is the single most common reason a textbook architecture fails to learn anything on a real dataset.

Principle: If your network refuses to train, the first thing to check after the data loader is the activation choice. Replace tanh with ReLU and see what happens.

Code

A tour of activations·python

import torch, torch.nn.functional as F
import matplotlib.pyplot as plt

x = torch.linspace(-5, 5, 200)
acts = {
    "ReLU":    F.relu(x),
    "GELU":    F.gelu(x),
    "SiLU":    F.silu(x),
    "Sigmoid": torch.sigmoid(x),
    "Tanh":    torch.tanh(x),
}
for name, y in acts.items():
    plt.plot(x, y, label=name)
plt.legend(); plt.grid(True); plt.show()

Softmax goes at the very end·python

import torch, torch.nn as nn
logits = torch.randn(4, 10)
probs  = torch.softmax(logits, dim=-1)
print(probs.sum(dim=-1))  # tensor([1., 1., 1., 1.])

# DO NOT do softmax + CrossEntropyLoss. The loss already includes it.
loss = nn.CrossEntropyLoss()(logits, torch.tensor([3, 0, 9, 5]))
print(loss.item())

The activation menu, with reasons

Why activations decide trainability

Code

External links

Exercise

Progress

Comments 0