Activations: ReLU, GELU, SiLU, and Friends

Non-linearity is what makes the network powerful

Without a non-linear activation between layers, stacking linears just composes into one bigger linear — you'd have one trainable matrix, no matter how many nn.Linear calls. Activations are what let networks learn arbitrary functions.

The shortlist you'll actually use

ReLU — max(0, x). The default for hidden layers in CNNs and pre-Transformer MLPs. Cheap, effective, but suffers from "dying ReLU" when too many neurons hit zero permanently.
GELU — Gaussian Error Linear Unit. The default in modern Transformers (BERT, GPT, ViT). Smooth, slightly slower than ReLU, empirically better for attention-based models.
SiLU (a.k.a. Swish) — x * sigmoid(x). Standard in modern ConvNets (EfficientNet) and many recent vision Transformers. Also what gates the FFN in LLaMA-family models (SwiGLU).
Sigmoid — squash to (0, 1). Use for binary classification outputs and gating mechanisms (LSTM gates, gated MLPs). Don't use in deep hidden layers — the gradient saturates badly.
Tanh — squash to (-1, 1). Mostly historical; LSTMs still use it. Same saturation problem as sigmoid.
Softmax — turn logits into a probability distribution. Almost always on the LAST dim. Used as the final activation in classifiers and as the score normalizer in attention.

Module form vs functional form

Most activations come in two flavors: nn.ReLU() (a Module) and F.relu(x) (a function). For activations, both are stateless, so they're functionally equivalent. Use the Module form when you want it to show up in print(model) or model.modules(); use the functional form for one-off uses inside forward().

Code

The shortlist in code·python

import torch
import torch.nn as nn
import torch.nn.functional as F

x = torch.randn(4, 8)

# Module form
relu = nn.ReLU()(x)
gelu = nn.GELU()(x)
silu = nn.SiLU()(x)
sigmoid = nn.Sigmoid()(x)
tanh = nn.Tanh()(x)

# Functional form — same result, no module to declare
relu_f = F.relu(x)
gelu_f = F.gelu(x)
silu_f = F.silu(x)

# Softmax — almost always on the last dim
probs = F.softmax(x, dim=-1)
print(probs.sum(-1))   # tensor([1., 1., 1., 1.])

Picking an activation by task·python

import torch.nn as nn

# Vision CNN — ReLU is fine, ConvNeXt-style uses GELU
cnn_block = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.ReLU())

# Transformer block — GELU is the modern default
xfm_ffn = nn.Sequential(nn.Linear(512, 2048), nn.GELU(), nn.Linear(2048, 512))

# Modern ConvNet (EfficientNet flavor) — SiLU
modern_cnn = nn.Sequential(nn.Conv2d(3, 32, 3, padding=1), nn.SiLU())

# Binary classifier output — sigmoid
binary_head = nn.Sequential(nn.Linear(512, 1), nn.Sigmoid())

Softmax — last dim, log-stable variant·python

import torch
import torch.nn.functional as F

logits = torch.randn(4, 10)            # batch=4, classes=10

# Probabilities — for inspection / sampling
probs = F.softmax(logits, dim=-1)
assert torch.allclose(probs.sum(dim=-1), torch.ones(4))

# log-softmax — numerically stable for log-likelihood losses
log_probs = F.log_softmax(logits, dim=-1)

# F.nll_loss expects log-probabilities; F.cross_entropy expects raw logits
# (it does log_softmax internally — that's why CE doesn't need a softmax layer
#  in your network)

Activations: ReLU, GELU, SiLU, and Friends

Non-linearity is what makes the network powerful

The shortlist you'll actually use

Module form vs functional form

Code

External links

Exercise

Progress

Comments 0