Activations: ReLU → GELU → SwiGLU

The non-linear activation inside the FFN has evolved through three eras.

ReLU (original Transformer)

max(0, x). Simple, fast. Suffers from "dead neurons" — units that get pushed into the negative regime never recover because their gradient is exactly zero. Used by the 2017 Transformer and many early models, almost no modern LLM uses it.

GELU (GPT-2/3, BERT)

x · Φ(x) where Φ is the standard normal CDF. Smoothly approximates ReLU but the gradient is non-zero everywhere. No dead neurons. Better empirical performance. Used by GPT-2, GPT-3, BERT, RoBERTa, T5.

SwiGLU (modern standard)

Combines Swish (a smooth ReLU variant) with a multiplicative gate. SwiGLU has three weight matrices instead of two — the third one is a "gate" that elementwise modulates the activation:

SwiGLU(x) = (Swish(x · W₁) ⊙ (x · W₃)) · W₂

The gate gives the model fine-grained control over which features to pass through and which to suppress. To keep parameter count comparable to a 4× FFN, d_ff is reduced to ~8/3 × d_model when using SwiGLU. Llama, Mistral, Mixtral, Qwen, Gemma — every modern open-weight LLM uses SwiGLU.

Code

SwiGLU implementation·python

class SwiGLU(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        # Three weight matrices instead of two
        self.W1 = nn.Linear(d_model, d_ff, bias=False)
        self.W2 = nn.Linear(d_ff, d_model, bias=False)
        self.W3 = nn.Linear(d_model, d_ff, bias=False)
    def forward(self, x):
        return self.W2(F.silu(self.W1(x)) * self.W3(x))
        # F.silu is the same as Swish (x * sigmoid(x))

Activation comparison plot·python

import torch, matplotlib.pyplot as plt
x = torch.linspace(-3, 3, 200)
plt.plot(x, F.relu(x).numpy(),  label='ReLU')
plt.plot(x, F.gelu(x).numpy(),  label='GELU')
plt.plot(x, F.silu(x).numpy(),  label='Swish (SiLU)')
plt.legend(); plt.grid(); plt.savefig('activations.png')

Activations: ReLU → GELU → SwiGLU

ReLU (original Transformer)

GELU (GPT-2/3, BERT)

SwiGLU (modern standard)

Code

External links

Exercise

Progress

Comments 0