Residual Connections: Why You Can Stack 80 Layers

A residual connection adds the input of a sublayer to its output: output = x + Sublayer(x). The simplicity hides three properties that together let modern Transformers scale to 80+ layers without training collapse.

Gradient highway. Backpropagation flows through the residual addition without modification. Even if Sublayer(x) is poorly initialized, gradients can still reach earlier layers via the skip path. This solves the vanishing-gradient problem that limited pre-2015 deep networks to dozens of layers at most.
Identity initialization. If Sublayer's weights are small at init, x + Sublayer(x) ≈ x. Each block starts as approximately the identity; adding more blocks doesn't break a working model. Training then nudges blocks one at a time toward useful transformations.
Ensemble interpretation. A network with N residual blocks can be seen as an ensemble of 2^N paths through different combinations of "skip vs use" decisions. Veit et al. (2016) showed that most of the model's effective depth is shorter than its nominal depth — the architecture is robust to layer dropout.

Without residual connections, Transformers deeper than ~6 layers struggle to train. With them, you can stack 32 (Llama 3 8B), 80 (Llama 3.3 70B), or even 126 (Llama 4 Maverick variant) blocks and still optimize cleanly.

Code

Block with residuals·python

class TransformerBlock(nn.Module):
    def __init__(self, d_model, attn, ffn, norm_cls=RMSNorm):
        super().__init__()
        self.norm1 = norm_cls(d_model)
        self.attn = attn
        self.norm2 = norm_cls(d_model)
        self.ffn = ffn
    def forward(self, x):
        # Each sublayer is added to the residual stream
        x = x + self.attn(self.norm1(x))
        x = x + self.ffn(self.norm2(x))
        return x

Residual Connections: Why You Can Stack 80 Layers

Code

External links

Exercise

Progress

Comments 0