Residual Connections

The single most important architecture trick of the last decade

A residual connection is y = x + f(x): the output of a block is the input plus the block's transformation. The "skip" lets gradients flow through depth without being multiplied by the block's local Jacobian. This is what makes 100-layer networks (ResNet) and 175-billion-parameter transformers (GPT-3 and beyond) actually trainable.

Before residuals, training got worse as you added layers past 20-30. After residuals, depth pays off all the way up. This isn't a regularization trick — it's a depth-enabler.

Tip: Whenever you see x + sublayer(x) in code, that's a residual connection. Spot-check that the shapes match (skip + transform must have the same shape). When they don't, you need a projection (nn.Linear or nn.Conv2d with stride) on the skip.

Where residuals appear

Everywhere modern. ResNet (CNNs), Transformers (every block has two residuals — attention and FFN), DenseNet (concat instead of add), U-Net (encoder-to-decoder skip), Mamba/SSM blocks. The pattern transcends architecture family.

The math, briefly

If y = x + f(x), then dy/dx = 1 + f'(x). Even when f'(x) is tiny (vanishing gradient), the +1 keeps the gradient alive. The chain rule's product becomes a chain rule's sum-of-products, which is much more numerically forgiving.

Principle: Residual connections are the closest thing to a 'free' architectural improvement deep learning has. Every architecture you design or borrow should have residuals from the start, not bolted on later.

Code

ResNet basic block·python

import torch.nn as nn

class BasicBlock(nn.Module):
    def __init__(self, in_ch, out_ch, stride=1):
        super().__init__()
        self.conv1 = nn.Conv2d(in_ch, out_ch, 3, stride=stride, padding=1, bias=False)
        self.bn1   = nn.BatchNorm2d(out_ch)
        self.conv2 = nn.Conv2d(out_ch, out_ch, 3, stride=1, padding=1, bias=False)
        self.bn2   = nn.BatchNorm2d(out_ch)
        self.relu  = nn.ReLU(inplace=True)
        # Projection if shapes don't match
        if stride != 1 or in_ch != out_ch:
            self.skip = nn.Sequential(
                nn.Conv2d(in_ch, out_ch, 1, stride=stride, bias=False),
                nn.BatchNorm2d(out_ch),
            )
        else:
            self.skip = nn.Identity()

    def forward(self, x):
        identity = self.skip(x)
        out = self.relu(self.bn1(self.conv1(x)))
        out = self.bn2(self.conv2(out))
        return self.relu(out + identity)

The single most important architecture trick of the last decade

Where residuals appear

The math, briefly

Code

External links

Exercise

Progress

Comments 0