Skip Connections and Residual Blocks

The pattern that unlocked very deep nets

Residual connections (skip connections) are the pattern behind ResNet — one of the most influential architectures in deep learning. The idea: let information skip over a couple of layers and rejoin downstream via layers.add, so the block computes input + transformation rather than transformation alone. The Code block shows the canonical move — stash the input in shortcut, run it through a couple of layers, then add the two back together.

Why adding the input back helps

Before ResNet (2015), stacking layers past a couple dozen made networks harder to train, not better — accuracy degraded. Skip connections fixed that on two fronts. First, the add creates a gradient highway: during backprop the gradient flows straight through the identity path, so it reaches early layers without vanishing through every intervening nonlinearity. Second, the block only has to learn the residual — the small correction on top of the identity — which is a far easier target than re-learning the whole mapping from scratch. Together these are what made 50- and 152-layer networks trainable, and the pattern now shows up almost everywhere depth matters, including the transformer blocks behind modern LLMs.

The layers.add([x, shortcut]) performs element-wise addition, which requires both tensors to have the same shape. When a block changes the channel count or downsamples, that's no longer true — add a 1×1 convolution (or a Dense layer) on the shortcut to project it to the matching shape. This is the single most common reason a residual block fails to build.

Code

A residual block, used twice in a model·python

# A simple residual block
def residual_block(x, filters):
    shortcut = x  # Save the input

    x = layers.Dense(filters, activation="relu")(x)
    x = layers.Dense(filters)(x)

    # Add the skip connection
    x = layers.add([x, shortcut])
    x = layers.Activation("relu")(x)
    return x

# Use it in a model
inputs = keras.Input(shape=(128,))
x = layers.Dense(128, activation="relu")(inputs)
x = residual_block(x, 128)
x = residual_block(x, 128)
outputs = layers.Dense(10, activation="softmax")(x)
model = keras.Model(inputs, outputs)

Skip Connections and Residual Blocks

The pattern that unlocked very deep nets

Why adding the input back helps

Code

External links

Exercise

Progress

Comments 0