Long-Range Dependencies and the Vanishing Gradient

Even with LSTM gates, information from early positions degrades as it passes through hundreds of sequential transformations. This is the long-range dependency problem, and it is what limited practical RNN context to the order of a few hundred tokens for most of the 2010s.

Two failure modes come together here. First, the forward path: every position your information passes through can lossy-compress it into a fixed-size hidden state. By position 200, the signal from position 1 is buried under 199 layers of mixing. Second, the backward path: gradients during training must flow back through all those steps, and repeated multiplication by Jacobians shrinks them exponentially. This is the famous vanishing gradient problem.

Why attention is qualitatively different

Self-attention puts every pair of positions one hop away — there is a single weight matrix that compares any two tokens directly. There is no chain to walk, so neither forward information loss nor backward gradient decay accumulates with distance. The cost shows up as O(n²) compute, not as gradient pathology.

This is the cleanest single explanation for why "longer context" became practical only after attention. Modern Transformers handle 128K, 1M, even 10M tokens (LLaMA 4 Scout) — not because RNNs got better, but because attention removed the mechanism that made distance lossy in the first place.

Code

Path length: O(n) vs O(1)·python

# RNN: position 1 reaches position n through n hops
def reach_rnn(x, n):
    h = encode(x[0])
    for t in range(1, n):
        h = step(h, x[t])    # information passes through every step
    return h                  # signal from x[0] now buried n layers deep

# Attention: position 1 reaches position n in one matmul
def reach_attention(X):
    # X is (n, d). Q, K, V derived once.
    Q, K, V = project(X), project(X), project(X)
    return softmax(Q @ K.T / sqrt(d)) @ V
    # Every (i, j) pair has a direct weight. No depth in i.

Long-Range Dependencies and the Vanishing Gradient

Why attention is qualitatively different

Code

External links

Exercise

Progress

Comments 0