Sequence Modeling Before Transformers: RNNs and LSTMs

Before 2017, Recurrent Neural Networks (RNNs) and their gated variant LSTMs were the default architecture for sequential data — text, speech, time series, even music. Understanding what they did right and where they hit a wall is the cleanest way to see why the Transformer was such a big deal.

An RNN consumes a sequence one element at a time, carrying a hidden state that compresses everything it has seen so far. At each step, the network mixes the current input with the previous hidden state, producing the next hidden state and (optionally) an output token.

LSTMs improved on vanilla RNNs by adding gating — forget gates, input gates, output gates — that explicitly control which information to keep, overwrite, or release. This made longer-range dependencies easier to learn, but the fundamentally sequential update remained.

What was actually wrong

Three problems that all stem from "one element at a time":

No parallelism inside a sequence. Position 100's hidden state requires position 99's, which requires 98's, all the way down. GPUs love parallel matmuls; RNNs starve them.
Vanishing/exploding gradients across many steps. Even with LSTM gates, gradients shrink or blow up after a few hundred steps in practice.
Information bottleneck. Everything earlier than the current step must squeeze through a single hidden state vector. Long-range dependencies get blurred.

This isn't ancient history. RNNs and LSTMs powered Google Translate (2016), early Siri voice recognition, and most production NLP through 2017. They worked. The Transformer didn't replace something broken — it replaced something fundamentally bottlenecked.

Code

RNN forward pass — sequential by construction·python

# RNN: each hidden state depends on the previous one
import numpy as np

def rnn_forward(x, W_h, W_x, b):
    # x: (seq_len, d_in), h: (d_h,)
    seq_len = x.shape[0]
    h = np.zeros(W_h.shape[0])
    outputs = []
    for t in range(seq_len):
        h = np.tanh(W_h @ h + W_x @ x[t] + b)
        outputs.append(h.copy())
    return np.stack(outputs)

# Notice: cannot start step t until step t-1 finishes.
# This is the bottleneck the Transformer eliminates.

LSTM cell — gates around the same recurrence·python

# LSTM cell (simplified): same sequential structure, more parameters
def lstm_step(x_t, h_prev, c_prev, W):
    # f: forget, i: input, o: output, g: candidate
    f = sigmoid(W['f'] @ np.concatenate([x_t, h_prev]))
    i = sigmoid(W['i'] @ np.concatenate([x_t, h_prev]))
    o = sigmoid(W['o'] @ np.concatenate([x_t, h_prev]))
    g = np.tanh(W['g'] @ np.concatenate([x_t, h_prev]))
    c = f * c_prev + i * g       # update cell state
    h = o * np.tanh(c)            # produce hidden state
    return h, c
# Gates help long dependencies, but t still waits for t-1.

Exercise

Build a 2-layer LSTM in PyTorch for character-level text generation on Tiny Shakespeare. Train for 5 epochs on a CPU. Note three things: (a) wall-clock time per epoch, (b) what happens to perplexity, (c) where the generated text starts to feel coherent vs. degenerate. Save these numbers — you'll compare them against a tiny Transformer in Track 4.

Sequence Modeling Before Transformers: RNNs and LSTMs

What was actually wrong

Code

External links

Exercise

Progress

Comments 0