Recurrent Neural Networks

The basic recurrence

An RNN processes a sequence one token at a time, maintaining a hidden state h that summarizes everything seen so far. At each step: h_t = f(W_h h_{t-1} + W_x x_t + b). The same weights are applied at every step (parameter sharing across time, like CNNs share across space).

Vanilla RNNs are simple and elegant but have terrible vanishing gradients on long sequences. The standard fixes are LSTM (Long Short-Term Memory, 1997) and GRU (Gated Recurrent Unit, 2014), both of which add gating mechanisms that let gradients flow further back through time.

Tip: RNNs feel like they should be the obvious choice for sequences. They were dominant from ~2014 to ~2017. Then transformers ate their lunch on almost everything. Today RNNs are mostly used for: extremely long sequences where attention's quadratic cost hurts, embedded systems where the constant per-token compute matters, and reinforcement learning where the sequential rollout aligns naturally.

LSTM cells, in pictures and in code

An LSTM cell maintains TWO state vectors: a hidden state h (output) and a cell state c (long-term memory). Three gates (forget, input, output) control what flows where. The forget gate is the key insight — it lets the cell selectively keep or drop information across hundreds of timesteps without vanishing.

When RNNs are still the right answer in 2026

Streaming inference (one token at a time, fixed memory). State-space models (Mamba, S5) which are RNN-like and competitive with transformers at very long context. Some music/audio generation pipelines. Most other sequence tasks have moved to transformers.

Principle: Learn LSTM and GRU shapes once. You probably won't reach for them as your first choice, but you'll see them in legacy code, RL agents, and the occasional embedded system where their constant per-step cost matters.

Code

An LSTM-based sequence classifier·python

import torch
import torch.nn as nn

class LSTMClassifier(nn.Module):
    def __init__(self, vocab_size, embed_dim, hidden_dim, num_classes):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)
        self.lstm  = nn.LSTM(embed_dim, hidden_dim, batch_first=True,
                             bidirectional=True)
        self.head  = nn.Linear(hidden_dim * 2, num_classes)

    def forward(self, ids):                    # ids: [B, T]
        x = self.embed(ids)                    # [B, T, embed_dim]
        out, (h, c) = self.lstm(x)             # out: [B, T, 2*hidden_dim]
        pooled = out.mean(dim=1)               # [B, 2*hidden_dim]
        return self.head(pooled)               # [B, num_classes]

The basic recurrence

LSTM cells, in pictures and in code

When RNNs are still the right answer in 2026

Code

External links

Exercise

Progress

Comments 0