LSTMs and GRUs

The gating insight

Vanilla RNNs vanish gradients on long sequences because the same weight matrix is multiplied through every timestep. LSTMs (Long Short-Term Memory, Hochreiter & Schmidhuber, 1997) solve this by adding a separate cell state c_t that flows through time mostly unchanged, with three learned gates deciding what to forget, what to add, and what to read out.

The forget gate is the key piece. It's a sigmoid-valued vector (0 to 1, per dimension) that elementwise multiplies the previous cell state. If a dimension's forget gate is near 1, that piece of memory survives for many steps. If it's near 0, that piece is wiped immediately. Backprop through this gate is much friendlier than backprop through a tanh layer.

Tip: If you've seen 'LSTM' as a black box label and want to demystify it: it's a vanilla RNN cell with three sigmoid gates plus an extra state vector. The math fits on one slide. The intuition is 'a value that survives many timesteps with controllable updates.'

GRUs — the leaner cousin

GRUs (Gated Recurrent Unit, Cho et al., 2014) merge the forget and input gates into one update gate, and drop the separate cell state. Fewer parameters, slightly less expressive on some tasks, much closer to LSTM in practice. Good default if you want recurrence and don't want to think about it.

In PyTorch you use them with one line — nn.LSTM or nn.GRU — and treat them as drop-in replacements for each other. The choice rarely matters much; pick the one your team is already comfortable with.

Where they still win in 2026

Streaming inference at constant per-token cost (transformer attention is O(n) per token, RNNs are O(1)). Reinforcement learning policies that consume sequential observations. Some embedded ASR pipelines. Most general sequence work has moved to transformers.

Principle: LSTM and GRU are competent veterans, not central characters in 2026. Know them well enough to read legacy code; reach for transformers (or Mamba-class state-space models) for new work.

Code

LSTM and GRU as drop-in sequence encoders·python

import torch
import torch.nn as nn

x = torch.randn(8, 50, 64)              # [B, T, in_dim]

lstm = nn.LSTM(input_size=64, hidden_size=128, num_layers=2,
               batch_first=True, bidirectional=False)
out_lstm, (h, c) = lstm(x)
print("LSTM out:", out_lstm.shape)       # [8, 50, 128]
print("LSTM h:",   h.shape)              # [num_layers, B, hidden]
print("LSTM c:",   c.shape)              # [num_layers, B, hidden]

gru = nn.GRU(input_size=64, hidden_size=128, num_layers=2,
             batch_first=True)
out_gru, h_gru = gru(x)
print("GRU out:", out_gru.shape)         # [8, 50, 128]
print("GRU h:  ", h_gru.shape)           # [num_layers, B, hidden]

The gating insight

GRUs — the leaner cousin

Where they still win in 2026

Code

External links

Exercise

Progress

Comments 0