S4 made SSMs practical with HiPPO
S4 (Gu, Goel, Ré, 2022) was the breakthrough that turned SSMs from a control-theory curiosity into a credible deep-learning architecture. The trick was the HiPPO initialization (High-order Polynomial Projection Operator), which gives the A matrix a structured form that's mathematically optimal for compressing the input history into a polynomial basis. With HiPPO, the model could remember information from thousands of timesteps back without exploding or collapsing.
S4 also worked out the convolution view of the linear recurrence: when A is fixed, the SSM can be "unrolled" into a global convolution kernel K = (CB, CAB, CA²B, CA³B, ...). Applying that kernel via FFT gives O(n log n) training, fully parallel on GPUs. Same model, two views: recurrent at inference, convolutional at training. S4 dominated the Long Range Arena benchmark and proved that SSMs could be more than a footnote.
S4D and H3 — simplification and gating
S4D simplified S4 by restricting A to be diagonal. The full HiPPO matrix had a complex structured form that was awkward to implement; the diagonal restriction lost a small amount of expressiveness for a much friendlier GPU implementation. This was the engineering pattern that would repeat: simplify the math to win on the hardware, recover quality with bigger models or other tricks.
H3 (Hungry Hungry Hippos, 2023) added gating mechanisms to SSM blocks — multiplicative interactions inspired by the gates in LSTMs and GRUs. With H3, SSMs started approaching Transformer quality on language modeling for the first time, especially on associative-recall-style tasks. But all of S4, S4D, and H3 were still Linear Time-Invariant.
The LTI ceiling
LTI means the matrices A, B, C are fixed, regardless of input. The same A is applied at every timestep — no matter what the current token is. This is what enables the convolution view (a global kernel exists because A doesn't depend on x), but it's also a hard ceiling on expressiveness. An LTI SSM cannot decide "this particular token is important — remember it more" or "this token is filler — fade it out". Every input is processed by the same time-invariant filter.
The S6 step — "Selective State Space", which became Mamba — broke LTI by making B, C, and Δ functions of the input. That's the next track. Everything in this lesson is the runway that selectivity finally took off from.