The case for going back to convolutions
Convolutions were the dominant sequence-mixing primitive before attention won. They have well-understood mathematical properties, well-tested kernels, and three structural advantages worth re-examining: shift-invariance (the same pattern is recognized regardless of where it appears in the sequence), natural FFT computation at O(L log L) cost, and parameter sharing across positions (a kernel of size k is one set of weights applied everywhere).
The historical objection: standard 1D convolutions have a fixed receptive field. A convolution with kernel size k can only mix information within a k-token window. To handle long-range dependencies you'd need astronomically large kernels — and the kernel weights would have to be stored, requiring proportional memory.
The implicit-filter unlock
Hyena's insight was: you don't have to store the filter. Generate it on-the-fly from a small neural network that takes position as input and outputs filter weights. With this trick, the filter can be arbitrarily long — say, the full sequence length — without ever existing in memory all at once. The FFN that generates the filter has a fixed parameter count regardless of how long the filter is.
This is the same engineering pattern as positional encodings: instead of learning explicit position embeddings (one parameter per position), use a function (RoPE rotates by an angle that's a function of position). Implicit filters are the convolution analog. The compute cost shifts from "store a giant filter" to "call a small FFN at every position" — which on modern hardware is a much better trade.