The Hyena Operator

The operator in one paragraph

Hyena (Poli et al., February 2023, ICML 2023, arXiv:2302.10866) is built around an operator that interleaves implicit long convolutions with multiplicative gating. An order-N Hyena operator applies N convolutions with N gates in alternating sequence. Each filter is generated implicitly by a small FFN that takes positional encoding as input.

The complexity is O(L · D · (log L + D)), where L is sequence length and D is model dimension. The log L term comes from FFT. At short sequences this is similar to attention's O(L² · D) cost (roughly comparable for L ~ D). The Hyena advantage scales with sequence length: at 8K tokens, Hyena is roughly 2× faster than attention; at 64K tokens, 100× faster. The longer the sequence, the bigger the gap.

The multiplicative gating piece

The convolution alone doesn't give the model enough nonlinear capacity to compete with attention. Hyena adds multiplicative gating between convolution layers: x ← x * gate(v), where v is a learned per-token modulator. This is the same kind of gating you see in GLU and SwiGLU activations, just applied within the Hyena operator. It's what gives Hyena enough expressive power to be competitive on language tasks at all.

Where the speed actually matters

The 2× speedup at 8K is interesting but not transformative — FlashAttention buys you a comparable speedup over naive attention. The 100× speedup at 64K is where Hyena starts mattering as a deployment choice. And at megabase-scale genomic sequences, it's not even a comparison anymore — Transformers can't run at all on those lengths, while Hyena can.

Code

Hyena operator structure (pseudocode)·python

def hyena_operator(x, order=3):
    # x: (batch, seq_len, d_model)
    L = x.shape[1]
    positions = positional_encoding(L)
    for i in range(order):
        # Generate implicit filter from positions (small FFN, fixed params)
        h_i = filter_ffn(positions)         # (L,) or (L, d)
        # Apply via FFT — O(L log L) per filter
        x = fft_conv(x, h_i)
        # Multiplicative gate using a per-token learned vector
        x = x * silu(linear_gate_i(x))
    return x

Exercise

Implement a single-order Hyena operator (one implicit-filter convolution + one gate) and benchmark it against torch.nn.MultiheadAttention at sequence lengths 1K, 8K, 32K, 128K, on a single GPU. Plot wall-clock latency. You should see Hyena and attention crossing somewhere around 8K, with Hyena pulling clearly ahead by 32K. This is the practical curve that determines when Hyena is worth using.