Where Hyena shines
Hyena is at its best on workloads that look like the genomics workload: very long sequences, patterns that are more regular than natural language, and tolerance for some loss of exact retrieval. Specifically:
- Genomic modeling — DNA, RNA, protein sequences. Evo 2 is the canonical example. HyenaDNA was the predecessor.
- Byte-level modeling — operating directly on bytes (no tokenizer), where sequences are 4× longer than tokenized form.
- Time-series forecasting on data with strong periodic structure, where convolutional inductive bias helps.
- Long-context document processing where the task is statistical pattern capture, not retrieval.
Where Hyena struggles
On natural language modeling at standard scale, Hyena is competitive but doesn't dominate. At 360M scale, Hyena has perplexity 10.11 vs. attention's 8.39 — close but a real gap. At larger scales the gap shrinks but doesn't fully close, and Mamba-family architectures have generally pulled ahead on language tasks.
The biggest practical limitation is recall. On recall-intensive benchmarks, Hyena scores around 5.1 compared to attention's 47.7 — that is, Hyena is roughly an order of magnitude worse at retrieving specific past tokens. This is even weaker than Mamba on the same family of tasks, because Hyena's convolution-based mixing has no equivalent of selectivity. Filters are generated from positions, not from content.
Training fragility
Hyena is notably learning-rate sensitive — narrower stable LR window than even Mamba. The implicit-filter FFNs are sensitive to initialization, and convergence can be brittle if you're not following the published recipes carefully. For research labs this is a tax; for production teams it's a barrier. This is part of why Hyena hasn't seen broader language-modeling adoption beyond the StripedHyena line.