StripedHyena and Real-World Performance

StripedHyena-7B — the first credible Hyena LLM

StripedHyena-7B (Together AI, December 2023) was the first alternative architecture to be competitive with Llama-2 7B and Mistral 7B on standard benchmarks. The trick was hybrid: use a majority of Hyena layers for the bulk of the model, plus a minority of standard attention layers (with RoPE) where the model needs precise recall. This is the same hybrid pattern that emerged in the SSM space.

Training was dramatically faster — over 50% faster training at 128K context compared to a Transformer baseline. This isn't a paper-only number; the StripedHyena codebase is open and the speedup is reproducible.

StripedHyena 2 — multi-scale operators

StripedHyena 2 (February 2025, arXiv:2503.01868) introduced something genuinely new: three operator types specialized for different sequence ranges. Hyena-SE handles short-range efficient mixing. Hyena-MR handles medium-range. Hyena-LI uses long implicit convolutions for the deep range. Different layers in the model assign different operators based on which scale of dependency they're meant to capture.

This multi-scale design is interesting beyond Hyena specifically — it's a pattern that could (and probably will) show up in future hybrid architectures across families. Instead of one operator forced to handle every range, assign specialized operators to the ranges they're best at.

The killer app: Evo 2

Here's where the story gets interesting. The most impactful Hyena deployment isn't a chatbot. It's Evo 2 — a 40B-parameter genomic model with 1M context that generates viable mitochondrial genomes. Powered by StripedHyena 2. Built by the Arc Institute, Stanford, and Together AI.

Genomics is where Hyena's profile fits perfectly: extremely long sequences (DNA strands run to billions of bases), patterns that are more regular and more local than natural language, and exact recall that matters less than capturing statistical patterns across the full sequence. A Transformer at 1M context length is essentially impossible without specialized infrastructure; Hyena handles it natively.

Exercise

Read the Evo 2 announcement and any one of its accompanying technical appendices. Note specifically how the team describes why they chose Hyena over a Transformer or an SSM. The reasoning will be: "context length we need is in the millions, recall isn't the main capability we want, statistical pattern capture is". That reasoning template — match the architecture to the data shape — is portable to any non-language workload you might build.