Mamba and State-Space Models — Linear Scaling · Beyond the Transformer Quest

The premise

Transformers have a quadratic scaling problem: attention is O(n²) in sequence length. Doubling context length quadruples attention compute and KV-cache size. State-Space Models (SSMs) replace attention with a linear-scaling alternative: O(n). At very long sequences, this is an asymptotic win that no amount of attention-optimization can match.

What Mamba added to SSMs

Earlier SSMs (S4, S5) had fixed dynamics — the state-update equations didn't depend on the input. Mamba (December 2023) introduced selective state spaces: input-dependent dynamics, where the model can decide based on the current token how much of the past state to keep, forget, or update. This single change closed most of the quality gap with attention while preserving the linear-scaling advantage.

The mental model

Attention is "look at every previous token weighted by similarity". SSM is "carry a fixed-size state, update it as each new token arrives, throw away the original tokens". Attention has perfect recall but expensive scaling. SSM has bounded memory but linear scaling. They are different bets about what matters.

Mamba 2 — the duality result

Mamba 2 (2024) showed that selective SSMs and attention are unified under a "structured state-space duality" (SSD) framework — they are different parameterizations of a similar underlying computation. This let Mamba 2 borrow optimization techniques from attention engineering and yielded faster training. Mamba 3 (2026) refined this further.

Where SSMs win

Above ~370 tokens, Mamba inference becomes faster than equivalent Transformers. By 4K tokens, Mamba has a ~12× memory efficiency advantage. For very-long-context streaming workloads (transcription, log analysis, real-time monitoring), SSMs are competitive or superior.

Where SSMs still struggle

Exact recall. SSMs compress past context into a fixed-size state. Long-range exact recall (the kind needed for "find this fact buried 90% into a 100K-token doc") is harder.
Frontier-scale training. Pure SSM models above ~10B parameters are rare; the scaling-law confidence is much weaker than for Transformers.
Reasoning workloads. Extended thinking benefits from precise recall over long sequences — a strength of attention.

Code

Selective SSM update (intuition, not real implementation)·python

def selective_ssm_step(x_t, h_t, A, B, C, delta_t):
    # x_t: current input token's hidden representation
    # h_t: state carried from previous step
    # delta_t: input-dependent step size — the 'selective' part
    h_t_new = (1 - delta_t) * (A @ h_t) + delta_t * (B @ x_t)
    y_t = C @ h_t_new
    return y_t, h_t_new
# Linear in sequence length — no quadratic attention computation.

Exercise

Read the Mamba paper's section comparing inference latency vs Transformers across different sequence lengths. Note where the crossover is for your typical workload's context length. If your workload is mostly under 4K tokens, Mamba is not where the gains live for you. If it's mostly over 32K, take a closer look.

Mamba and State-Space Models — Linear Scaling