Gemma — Sliding Window Attention as a Dense Optimization

Gemma's defining trick

Google's Gemma family is the cleanest demonstration that you can stay dense and still squeeze efficiency out of the attention computation. Gemma 2 (June 2024) introduced local/global sliding-window attention interleaving — alternating layers between cheap local attention (4K window) and full global attention. Gemma 3 sharpened the ratio to 5:1 (5 local layers per 1 global), with 262K vocabulary and 128K context.

Why interleaving matters

Full attention is O(n²) in sequence length — every token attends to every other token. At a 128K context window that gets painful quickly. Local sliding window attention reduces each layer to O(n × w) where w is the window size (e.g., 4K). By interleaving, the model still has full attention available somewhere — just not at every layer.

The intuition

Most tokens really only need to look at the last few thousand tokens to do their job. The few tokens that need long-range dependencies can rely on the global-attention layers to pull information across the entire context. You get most of the cost savings of pure local attention, with most of the recall of full attention. It is a principled compromise, not a hack.

Gemma family snapshot

Gemma 2 (2B, 9B, 27B): introduced the local/global pattern.
Gemma 3 (1B, 4B, 12B, 27B): 5:1 ratio, 128K context, 262K vocab. The 27B-IT is a strong dense workhorse.
Gemma 4 (2025): 31B dense + 26B-A4B MoE variant — Google's first MoE Gemma, but the 31B dense is still the easier default.

What this teaches about dense

"Dense" does not mean "uniform". Gemma's interleaving lives entirely within the dense paradigm — every token still activates every parameter, the FFNs are still dense, there are no experts. The only thing that changes is which tokens each attention head can look at. It is a useful counterexample if you start to think dense means boring.

Code

Local vs global attention — one-line difference·python

# Global attention: token at position i can attend to positions [0..i]
def global_mask(n):
    return [[1 if j <= i else 0 for j in range(n)] for i in range(n)]

# Local sliding window: token at i attends only to [max(0, i-w+1)..i]
def local_mask(n, w=4096):
    return [[1 if (i - w + 1) <= j <= i else 0 for j in range(n)] for i in range(n)]

# Gemma interleaves: ~5 local layers, then 1 global layer, repeat.

Exercise

Read the Gemma 3 technical report's section on sliding-window attention. Calculate by hand: for a sequence of 128K tokens, with 5:1 local:global ratio and a 4K local window, what fraction of attention compute does each global layer represent compared to each local layer? This is the kind of intuition you need to read efficient-attention papers.