The KV Cache Grows While You Talk

"The weights are the model's memory of training. The KV cache is its memory of this conversation, and it is paid for in bytes, per token, per layer."

What the Cache Is

Attention lets each new token look at every earlier token. To avoid recomputing the earlier tokens' keys and values every step, the model keeps them: for every layer that has full attention, for every token so far, a key vector and a value vector. That store is the KV cache. It is written during prefill, grows by one entry per decoded token, and is read in full by every decode step — which is why it belongs in the bytes-per-token formula alongside the weights, and why decode slows as a conversation lengthens.

Its size per token is arithmetic on the architecture. NVIDIA's inference guide gives the plain form: "Size of KV cache per token in bytes = 2 × (num_layers) × (num_heads × dim_head) × precision_in_bytes." Three refinements matter for real checkpoints. Grouped-query attention shares key/value heads across several query heads, so the count is the number of key-value heads, usually much smaller. Hybrid architectures — Qwen3.5's 3:1 linear-attention-to-full-attention layout — keep a KV cache only on the full-attention layers, and a small fixed state on the linear ones. Latent attention (DeepSeek's MLA, which GLM-5.3 also uses beneath its DSA sparse indexer) compresses keys and values into a low-rank latent before caching, which cuts the per-token bytes by an order of magnitude or more.

Read From the Configs

Checkpoint	Layers with a KV cache	KV heads × head dim	Bytes per token (bf16 cache)	At 32K tokens	At 1M tokens	Evidence
Llama-3.2-1B (pure GQA)	16 of 16	8 × 64	32 KB	1.07 GB	34 GB (past its 131K window)	physics from config.json
Qwen3.5-0.8B / 2B (hybrid)	6 of 24	2 × 256	12 KB	0.40 GB	12.9 GB	physics
Qwen3.5-4B / 9B (hybrid)	8 of 32	4 × 256	32 KB	1.07 GB	34.4 GB	physics
Qwen3.5-27B (hybrid)	16 of 64	4 × 256	64 KB	2.15 GB	68.7 GB	physics
Qwen3.5-35B-A3B (hybrid MoE)	10 of 40	2 × 256	20 KB	0.67 GB	21.5 GB	physics
GLM-5.3 (MLA + sparse indexer)	78	latent 512 + 64 rope	~95 KB (derived)	3.0 GB	~100 GB	physics, derived from the card and config
Kimi K3 (KDA + gated MLA)	24 of 93	latent	~27.6 KB (derived) + ~232 MB fixed	0.9 GB	29 GB	physics, derived
Qwen3.8-Flash-Next (DeltaNet + sparse)	12 of 48	—	~25 KB (derived) + ~115 MB fixed	0.8 GB	25 GB	physics, derived
DeepSeek-V4.1-Flash (CSA2 + sliding window)	—	—	890 B (model card)	0.03 GB	0.93 GB	vendor (card)

Two orders of magnitude, top to bottom, for the same job. A pure-attention model with sixty-four heads would sit above the top of this table — GLM-5.3's latent cache is roughly 59 times smaller than an uncompressed cache for its shape would be — and DeepSeek's design sits at the bottom with under a kilobyte per token, so that a million tokens of context costs a gigabyte. The Qwen3.5 ladder in the middle is where the household's measurements live, and its hybrid layout is why the 9B and the 1B share a 32 KB figure despite a nine-fold difference in weights.

Why the Slope Is the Ratio

Decode reads weights plus cache. So the fractional slowdown from a long context is roughly the cache's size relative to the weights: at 29K tokens the Llama-3.2-1B carries 0.94 GB of cache on 0.70 GB of weights — more than double the bytes — while the Qwen3.5-9B carries the same 0.94 GB on 4.47 GB, a fifth more, and the 27B carries 1.88 GB on 14.42, an eighth more. The next lesson measures exactly those three curves. The point to carry: the cache's cost is not a property of the context length alone; it is the ratio of KV bytes to weight bytes, and that ratio is set by the architecture and read from the config file.

Code

kv_per_token.py — the cache's per-token size, from config.json·python

#!/usr/bin/env python3
"""KV cache bytes per token from a checkpoint's config: 2 (K and V) × layers
that keep a cache × kv_heads × head_dim × bytes. Hybrids cache only on the
full-attention layers (full_attention_interval). Latent-attention models
(MLA) need the latent dims instead; this script handles GQA and hybrids."""
import json, sys
from pathlib import Path
from huggingface_hub import snapshot_download

repos = sys.argv[1:] or ["mlx-community/Llama-3.2-1B-Instruct-4bit", "mlx-community/Qwen3.5-9B-4bit",
                         "mlx-community/Qwen3.5-27B-4bit", "mlx-community/Qwen3.5-35B-A3B-4bit"]
BYTES = 2   # bf16 / fp16 cache

for repo in repos:
    cfg = json.loads((Path(snapshot_download(repo, local_files_only=True)) / "config.json").read_text())
    tc = cfg.get("text_config", cfg)
    L = tc["num_hidden_layers"]
    interval = tc.get("full_attention_interval")           # Qwen3.5: every 4th layer is full attention
    cached_layers = L // interval if interval else L
    kv_heads = tc["num_key_value_heads"]
    head_dim = tc.get("head_dim") or tc["hidden_size"] // tc["num_attention_heads"]
    per_token = 2 * cached_layers * kv_heads * head_dim * BYTES
    print(f"{repo.split('/')[-1]:28} cache on {cached_layers:3}/{L:3} layers, {kv_heads}×{head_dim} -> "
          f"{per_token/1024:5.1f} KB/token; 32K = {per_token*32768/1e9:.2f} GB, 1M = {per_token*1_048_576/1e9:.1f} GB")

# office, 2026-09-15:
# Llama-3.2-1B-Instruct-4bit   cache on  16/ 16 layers, 8×64  ->  32.0 KB/token; 32K = 1.07 GB, 1M = 34.4 GB
# Qwen3.5-9B-4bit              cache on   8/ 32 layers, 4×256 ->  32.0 KB/token
# Qwen3.5-27B-4bit             cache on  16/ 64 layers, 4×256 ->  64.0 KB/token
# Qwen3.5-35B-A3B-4bit         cache on  10/ 40 layers, 2×256 ->  20.0 KB/token

Exercise

Run kv_per_token.py on every checkpoint you have downloaded. For the one you use most, compute cache bytes at your typical context and add the ratio cache ÷ weights to your card. Then predict the decode slowdown from an empty context to your typical one — and keep the prediction for the next lesson, which measures it.

Hint

Slowdown ≈ (weights + cache) ÷ weights, diluted by fixed overhead on small models. If your ratio is under 0.1, long context will barely show in decode and will show entirely in time-to-first-token; if it is above 1, the conversation has become heavier than the model.