Speculative Decoding and KV-Cache Tricks Are Not Architecture

Speculative decoding — a serving optimization

Speculative decoding uses a small "draft" model to quickly generate K candidate tokens, then the larger "verifier" model checks all K in parallel — accepting the prefix that matches what it would have produced. The output is mathematically identical to standard decoding. It is purely a latency optimization that trades extra compute for fewer sequential model calls.

Why this is not a new architecture

Speculative decoding does not change either model. It is a runtime trick that runs two models in lockstep. Both can be off-the-shelf. The "speculation" lives in the orchestration code that decides when to accept or reject draft tokens.

KV-cache attention variants — GQA, MQA, MLA

These are real changes inside the attention layer, but they are refinements within the Transformer paradigm, not new architecture families.

MHA (Multi-Head Attention) — the original. Each head has its own K, V projections.
MQA (Multi-Query Attention) — share a single K, V projection across all heads. Smallest KV cache, most aggressive.
GQA (Grouped Query Attention) — group heads, share K, V within each group. Compromise between MHA and MQA. Llama 3, Gemma, Qwen all use GQA.
MLA (Multi-head Latent Attention) — DeepSeek's variant. Compresses K, V into a low-rank latent space, decompresses on demand. Even smaller KV cache.

Why these matter for serving but not for taxonomy

GQA, MQA, and MLA reduce the KV-cache memory footprint dramatically — important for long-context serving and multi-tenant inference. But the basic attention pattern (Query attends to Keys, weighted by softmax similarity, applied to Values) is unchanged. They are engineering choices inside the same architectural family.

The reading rule

If a paper introduces a new attention variant, the question is "does the basic Q×K→softmax×V pattern still hold?" If yes, it is an attention refinement, not a new architecture family. If no, you are looking at a genuine non-attention design (Mamba, RWKV — covered in the Frontier track).

Code

Speculative decoding skeleton·python

def speculative_decode(prompt, draft_model, verifier_model, k=4):
    output = []
    while not done(output):
        draft_tokens = draft_model.generate(prompt + output, max_new=k)
        verifier_logprobs = verifier_model.score(prompt + output, draft_tokens)
        accepted = []
        for tok, lp in zip(draft_tokens, verifier_logprobs):
            if accept(tok, lp):
                accepted.append(tok)
            else:
                accepted.append(verifier_model.sample_one(prompt + output + accepted))
                break
        output.extend(accepted)
    return output
# Output is identical in distribution to verifier_model.generate alone.

MHA vs GQA vs MQA — KV head count·python

# MHA: num_kv_heads == num_q_heads (e.g., 32 query heads, 32 KV heads)
# GQA: num_kv_heads <  num_q_heads (e.g., 32 query heads,  8 KV heads in Llama 3)
# MQA: num_kv_heads == 1            (e.g., 32 query heads,  1 KV head)

Exercise

Pick three open-weight models from different families (Llama, Qwen, DeepSeek) and find their config.json on Hugging Face. Note num_attention_heads vs num_key_value_heads — that ratio tells you whether they use MHA, GQA, or MQA. Notice all of them are still Transformers; the wiring family is the same.