Inference Optimization: Speculative Decoding, PagedAttention, Continuous Batching

Modern serving systems combine four major optimizations to extract usable throughput from frontier models. Knowing what each does is essential for choosing a stack and debugging performance.

Flash Attention 2/3

Already covered in Track 4. Tile attention so the (n × n) score matrix never materializes in HBM. 2-4× attention throughput vs naive. FA3 supports FP8 + warp specialization on H100 (~740 TFLOPs/s).

GQA / MQA

Architecture-level. Reduces KV cache by sharing K, V across query heads. Llama 3.3's 8 KV heads (vs 64 Q heads) shrink the cache 8×. Allows long context to fit in less GPU memory.

Speculative decoding

Use a small draft model to generate K candidate tokens; verify them in one forward pass of the target (large) model. Accept the longest matching prefix. 2-3× wall-clock speedup with identical output distribution. Implementations: Llama-cpp, vLLM, MLC-LLM.

PagedAttention (vLLM)

Manage KV cache like an OS virtual-memory system: fixed-size pages, block tables mapping logical to physical, copy-on-write for shared prefixes. Reduces fragmentation waste from 60-80% to ~4%, increasing the number of concurrent requests per GPU dramatically.

Continuous batching

Iteration-level scheduling: at each decoding step, sequences that have finished are removed and new requests slotted in. Reduces GPU idle time from ~40% to under 10%. 5-23× throughput improvement vs static batching.

Technique	What it solves	Speedup
Flash Attention	Memory bandwidth in attention	2-4×
GQA / MQA	KV cache size	2-8× cache reduction
Speculative decoding	Sequential decode bottleneck	2-3× wall-clock
PagedAttention	Memory fragmentation	~20× more requests/GPU
Continuous batching	GPU idle time	5-23× throughput

Code

Serving with vLLM (PagedAttention + continuous batching)·python

from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Llama-3-8B-Instruct",
          gpu_memory_utilization=0.9,
          max_model_len=8192)

# Speculative decoding (small draft model)
# llm = LLM(model="...", speculative_model="meta-llama/Llama-3.2-1B")

params = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=512)
outputs = llm.generate([prompt for prompt in many_prompts], params)
# vLLM batches them dynamically with continuous batching;
# PagedAttention manages KV cache pages.
# At scale this is 10-20× faster than naive HuggingFace generate.

Inference Optimization: Speculative Decoding, PagedAttention, Continuous Batching

Flash Attention 2/3

GQA / MQA

Speculative decoding

PagedAttention (vLLM)

Continuous batching

Code

External links

Exercise

Progress

Comments 0