vLLM: PagedAttention and Continuous Batching

~28 min · serving, vllm

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

What vLLM is famous for

vLLM's claim to fame is PagedAttention: KV cache stored in fixed-size blocks like OS memory pages, with a block table per sequence. The result is near-zero KV-cache fragmentation — you can fit many more concurrent requests on the same GPU than a naive engine.

Continuous batching, in one paragraph

Naive batching: collect N requests, run them through together, emit N responses. The slowest request stalls all others. Continuous batching: at every decoding step, the engine can swap finished sequences out and pull queued sequences in. Sequence A on token 50 can share a step with sequence B on token 1. Throughput on bursty traffic goes up by 5-10x.

Two ways to run vLLM

OpenAI-compatible server: vllm serve {model_id} — drop-in for OpenAI clients.
Python library: LLM + SamplingParams for offline batch jobs.

Code

Run vLLM as an OpenAI-compatible server·bash

pip install vllm

# Start the server
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.85

# In another terminal:
curl http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Hi"}]}'

vLLM as a Python library (offline batch)·python

from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-3.1-8B-Instruct",
    quantization="awq",
    max_model_len=4096,
)

prompts = [
    "Hello",
    "Translate 'open source' to Korean",
    "Write a haiku about Hugging Face",
] * 100  # 300 prompts, batched

params = SamplingParams(temperature=0.7, max_tokens=80)
outputs = llm.generate(prompts, params)
print(len(outputs), outputs[0].outputs[0].text[:100])

External links

Exercise

Run the same model through TGI and vLLM with similar quantization (e.g. AWQ on both, or bnb-nf4 on both if available). Send 100 concurrent requests to each. Compare: tokens/sec total throughput, p50 latency, p95 latency, peak GPU memory. Note differences and likely causes.