Quantization at Serve Time

~24 min · serving, quantization

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Quantization formats, calibrated

Format	Bits	Speed	Quality	Notes
`fp16 / bf16`	16	baseline	baseline	What unquantized "base" usually means in 2026.
`fp8`	8	1.3-1.5x	~baseline	Hopper / Ada GPUs only. Requires a fp8-aware engine.
`bitsandbytes (8/4-bit)`	8 / 4	1-1.5x	~baseline / minor	Fastest path to "fits on smaller GPU." Less optimal for serving.
`GPTQ`	4 / 3	2x	small drop	Per-channel quantization. Variant set on Hub: `{model}-GPTQ`.
`AWQ`	4	2-3x	small drop	Activation-aware quantization. Currently the strongest 4-bit serving option.
`GGUF`	2-8	n/a	varies	llama.cpp / Ollama format. Not for TGI / vLLM. Covered in ops track.

The decision rule

Production serve target on modern GPU: AWQ-4bit if a variant exists, else fp8 if your GPU supports it, else bnb-nf4 as a fallback. Avoid GPTQ in 2026 unless you already have it — it's not strictly worse, but AWQ has eaten its lunch.

Code

Pick variants from the Hub·bash

# Search for AWQ variants of a model
huggingface-cli download "TheBloke/Llama-2-7B-Chat-AWQ" --local-dir ./awq

# In TGI launch:
docker run ... \
  --model-id /data/awq \
  --quantize awq

# In vLLM:
vllm serve TheBloke/Llama-2-7B-Chat-AWQ --quantization awq

Cross-format perplexity sanity check·python

# Pseudo-code: load each variant via transformers, compute logprobs on a held-out
# eval set (e.g. 100 prompts from your real traffic), and compare perplexity.
# A 4-bit AWQ model that comes within 0.2 perplexity of fp16 on YOUR distribution
# is good enough. A 4-bit model that drifts > 1 perplexity is risky.

# from transformers import AutoModelForCausalLM, AutoTokenizer
# import torch
# ... load fp16 vs awq vs gptq, compute -log p(eval_text) per token,
# average across the eval set, compare.

External links

Exercise

Pick a 7B model with both AWQ and GPTQ variants on the Hub. Serve each via vLLM. Hold out 50 prompts from your domain. Run them against fp16, awq, gptq. Compute perplexity (or just bleu/exact-match if it's a deterministic task). Pick the variant that wins.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousvLLM: PagedAttention and Continuous Batching Next →Health Checks, Metrics, and Restart Strategy

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.