Quantization: Q4, Q5, Q8 in Practice

Quantization reduces the precision of model weights to save memory and accelerate inference. Modern post-training quantization techniques (GPTQ, AWQ, GGUF) lose surprisingly little quality.

Quantization	Bits / param	7B model size	Quality impact
FP16	16	14 GB	Baseline
INT8 / Q8	8	7 GB	Near-zero quality loss
Q5	5	~4.4 GB	Very minor; often imperceptible in chat
Q4	4	3.5 GB	Small loss (~1-3%); the sweet spot for most uses
Q3	3	~2.6 GB	Noticeable degradation
Q2	2	~1.8 GB	Major degradation

Format families

GPTQ (Frantar et al., 2022): post-training, calibration-data based, 4-bit weights with FP16 activations. Standard for serving.
AWQ (Lin et al., 2023): activation-aware weight quantization. Identifies salient weights via activation magnitudes and protects them. Often slightly higher quality than GPTQ at the same bit width.
GGUF: file format used by llama.cpp. Supports many quantization schemes (Q4_K_M, Q5_K_S, Q8_0, etc.). Common for on-device inference.

For most production use, Q4 is the right starting point: fits a 70B model on a single high-end GPU, quality loss is typically 1-3 points on benchmarks, often invisible in chat. Drop to Q5 or INT8 when quality matters more; drop to Q3 only when memory is the absolute constraint.

Code

Loading a quantized model·python

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

# Llama 3.3 70B in 4-bit via bitsandbytes
model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.3-70B-Instruct",
    quantization_config={"load_in_4bit": True,
                         "bnb_4bit_compute_dtype": torch.bfloat16},
    device_map="auto",
)
# Now ~35 GB instead of ~140 GB; runs on a single 80GB GPU.

GGUF / llama.cpp on Mac (no GPU)·python

# Install llama-cpp-python, download a GGUF file (Q4_K_M is a great default)
from llama_cpp import Llama

llm = Llama(
    model_path="llama-3.3-70b-instruct.Q4_K_M.gguf",
    n_ctx=8192, n_gpu_layers=-1,
)
print(llm("Explain attention in three sentences:")["choices"][0]["text"])

Quantization: Q4, Q5, Q8 in Practice

Format families

Code

External links

Exercise

Progress

Comments 0