Quantization Formats Cheat Sheet

~24 min · ops, quantization

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The format constellation, by goal

Goal	Format	Why
Train smaller	bnb-nf4 (QLoRA)	Designed for training. NF4 fits normally distributed weights well.
Serve on GPU (TGI/vLLM)	AWQ, GPTQ	Engine-aware, fast inference kernels.
Serve on Hopper/Ada	fp8	Native fp8 tensor cores.
Run locally on Mac/Linux laptop	GGUF (via Ollama / llama.cpp)	Optimized for CPU + Metal + lite GPU.
Run on Apple Silicon natively	MLX	Native Metal-accelerated kernels, unified memory.
Run on edge / mobile	ONNX (with INT8/INT4)	Cross-runtime portability.

The mental model

Quantization formats encode bits-per-weight, calibration scheme, and runtime expectations. Mismatch any of those and the model fails to load or runs slowly. Always check that your serving engine matches the format's runtime: vLLM groks AWQ/GPTQ; TGI groks both plus bnb; Ollama wants GGUF; MLX wants .mlx.

Code

Load a 4-bit bnb quantized model with transformers·python

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                          bnb_4bit_compute_dtype=torch.bfloat16)

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-3.1-8B-Instruct",
    quantization_config=bnb,
    device_map="auto",
)

Convert a Hub model to ONNX·python

# pip install optimum[onnxruntime]
from optimum.onnxruntime import ORTModelForCausalLM
from transformers import AutoTokenizer

repo = "Qwen/Qwen2.5-1.5B-Instruct"
ort_model = ORTModelForCausalLM.from_pretrained(repo, export=True)
ort_model.save_pretrained("./qwen-onnx")

# Now load it in any ONNX runtime

External links

Exercise

Take a 1B-3B model. Produce four quantized variants of it: bnb-nf4, AWQ (if a variant exists or via autoawq), GGUF (via llama.cpp's converter), ONNX (via optimum). Note: which formats can be re-uploaded to the Hub, which loaders read each, and final disk size for each.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousSafetensors: Why It's the Default Next →GGUF Export for Ollama / llama.cpp

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.