Quantization — From fp32 to int8 (and int4)

Trade a tiny accuracy budget for huge size and speed wins

Modern hardware is much faster at int8 matmul than fp32 — sometimes 4x. Memory consumption drops proportionally. The cost: a small accuracy hit, usually <1% on standard benchmarks for well-quantized models. For LLMs, int4 weight-only quant (one of the most active areas in 2025-2026) lets you fit 7B-param models in 4GB.

Three flavors of quantization

Dynamic quantization — weights stored as int8, activations quantized on-the-fly during inference. One-line setup. Best for transformer-shaped models dominated by Linear layers.
Static (post-training) quantization (PTQ) — both weights and activations quantized, calibrated on a small dataset. Faster than dynamic but more setup.
Quantization-Aware Training (QAT) — train with simulated quantization in the forward pass. Best accuracy but slowest to set up.

torchao — the modern API

The historical torch.quantization / torch.ao.quantization modules are being migrated to the standalone torchao package. torchao is where modern int8 / int4 / weight-only / GPTQ / AWQ techniques live. For new projects, start there.

Where quantization helps and hurts

Helps: large transformer FFN, big embedding tables, LLM serving.
Hurts: tiny models where the per-op overhead dominates, models with many non-Linear ops (the unquantized portions become the bottleneck).

Code

Dynamic int8 quantization — one liner for transformers·python

import os
import torch
import torch.nn as nn
from transformers import AutoModelForSequenceClassification

model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased-finetuned-sst-2-english"
)
model.eval()

quantized = torch.quantization.quantize_dynamic(
    model, {nn.Linear}, dtype=torch.qint8,
)

# Compare disk size
torch.save(model.state_dict(), '/tmp/fp32.pt')
torch.save(quantized.state_dict(), '/tmp/int8.pt')
print(f"fp32: {os.path.getsize('/tmp/fp32.pt')/1e6:.1f} MB")
print(f"int8: {os.path.getsize('/tmp/int8.pt')/1e6:.1f} MB")

torchao — int8 weight-only quantization·python

# pip install torchao
import torch
import torchao
from torchao.quantization import int8_weight_only
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("gpt2")

# Apply int8 weight-only quant in place
torchao.quantize_(model, int8_weight_only())

# Use the model normally — the quantized layers handle dequant in their forward

torchao — int4 weight-only for LLMs·python

import torchao
from torchao.quantization import int4_weight_only
from transformers import AutoModelForCausalLM

# int4 — even smaller, designed for LLMs
# group_size controls the granularity: smaller groups = better accuracy, more overhead
model = AutoModelForCausalLM.from_pretrained("gpt2")
torchao.quantize_(model, int4_weight_only(group_size=128))

# A 7B-param model at fp16 = ~14GB; at int4 = ~3.5GB
# That's the difference between "needs an A100" and "fits on a 4090"

Validate accuracy after quantization — always·python

import torch

# Run both fp32 and quantized model on a calibration / val set
# Compare outputs on a per-sample basis

def compare_models(fp32_model, quant_model, val_loader):
    fp32_model.eval(); quant_model.eval()
    abs_diff_total = 0
    n = 0
    with torch.inference_mode():
        for x, y in val_loader:
            out_fp = fp32_model(x).logits
            out_q  = quant_model(x).logits
            abs_diff_total += (out_fp - out_q).abs().mean().item()
            n += 1
    print(f"mean |fp32 - quantized| logit diff: {abs_diff_total / n:.4f}")
    # Also recompute task accuracy on both — that's the number that matters

Quantization — From fp32 to int8 (and int4)

Trade a tiny accuracy budget for huge size and speed wins

Three flavors of quantization

torchao — the modern API

Where quantization helps and hurts

Code

External links

Exercise

Progress

Comments 0