Quantization Basics — Q4, Q8, and What You're Actually Losing

What quantization actually does

A model's weights are floating-point numbers. fp32 uses 4 bytes per number; fp16/bf16 uses 2 bytes; int8 / Q8 uses 1 byte; Q4 uses half a byte. Quantization replaces high-precision weights with low-precision approximations, trading a bit of model quality for a large reduction in disk size and memory footprint.

For a 7B model, the difference is dramatic. At fp16 you need ~14 GB just for weights. At Q8, ~7 GB. At Q4, ~3.5 GB. Same model, three different memory budgets.

What "Q4" actually means

The naive interpretation is "every weight is a 4-bit integer." The real story is more nuanced. MLX (and most modern quantization schemes) store weights in groups. Each group of 32 / 64 / 128 weights shares a scale factor and a zero point — those are stored at higher precision (typically fp16). Inside a group, individual weights are quantized to 4 bits. At inference time, weights are reconstructed by dequantized = scale * (quantized_int4 - zero_point).

This is called affine quantization and it's what --q-mode affine (the default) gives you. The newer MX modes (mxfp4, nvfp4, mxfp8) use micro-scaling formats that map to specialized hardware on the latest GPUs.

Q4 vs Q8 — what you give up

Disk and memory: Q4 is roughly half the size of Q8 (for the same model).
Inference speed: Q4 is sometimes slightly faster (less memory bandwidth needed), sometimes slightly slower (extra dequantization work). Treat it as roughly comparable.
Quality: Q4 loses small amounts of perplexity / benchmark score on most tasks, more on some (long-form reasoning, low-resource languages). For chat use cases on instruct models, Q4 is usually indistinguishable from Q8 in casual evaluation; benchmarks tell a more nuanced story.
Edge cases: very small models (1B and below) sometimes degrade more at Q4 than larger models — there are fewer parameters to absorb the precision loss. Think Q8 for sub-1B; Q4 for 3B+.

The verdict you can carry around

For 7B+ instruct models on Apple Silicon, Q4 is the default — it fits more memory budgets and the quality loss is rarely the bottleneck. Step up to Q8 only when you're benchmarking precision-sensitive workloads or running a sub-3B model where the loss matters. Skip quantization (use bf16/fp16) only when you have memory to burn and a specific reason to care about that last sliver of quality.

Code

Quantize the same model to Q4 and Q8 — compare on disk·bash

# Convert at Q4
python -m mlx_lm convert \
  --hf-path meta-llama/Llama-3.2-1B-Instruct \
  --mlx-path ./Llama-3.2-1B-Instruct-Q4 \
  --quantize --q-bits 4 --q-group-size 64

# Convert at Q8
python -m mlx_lm convert \
  --hf-path meta-llama/Llama-3.2-1B-Instruct \
  --mlx-path ./Llama-3.2-1B-Instruct-Q8 \
  --quantize --q-bits 8 --q-group-size 64

# Compare on disk
du -sh ./Llama-3.2-1B-Instruct-Q4 ./Llama-3.2-1B-Instruct-Q8

# Sample (1B Llama variant):
#   ~700 MB for Q4
#   ~1.3 GB for Q8

Eyeball quality on the same prompt·python

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler
import mlx.core as mx

prompt = "Explain the difference between weather and climate in two sentences."
sampler = make_sampler(temp=0.0)   # greedy for fair comparison

for path in ["./Llama-3.2-1B-Instruct-Q4", "./Llama-3.2-1B-Instruct-Q8"]:
    print(f"--- {path} ---")
    model, tok = load(path)
    mx.random.seed(42)
    print(generate(model, tok, prompt=prompt, max_tokens=80, sampler=sampler, verbose=False))
    print()

# In casual reading the Q4 and Q8 outputs are usually indistinguishable;
# the differences appear in benchmark scores, not in this kind of demo.

Exercise

Convert any small base model at three quantization levels: Q4 group-size 32, Q4 group-size 128, and Q8 group-size 64. Run the same prompt through all three (greedy sampling for fair comparison). Compare disk sizes and read the outputs. The exercise is to feel — qualitatively — how much smaller Q4 gets, how much group size matters, and whether you can distinguish the outputs by reading them.