Quantization Buys Room and Speed Together

"The same bytes that decide whether a model fits decide how fast it runs. Shrink them and you get both — and you pay in a currency this quest refuses to price."

One Lever, Two Effects

A model trained in 16-bit floats stores two bytes per weight. Quantization stores fewer — four bits per weight in every checkpoint the lab ran — by keeping, for each small group of weights, a scale and an offset in higher precision and packing the weights themselves as small integers. The memory track showed that fit is decided by bytes and the previous lessons showed that decode is decided by bytes per token; those are the same bytes. Quantization therefore moves both numbers at once: a 27B model at bf16 is 54.7 GB and cannot load on a 24 GB Mac or a 32 GB card; at 4 bits it is 16.05 GB and its decode ceiling is 3.4 times higher on the same bus.

Read From the Files

The code block counts the weights in a checkpoint's safetensors headers — packed 4-bit weights hold eight values per 32-bit word — and divides the file size by the count:

Checkpoint	Format (config)	Parameters	File	Effective bits per weight	Smaller than bf16	Evidence
Qwen3.5-27B-4bit	4-bit affine, group 64	27.36 B	16.05 GB	4.69	3.41x	physics from the headers
Qwen3.5-9B-4bit	4-bit affine, group 64	9.41 B	5.95 GB	5.06	3.16x	physics
Llama-3.2-1B-Instruct-4bit	4-bit, group 64	1.24 B	0.70 GB	4.50	3.56x	physics

The effective figure is above 4 for two reasons that are worth seeing rather than assuming. Every group of 64 weights carries a 16-bit scale and a 16-bit bias — half a bit per weight of overhead, which is the 4.50 on the Llama. And parts of a checkpoint are left at higher precision: the 9B's untied embedding and its vision encoder push it to 5.06. So "4-bit" is a name for the dominant tensors, and the file is the truth about the rest.

The Landscape, 2026

Quantization is no longer only a downloader's afterthought. NVIDIA's Blackwell hardware computes natively in an FP4 format (its "NVFP4", introduced with that architecture, with FP8 on both Hopper and Blackwell). Two of the household's stored checkpoints ship quantized by their makers: DeepSeek-V4.1-Flash with FP4 experts and FP8 attention in a 510 GB file that would be over a terabyte at bf16, and Kimi K3 with MXFP4 experts from quantization-aware training at 1.56 TB. Apple's own on-device model uses a compression scheme decoded by "a dedicated hardware component in Apple GPUs". On the Mac, MLX supports several bit widths and group sizes, and mlx-community publishes the 4-bit files the lab used. The direction is clear: the bytes per weight are becoming a design parameter of the model, not a post-processing choice.

What This Quest Will Not Say

Every quantization costs something in the model's outputs, and the size of that cost depends on the model, the method, the bit width, the task and the prompt. A number for it would be a quality score, and this quest carries none — the household's rule, measured and closed elsewhere, is that quality is not scanner-rankable. What the quest does say is the physics: a 4-bit file fits where a bf16 file does not, decodes about three and a half times as fast on the same bus, and is the reason every model in the fleet's ladder ran at all on a 24 GB Air. Whether the answers are good enough for your work is something you judge by reading them, on your task, and the honest label for that judgement is your own.

Code

quant_bits.py — effective bits per weight, from the safetensors headers·python

#!/usr/bin/env python3
"""Effective bits per weight of a quantized checkpoint, from the safetensors
headers: packed uint32 weights hold 32/bits values each; the scales and biases
per group are the overhead that turns 4 bits into ~4.5."""
import json, struct, sys
from pathlib import Path
from huggingface_hub import snapshot_download

repo = sys.argv[1] if len(sys.argv) > 1 else "mlx-community/Qwen3.5-27B-4bit"
path = Path(snapshot_download(repo, local_files_only=True))
cfg = json.loads((path / "config.json").read_text())
q = cfg.get("quantization") or cfg.get("text_config", {}).get("quantization") or {}
bits = q.get("bits", 16)

params = bytes_total = 0
for f in sorted(path.glob("*.safetensors")):
    with f.open("rb") as fh:
        header = json.loads(fh.read(struct.unpack("<Q", fh.read(8))[0]))
    for name, meta in header.items():
        if name == "__metadata__":
            continue
        n = 1
        for d in meta["shape"]:
            n *= d
        bytes_total += meta["data_offsets"][1] - meta["data_offsets"][0]
        if name.endswith(".weight") and meta["dtype"] == "U32":
            params += n * (32 // bits)          # packed quantized weights
        elif name.endswith((".scales", ".biases")):
            pass                                # quantization overhead: bytes, not weights
        else:
            params += n                          # unquantized tensors (norms, some embeddings)

print(f"{repo}: {q or 'unquantized'}")
print(f"  parameters {params/1e9:.2f} B, file {bytes_total/1e9:.2f} GB -> {bytes_total*8/params:.2f} bits per weight overall")
print(f"  bf16 would be {params*2/1e9:.1f} GB; this file is {params*2/bytes_total:.2f}x smaller -> ceiling {params*2/bytes_total:.2f}x higher")

# office, 2026-09-15:
# Qwen3.5-27B-4bit: 27.36 B params, 16.05 GB -> 4.69 bits/weight; bf16 54.7 GB; 3.41x
# Qwen3.5-9B-4bit:   9.41 B params,  5.95 GB -> 5.06 bits/weight; bf16 18.8 GB; 3.16x
# Llama-3.2-1B-4bit: 1.24 B params,  0.70 GB -> 4.50 bits/weight; bf16  2.5 GB; 3.56x

Exercise

Run quant_bits.py on every checkpoint you have. For one model available in both a 4-bit and an 8-bit (or bf16) build, compute the ceiling ratio, then measure both with the lab's ladder script and record the real ratio. Finally, write one sentence about a difference you noticed in the outputs — in words, without a number.

Hint

The ceiling ratio is bytes ÷ bytes; the measured ratio will be smaller on a small model because fixed overhead does not shrink with the weights. The sentence about outputs is the part that matters and the part no script can write for you.