Quantize to Fit One Machine

"Bits are the only knob that moves a model from the store's shelf to a Mac's pool. Turn it and measure what it costs — the lab did, on a small file, in three seconds a turn."

The Ladder, Measured

Quantization is the physics track's byte count applied on purpose: fewer bits per weight, fewer bytes per token, a higher ceiling and a smaller file. The one command that does it in MLX is mlx_lm convert with a bit width and a group size; the lab ran it on office against the bf16 Llama-3.2-1B from the journey track, four times, and timed each conversion at three seconds. The results are the whole lesson in a row: bf16 2.47 GB decodes at 188 tokens per second; 8 bits, 1.2 GB, 292; 6 bits, 974 MB, 330; 4 bits, 680 MB, 429; 3 bits, 532 MB, 462. Each step down the ladder reads fewer bytes per token, and decode rises with it — nearly linearly until the fixed cost from the bandwidth lesson takes over at the bottom, where 3 bits buys only 8% over 4. The converter's own log states the real bits per weight — 4.501 at nominal 4, 8.500 at nominal 8 — which is the scales-and-biases overhead — 0.5625 bytes per parameter, 424 GB for GLM-5.3 — where the fit lesson's rounder 0.6 (452 GB) also covers the tensors a converter leaves unquantized; both land inside 498. And the 4-bit file it produced decoded at 428.6 tokens per second against 428.0 for the community's 4-bit build of the same model, which is the lab's way of saying the command is the same command.

Llama-3.2-1B on office	Bits per weight (converter's count)	File	Decode	vs bf16	Evidence
bf16 as shipped	16	2.47 GB	187.8 tok/s	1.0×	measured 2026-09-15
8-bit, group 64	8.500	1.2 GB	292.2	1.6×	measured
6-bit	6.501	974 MB	330.1	1.8×	measured
4-bit	4.501	680 MB	428.6	2.3×	measured
3-bit	3.501	532 MB	461.7	2.5×	measured

The Same Ladder on the Store's Files

The code block applies the converter's own bits-per-weight to the store's checkpoints. GLM-5.3 at 8 bits is 800 GB and at 6 is 612 — neither fits a 512 GB Studio — and at 4 bits it is 424, which does, with the fit lesson's cache on top. GLM-5.3-Flash fits at 8 bits, 340 GB, and can keep most of its precision; Qwen3.8-Flash-Next fits at 8 bits at 191 GB and at 4 bits in a laptop. Kimi K3 fits at nothing: 3 bits is still 1.2 TB. So the store's shelf sorts itself by the first bit width at which each file crosses the working-set line, and the household's decision for each is a row of this table plus a quality judgment the table cannot make.

What the Quest Did Not Measure

Two things, and the card must say both. Quality: fewer bits cost accuracy, and how much depends on the model, the task and the method — the lab decoded faster at 3 bits and did not evaluate a single answer, because the quest is about the chip; the mlx quest's quantization lessons and the model's own community evaluations are where that judgment lives. And the big conversion itself: three seconds for 2.5 GB scales, on paper, to about half an hour of converter time for 1.5 TB — and reading the original once at the Air's measured SSD rate is nine minutes of that — but whether the converter needs the whole bf16 set resident at once on a 512 GB Mac was not tested on a 1.5 TB input. A reader with the store's GLM-5.3 and a Studio would find out in an afternoon; the experiment-line lesson is about what to do with the afternoon's result.

Code

quantize_plan.py — the measured bits ladder, and the same arithmetic on the store's files·python

#!/usr/bin/env python3
"""Quantize to fit one machine. The bits ladder measured on a small checkpoint on office
(one command, seconds), then the same arithmetic applied to the store's big files. The
store is not a quantizer; this runs on a Mac that is."""
import subprocess, sys
# --- the command, run for real on a 1B so its cost is known (office, 2026-09-15, mlx-lm 0.31.3) ---
# python -m mlx_lm convert --hf-path unsloth/Llama-3.2-1B-Instruct --mlx-path q4-llama -q --q-bits 4 --q-group-size 64
# bf16 2.47 GB -> 3 bits: 532 MB (3.501 b/w), 461.7 tok/s | 4 bits: 680 MB (4.501), 428.6 | 6 bits: 974 MB (6.501), 330.1
#                        8 bits: 1.2 GB (8.500), 292.2 | bf16 as shipped: 187.8 tok/s.  3 s per conversion.
BW, WS = 819.0, 498.0
STORE = [("GLM-5.3", 753, 1506.7), ("GLM-5.3-Flash", 320, 642.7), ("Kimi K3", 2800, 1561.0), ("Qwen3.8-Flash-Next", 180, 360.0)]
print(f"{'checkpoint':20} {'params B':>8} {'bf16 GB':>8} | " + " ".join(f"{b}-bit GB" for b in (8, 6, 4, 3)) + "   fits 498 at")
for name, params, bf16 in STORE:
    sizes = {b: params * (b + 0.5) / 8 for b in (8, 6, 4, 3)}                     # b bits + ~0.5 for scales/biases at group 64
    fits = next((f"{b} bits" for b in (8, 6, 4, 3) if sizes[b] + 16 <= WS), "nothing")
    print(f"{name:20} {params:8d} {bf16:8.0f} | " + " ".join(f"{sizes[b]:8.0f}" for b in (8, 6, 4, 3)) + f"   {fits}")
print("\nreading the 1.5 TB original once at the Air's measured 2.85 GB/s is ~9 minutes; a Studio SSD is faster and unmeasured here.")
print("whether the converter needs the whole BF16 set resident at once on a 512 GB Mac was not tested on a 1.5 TB input -- the 2.5 GB one fit trivially.")

Exercise

Take one bf16 checkpoint you have and run mlx_lm convert at 8, 6, 4 and 3 bits. Record the converter's bits-per-weight line, the file size and the decode rate for each on your card. Then write which width you would actually run and why — the fit, the speed, or an evaluation you read.

Hint

Expect the decode ladder to flatten at the bottom on a small model and stay steep on a large one; expect the file to be about (bits + 0.5) ÷ 8 bytes per parameter. If a step down does not speed decode at all, the fixed cost owns that model and the width should be chosen for quality alone.