What quantization buys
Quantization compresses model weights to fewer bits per parameter. Trading precision for capacity is the single most effective lever for fitting a model on consumer hardware. A 7B model at FP32 wants 28 GB; the same model at Q4 wants ~4 GB and runs visibly faster because less memory has to stream per token.
The quant ladder you'll actually use
| Format | Bits | 7B size | Quality | When to pick |
|---|---|---|---|---|
| FP16 / BF16 | 16 | ~14 GB | Negligible loss | Eval rigs, fine-tuning prep |
| Q8_0 | 8 | ~7 GB | Near-lossless | Final answer machines with headroom |
| Q6_K | ~6 | ~5.5 GB | Good balance | Quality-first daily driver |
| Q5_K_M | ~5 | ~4.8 GB | Acceptable | Big models on tight machines |
| Q4_K_M | ~4 | ~4 GB | Sweet spot | Default daily driver |
| IQ4_XS | ~4 | ~3.6 GB | Smarter than Q4_0 | Squeezing a quant smaller |
| Q3_K_M | ~3 | ~3 GB | Noticeable loss | Last resort to fit at all |
| Q2_K | ~2 | ~2 GB | Significant loss | Demos / curiosity only |
K-quants vs I-quants
- K-quants (e.g.
Q4_K_M,Q5_K_S) split the tensor into blocks, then assign each block its own scale and bias. Better than uniform Q4_0 at the same average bit-width. - I-quants (e.g.
IQ4_XS,IQ3_XXS) use an importance matrix from calibration data to spend bits on the weights that matter. Smaller files at similar quality, but slower to decode on some hardware.
Operating heuristic
Always pick the largest quant that still leaves 15% memory headroom with your target context window. Q4_K_M is the right starting point on consumer hardware; step up to Q5/Q6/Q8 if the model fits, step down to IQ4_XS or Q3_K_M only when you can't fit otherwise.