Precision Playbook on Apple Silicon

The precision ladder — and the memory ceiling each step lifts

On a Mac Studio with 192–512 GB unified memory, the question 'will my model fit?' is almost always answered by precision, not by FP32 raw weight count. The ladder:

Precision	tok/s (typical 7B)	Memory footprint	Bottleneck	When to pick
FP32	4–6	26 GB (7B)	Memory	Debugging, parity checks against numpy reference
BF16 / FP16	8–12	13 GB	Compute	Fine-tune, serve medium models
Q8_0 (Int8)	12–18	7 GB	L2 hits + DRAM BW	Default deployment
Q4_K (Int4)	17–25	4 GB	Compute (dequant)	Large models, single-node serving

Three things to internalize:

Each step roughly halves memory. A 70B FP32 model needs 280 GB; FP16 brings it to 140 GB; Int4 brings it to 40 GB — Mac Studio 192 GB territory.
Speed gains aren't free. FP16 → Int8 is faster mostly because there's less data to fetch from DRAM. Int8 → Int4 is faster and compute-bounded again because dequant is FLOPs.
Quality degrades non-linearly. Going FP32 → BF16 is essentially free for inference. BF16 → Int8 is small. Int8 → Int4 starts to hurt some models more than others; benchmark your specific workload.

The memory hierarchy that backs each precision

Quantized weights live in unified DRAM. Hot tiles get pulled into L2 / threadgroup memory during a kernel — and that's the cache that decides whether your kernel hits its precision's compute ceiling or stalls on misses.

Code

Precision conversion in MLX — one line each·python

import mlx.core as mx

weight_fp32 = mx.array(np.load('weight.npy'))   # whatever the source is

# Cast paths
weight_bf16 = weight_fp32.astype(mx.bfloat16)
weight_fp16 = weight_fp32.astype(mx.float16)

# Quantize to int4 with mlx.nn — the standard path for serving large models
import mlx.nn as nn
linear = nn.Linear(input_dims=4096, output_dims=4096, bias=False)
linear.weight = weight_fp32
nn.quantize(linear, group_size=64, bits=4)   # in-place, replaces fp32 with int4 + scales

MLX-LM serving pattern — pick precision at load time·bash

# Q4 is the typical 'serves 70B on a Mac Studio' setting:
mlx_lm.generate \
    --model mlx-community/Llama-3.1-70B-Instruct-4bit \
    --prompt 'What is the unified memory advantage on Apple Silicon?' \
    --max-tokens 256

# FP16 — needs ~140 GB on a 70B model, fits a 192 GB Mac Studio:
mlx_lm.generate \
    --model mlx-community/Llama-3.1-70B-Instruct \
    --prompt '...' --max-tokens 256

# Pin a specific dtype if the repo has multiple variants:
mlx_lm.generate --model <repo> --dtype float16

Exercise

On any Mac with MLX-LM installed, run the same prompt at FP16 and Q4 against a 7B-class model. Capture: (a) tokens/sec, (b) memory used, (c) qualitative output difference. Q4 should be ~2× faster and ~4× smaller while producing nearly identical output for short prompts. If you don't see that, your model has poor Q4 calibration; try a different community-quantized variant.