Why 192 GB (and 512 GB) Changes the Conversation

Why memory size is the conversation

For most of deep learning's history, the conversation about "what hardware do you need" was a conversation about how many GPUs. The unspoken assumption was that any individual GPU's VRAM was small (16, 24, sometimes 80 gigabytes), so to hold a big model you spread it across N of them. That's why the canonical "can you run a 70B model?" answer used to be "yes, on an 8x A100 rack."

Apple Silicon's high-memory tiers ate that conversation. A single M-Ultra Studio with 192 GB or 512 GB of unified memory holds frontier-size models in one piece of hardware that fits in a desk drawer. Same model, no model parallelism, no NVLink, no rack. This lesson is the napkin math that makes that real.

The formula

For a transformer language model, the rough memory footprint at inference is:

memory ≈ params × bytes_per_param × overhead_factor

Where bytes_per_param depends on quantization (4 for fp32, 2 for fp16/bf16, 1 for int8/Q8, 0.5 for Q4), and overhead_factor accounts for the KV cache, activations, and framework allocator slack. 1.4× is a reasonable starting point for inference; for training (with optimizer state and gradients), use 4–6× instead.

The numbers, computed

The code block below computes the napkin math for a few representative model sizes and quantization levels. The output is what fits on which Mac tier.

What that table is actually telling you

A 7B model at Q4 (~5 GB) runs on essentially any M-series Mac. This is what "local LLMs on a laptop" looks like.
A 70B model at fp16 (~196 GB) needs a 192 GB tier and is still tight. At Q4 (~49 GB) it fits in a 64 GB MacBook Pro.
A 180B model at Q4 (~126 GB) needs the 192 GB tier — comfortable, with overhead.
A 405B model at Q4 (~284 GB) needs the 512 GB Studio. This is the tier that runs frontier-class models in a single box.

That last line is the conversation-changer. Until 2025–2026, running a 405B model meant either renting cloud GPUs or owning rack-mounted enterprise hardware. The M3 Ultra Studio at 512 GB does it on a desk, on wall power, with no networking required. MLX is the software half of that capability.

The training caveat (so you don't get sloppy)

Everything above is inference math. Full fine-tuning a 70B model needs the weights, plus gradients (same size), plus optimizer state (Adam adds 2× the weight size), plus activations. That's why Track 4 leans hard on LoRA / QLoRA — adapter-based fine-tuning is what brings 70B fine-tuning into the 64 GB regime instead of the 256 GB regime.

Code

Napkin math: which model fits where·python

def model_size_gb(num_params_b: float, dtype_bytes: float, overhead: float = 1.4) -> float:
    """Approximate inference-time memory footprint, in gigabytes."""
    return num_params_b * 1e9 * dtype_bytes / 1e9 * overhead

rows = [
    ('7B',   7,   2.0,  'fp16/bf16'),
    ('7B',   7,   0.5,  'Q4'),
    ('13B',  13,  2.0,  'fp16/bf16'),
    ('13B',  13,  0.5,  'Q4'),
    ('70B',  70,  2.0,  'fp16/bf16'),
    ('70B',  70,  0.5,  'Q4'),
    ('180B', 180, 0.5,  'Q4'),
    ('405B', 405, 0.5,  'Q4'),
]

print(f"{'model':6} {'fmt':10} {'GB (1.4x overhead)':>22}")
for label, p, b, fmt in rows:
    print(f'{label:6} {fmt:10} {model_size_gb(p, b):>22.1f}')

# Output (verified 2026-05-03):
# model  fmt        GB (1.4x overhead)
# 7B     fp16/bf16                19.6
# 7B     Q4                        4.9
# 13B    fp16/bf16                36.4
# 13B    Q4                        9.1
# 70B    fp16/bf16               196.0
# 70B    Q4                       49.0
# 180B   Q4                      126.0
# 405B   Q4                      283.5

Sanity check against your own machine·bash

# Your unified memory in GB (rounded down)
sysctl -n hw.memsize | awk '{ printf "Unified memory: %.0f GB\n", $1/1024/1024/1024 }'

# Sample (M3 Ultra Studio):
#   Unified memory: 512 GB

# Compare to the table above and the 70%-of-RAM rule of thumb (leave 30% for
# the OS, KV cache headroom, and other apps).

Exercise

Run the napkin-math code block. Now compute one custom row that's specific to your machine: pick one model from the Hugging Face mlx-community org you'd like to run, look up its parameter count and the published quantization levels, and compute the inference footprint with the 1.4× overhead. Compare against your unified memory (use the sanity-check command) and the 70% rule. Will it fit? Will it fit with room to also keep an editor open?