Performance & Memory — What Eats Your RAM and Why

Three things eat your unified memory during inference

Model weights. The big constant. A 7B Q4 model uses ~5 GB of weights; a 70B Q4 uses ~50 GB. This is the number you computed in foundations.lesson4 with the napkin formula.
The KV cache. A growing buffer that holds the keys and values for every token in the context window. Linear in context length, plus a per-layer multiplier. For a 7B model with a 32k context, the KV cache can be larger than the model itself in extreme cases.
Activation memory. Transient buffers used during a single forward pass. Smaller than the cache for inference, but real.

Plus macOS reserves wired memory for the OS itself and other apps. prod.lesson2 covers the wired-memory ceiling in detail. This lesson focuses on the part you control: KV cache size and how to measure it.

Measuring memory the right way

MLX exposes two helpful primitives: mx.get_active_memory() (currently allocated GPU memory in bytes) and mx.get_peak_memory() (peak since the last reset, in bytes). Reading these around a generation tells you exactly how much memory the model + KV cache + activations consumed.

For the 1B Q4 demo model on the office Mac, the verified numbers are:

After load: ~663 MB active
After short generation: ~663 MB active
After 200-token generation: ~663 MB active, ~685 MB peak

The KV cache for a 1B model is small enough that it doesn't dominate memory at typical context lengths. Scale up to a 7B+ model with a 32k context and the KV cache becomes the load-bearing number, not the weights.

The wired-memory ceiling, briefly

macOS doesn't let the GPU lock 100% of unified memory. The default iogpu.wired_limit_mb caps the GPU's accessible memory at a tier-dependent fraction (varies between Mac models). On a 192 GB Mac Studio, the ceiling is a hair under 192 GB but not exactly 192 GB. If a model fits in your napkin math but doesn't fit in actual generation, the wired-memory cap is usually why. prod.lesson2 has the full diagnostic.

What you do about all this

Pick a model size that fits in roughly 70% of your unified memory after weights and KV cache estimate (rule from foundations.lesson3).
For long contexts, use a smaller-quantization model — Q4 over fp16 — to leave room for the cache.
Measure with mx.get_peak_memory() before declaring a model "fits." Napkin math is reliable to ±10%; actual peaks can surprise you.
If you're hitting the ceiling, drop quantization or shorten context before reaching for system-level tweaks.

Code

Measure memory around model load and generation·python

import mlx.core as mx
from mlx_lm import load, generate


def mem_mb():
    return mx.get_active_memory() / (1024 * 1024)


def peak_mb():
    return mx.get_peak_memory() / (1024 * 1024)


# Reset peak counter and measure baseline
mx.reset_peak_memory()
print(f"baseline                    : active {mem_mb():.1f} MB, peak {peak_mb():.1f} MB")

model, tok = load("mlx-community/Llama-3.2-1B-Instruct-4bit")
print(f"after load                  : active {mem_mb():.1f} MB, peak {peak_mb():.1f} MB")

generate(model, tok, prompt="Hello.", max_tokens=10, verbose=False)
print(f"after short generate (10)   : active {mem_mb():.1f} MB, peak {peak_mb():.1f} MB")

generate(model, tok, prompt="Tell me a story:", max_tokens=200, verbose=False)
print(f"after longer generate (200) : active {mem_mb():.1f} MB, peak {peak_mb():.1f} MB")

# Verified output (2026-05-03, M3 Ultra Studio):
#   baseline                    : active 0.0 MB, peak 0.0 MB
#   after load                  : active 663.0 MB, peak ~675 MB
#   after short generate (10)   : active 663.0 MB, peak ~680 MB
#   after longer generate (200) : active 663.0 MB, peak ~685 MB

Inspect the wired-memory ceiling on your Mac·bash

# How much unified memory total?
sysctl -n hw.memsize | awk '{ printf "Unified memory: %.1f GB\n", $1/1024/1024/1024 }'

# What's the GPU wired-memory cap (the hard ceiling for MLX kernels)?
sysctl iogpu.wired_limit_mb
# A value of 0 means "system default" — typically a fraction (e.g. ~75%) of total RAM.
# Non-zero values are explicit overrides; raising this is reckless on small Macs.

# Sample (M3 Ultra Studio, 512 GB):
#   Unified memory: 512.0 GB
#   iogpu.wired_limit_mb: 0      (using system default ceiling)

Exercise

Run the memory measurement block on your Mac with the same 1B Q4 model. Then change the model to mlx-community/Llama-3.2-3B-Instruct-4bit (or any 3B Q4 you have cached) and re-run. Compare the post-load active memory and the peak after a 200-token generation. The ratio should be close to 3× since the model is roughly 3× larger; if it's much higher, the KV cache is starting to assert itself. Two sentences on what you measured.