The Swap Cliff

"Memory does not run out gradually. It runs out all at once, and then the SSD is your memory bus."

What Is Past the Edge

A Mac never refuses to allocate memory just because the pool is full. It has two more tiers. The first is the compressor: pages that have not been touched recently are compressed in place, trading CPU time for room, and it is why a Mac feels fine well past the point where the arithmetic says it should not. The second is swap: pages written out to the SSD and read back on demand. Both are invisible until you look, and both are catastrophic for one specific workload — a model whose weights are re-read every token — because neither tier can be streamed through at anything like memory speed.

The size of the cliff is the ratio between memory bandwidth and SSD bandwidth. The GPU track measured the Air's achieved memory bandwidth at 93–97 GB/s. A 20 GiB uncached sequential read of its SSD — a file larger than its memory, so the page cache cannot help — ran at 2.85 GB/s. That is a factor of 34. On a Mac Studio the memory side is six to eight times faster and the SSD is not, so the cliff there is a factor of a hundred or more. A model with a tenth of its weights on the wrong side of that edge decodes at a quarter of the speed (6.0 to 1.5 tok/s on the Air's numbers, by the formula in the code block); a model with half of them past it runs at a third of a token per second — for practical purposes, not running.

The Air, One Step From the Edge

The lab track pushed the 24 GB Air exactly to the edge. The 27B model — 16.05 GB on disk, 15.78 GB peak — sits inside the Air's 17.8 GiB GPU working set, and it ran: 6.09 tokens per second, at the same 88 GB/s effective bandwidth the small models achieved. No cliff. But vm.swapusage read afterwards showed 1.7 GB of swap in use and the compressor holding a further 0.9 GB: to make room for the GPU's working set, macOS had pushed everything else on the machine out to disk. The next model on the ladder, the 20 GB mixture-of-experts, would need 19.9 GB of working set on a machine that recommends 17.8, and the quest did not run it — deliberately, because a machine at the cliff can stop responding, and the Air is someone's instrument, not a test bench. That is the honest shape of the edge: the last model that fits leaves nothing behind it, and the first one that does not is not slower, it is gone.

Seeing It Before It Happens

Three readings tell you where a Mac stands. vm_stat reports pages free, active, wired, compressed and swapped. sysctl vm.swapusage reports swap in use. Activity Monitor's memory-pressure graph turns yellow when the compressor is working hard and red when swap is. The rule for an inference host is that memory pressure stays green while the model is loaded and generating at full context; yellow means the next allocation is a coin flip, and red means the model is already partly on disk and every token is paying for it. The code block reads the counters and does the cliff arithmetic for a model you name.

Code

cliff.py — SSD bandwidth uncached, the machine's swap state, and the cost of spilling·python

#!/usr/bin/env python3
"""Measure SSD sequential read with the page cache bypassed (F_NOCACHE, so the file
need not exceed RAM; 20 GiB is plenty), read the swap counters, and compute what spilling costs.
Usage: cliff.py <file GiB> <model GB> <spill GB>"""
import fcntl, os, subprocess, sys, time

file_gib = float(sys.argv[1]) if len(sys.argv) > 1 else 20
model_gb = float(sys.argv[2]) if len(sys.argv) > 2 else 16.05
spill_gb = float(sys.argv[3]) if len(sys.argv) > 3 else 1.6

path = os.path.expanduser("~/cliff-test.bin"); size = int(file_gib * 2**30)
if not os.path.exists(path) or os.path.getsize(path) != size:
    with open(path, "wb") as f:
        buf = os.urandom(1 << 20)
        for _ in range(size >> 20):
            f.write(buf)
fd = os.open(path, os.O_RDONLY); fcntl.fcntl(fd, fcntl.F_NOCACHE, 1)
t = time.perf_counter(); n = 0
while (b := os.read(fd, 8 << 20)):
    n += len(b)
ssd = n / (time.perf_counter() - t) / 1e9; os.close(fd); os.remove(path)
print(f"SSD sequential read, {file_gib:.0f} GiB uncached: {ssd:.2f} GB/s")

print(subprocess.run(["sysctl", "vm.swapusage"], capture_output=True, text=True).stdout.strip())

mem = 97.0   # your achieved memory bandwidth from stream.py (air: 97 GB/s)
fit = mem / model_gb
spilled = 1 / ((model_gb - spill_gb) / mem + spill_gb / ssd)
print(f"{model_gb:.1f} GB model: fits -> ceiling {fit:.1f} tok/s; {spill_gb:.1f} GB spilled to SSD -> {spilled:.2f} tok/s  ({fit/spilled:.0f}x slower)")

# air, 2026-09-15: SSD 2.85 GB/s; after the 27B run vm.swapusage used = 1723 MB;
# 16.05 GB model with 1.6 GB spilled: 6.0 -> 1.5 tok/s (4x); with 8 GB spilled: 0.34 tok/s

The counters, from the shell·bash

vm_stat | grep -E "Pages (free|active|wired|occupied by compressor|swapped)"
sysctl vm.swapusage
# air, after the 27B run, 2026-09-15:
# Pages occupied by compressor:  54804     <- ×16 KB = 0.9 GB compressed
# vm.swapusage: total = 3072.00M  used = 1723.56M  free = 1348.44M

# Activity Monitor > Memory > Memory Pressure: green = fine, yellow = compressor busy, red = swapping.

Exercise

Run cliff.py on your Mac (the default 20 GiB file is enough — F_NOCACHE bypasses the page cache, so it need not exceed your memory) with your own achieved bandwidth substituted for mem. Add the SSD figure and the memory-to-SSD ratio to your card. Then, for the largest model you run, compute the decode rate if 10% of it spilled, and decide: is the right response to a red memory-pressure graph a smaller model, a shorter context, or a bigger Mac?

Hint

It is almost always a smaller model or a shorter context, because the ratio is so large that no amount of tolerance survives it. The bigger Mac is the answer only when the model at the context you need is the point of owning the machine — which is the capacity-once lesson, decided at the configurator.