One Pool, Two Processors

"The hardware boundary is gone. Whether your framework noticed is a separate question."

The Same Bytes, Two Ways In

On a discrete-GPU machine a tensor lives in one of two places, host memory or device memory, and every framework has a verb for moving it across — .to('cuda'), cudaMemcpy — because the move is a physical event on a bus. On Apple silicon there is one pool, and a byte in it is reachable by the CPU and the GPU at the same address. MLX's documentation states the consequence exactly: "The CPU and GPU have direct access to the same memory pool … rather than moving arrays to devices, you specify the device when you run the operation. Any device can perform any operation on a and b without needing to move them from one memory location to another." There is no .to() in MLX. There is a stream, and a stream is a choice of which processor runs the next operation on data that stays put.

The code block shows it on office: one array, summed on the GPU stream and then on the CPU stream, with no transfer between. That is zero-copy in the literal sense, and it is the property every later argument about unified memory rests on — a model's weights are loaded once into the pool, the GPU reads them for decode, and if a CPU fallback or a tokenizer needs to look at the same bytes, it looks.

The Software Boundaries That Survive

Here is the part that surprises people. Run the second block: PyTorch's .to('mps') on a 1 GiB tensor takes about 66 milliseconds on office, and .to('cpu') 33. Nothing crossed a bus — there is no bus — but PyTorch's MPS backend keeps its own Metal buffers, so "moving" a tensor is a memcpy from one region of the pool to another. It is a software boundary the framework kept because its programming model was born on discrete cards. And converting a NumPy array into an MLX array, mx.array(np_array), took 250 milliseconds for the same gigabyte: another copy, across the frontier between two libraries' allocators. Unified memory deleted the hardware copy; it did not delete the copies programs make out of habit or necessity. When a runtime feels slower than the bandwidth says it should, this is one of the first places to look.

Operation, 1 GiB float32, office	Time	What moved	Evidence
PyTorch `tensor.to('mps')`	66.1 ms	a memcpy into a Metal buffer, same pool	measured 2026-09-15
PyTorch `tensor.to('cpu')`	33.0 ms	a memcpy back out	measured
MLX `mx.array(numpy)`	250.0 ms	a copy across two libraries' allocators	measured
MLX GPU sum of that array	16.8 ms	nothing; the GPU read it in place	measured
MLX CPU sum of the same array	33.8 ms	nothing; the CPU read it in place	measured

Storage Modes: Metal's Names for the Same Pool

Underneath every framework is Metal, and Metal's own documentation for Apple GPUs states the arrangement and its one subtlety in two sentences: "Apple GPUs have a unified memory model in which the CPU and the GPU share system memory. However, CPU and GPU access to that memory depends on the storage mode you choose for your resources." A buffer's storage mode is shared — "system memory that both the CPU and the GPU can access", the default — or private — "system memory that only the GPU can access", still the same DRAM, with the driver free to lay it out for the GPU — or, for textures only, memoryless: "tile memory within the GPU that only the GPU can access", which "has higher bandwidth, lower latency, and consumes less power than system memory" and never touches DRAM at all. (A fourth mode, managed, exists for Intel-era Macs with a discrete GPU and mirrors a buffer across two memories; on Apple silicon there is nothing to mirror.) When a runtime like MLX or llama.cpp allocates weights it is choosing among these, and the gpu-compute quest teaches the choice in detail. This lesson only wants you to know that the choice exists, and that on this hardware every option but memoryless is the same DRAM wearing a different access policy.

Code

one_pool.py — the same array on two processors, and the copies frameworks still make·python

#!/usr/bin/env python3
"""Zero-copy in MLX (one array, two streams) beside the copies PyTorch's MPS
backend and the NumPy->MLX frontier still make. Needs numpy, mlx and torch."""
import time
import numpy as np
import mlx.core as mx
import torch

N = 1 << 28                                     # 2^28 float32 = 1 GiB


def ms(fn):
    t = time.perf_counter(); fn(); return (time.perf_counter() - t) * 1e3


# --- PyTorch: a software boundary kept from the discrete-card era
x = torch.ones(N, dtype=torch.float32)
print(f"torch .to('mps')  1 GiB: {ms(lambda: (x.to('mps'), torch.mps.synchronize())):6.1f} ms   (memcpy inside one pool)")
y = x.to('mps'); torch.mps.synchronize()
print(f"torch .to('cpu')  1 GiB: {ms(lambda: y.to('cpu')):6.1f} ms")

# --- MLX: no device placement; streams choose the processor, data stays put
a_np = np.ones(N, dtype=np.float32)
print(f"mx.array(numpy)   1 GiB: {ms(lambda: mx.eval(mx.array(a_np))):6.1f} ms   (a copy across two allocators)")
a = mx.array(a_np); mx.eval(a)
print(f"mx.sum on gpu stream:    {ms(lambda: mx.eval(mx.sum(a, stream=mx.gpu))):6.1f} ms   (read in place)")
print(f"mx.sum on cpu stream:    {ms(lambda: mx.eval(mx.sum(a, stream=mx.cpu))):6.1f} ms   (same array, no transfer)")

# office, M3 Ultra, torch 2.11.0, mlx 0.32.2, 2026-09-15:
# torch .to('mps') 66.1 ms | .to('cpu') 33.0 ms | mx.array(numpy) 250.0 ms | gpu sum 16.8 ms | cpu sum 33.8 ms

Exercise

Run one_pool.py on your Mac (install torch in the same environment if needed) and add the five times to your card. Then compute the effective GB/s of the torch .to('mps') copy and compare it to your stream.py figure from lesson one. Write one sentence on why a copy inside a single pool runs at a small fraction of that pool's bandwidth.

Hint

A memcpy driven by one CPU thread through the page cache is bound by that thread and by page faults on freshly allocated memory, not by the memory bus. The GPU streaming kernel has eighty cores issuing requests; the copy has one core issuing them.