"The Neural Engine is a compute engine with a narrow door to the pool. Give it a picture and it shines. Give it a language model's decode and it reads the weights at two per cent of the bus."
The Third Door, and the Fourth Unit
Core ML is Apple's own inference path: a model compiled once into a form the operating system schedules across the CPU, the GPU and the Neural Engine, with the caller choosing a compute units setting and the framework choosing, op by op, where each runs. It is the only path in this track that can reach the Neural Engine at all — the CPU track's fixed-function unit that ioreg lists and nothing else in the quest has touched. The household's one Core ML user is the voice sibling, through WhisperKit, which runs a Whisper encoder and decoder as Core ML models with the Neural Engine in play; nothing in the family calls Core ML directly, and nothing uses Apple's Foundation Models framework. So this lesson measures the unit itself, with models built straight from Core ML's intermediate language, and asks the planner what it decided.
Two Shapes of Work, Four Settings
Two models, on office. A conv stack — eight 3×3 convolutions over a 224-pixel image, 104 GFLOP — is the Neural Engine's home shape. A single fp16 matrix-vector product over a 537 MB matrix is a language model's decode step in miniature: read every weight once, do almost nothing with it. Each ran under all four settings, twenty times after warm-up, and the compute plan reported the unit the framework preferred for each op.
| Model, office, coremltools 8.1 | CPU only | CPU+GPU | CPU+NE | ALL (planner's choice) | Evidence |
|---|---|---|---|---|---|
| conv stack, 104 GFLOP | 12.9 ms (8.0 TFLOP/s) | 4.65 ms (22.3) — GPU | 6.15 ms (17.0) | 6.12 ms — Neural Engine, 17 of 17 ops | measured |
| matvec, 537 MB fp16 | 7.6–14.9 ms (36–71 GB/s) | 1.71 ms (314 GB/s) — GPU | 30.2 ms (17.8 GB/s) | 30.7 ms — Neural Engine | measured |
| matvec, 134 MB fp16 | 2.15 ms | 2.18 ms — CPU | 2.56 ms | 2.10 ms — Neural Engine | measured |
| matvec, 34 MB fp16 | 0.65 ms | 0.66 ms — CPU | 0.62 ms | 0.63 ms — CPU | measured |
Read the conv row first: the Neural Engine runs the image at 17 TFLOP/s, within a quarter of the 80-core GPU, on a unit a fraction of its size — that is the fixed-function argument from the CPU track, made in numbers. Now the 537 MB row. The same unit reads a language model's weights at 17.8 GB/s — eighteen times slower than the GPU on the same bytes, two per cent of the pool's 819 — and under the default setting the planner chose it anyway, and spent 24 seconds compiling for it on every load. Apple wrote this down in 2022, in the article on deploying transformers to the unit: "many Transformer configurations become bandwidth-bound on the ANE when the sequence length is relatively short … large parameter tensors are being fetched from memory, only to be applied on too few inputs." That is the decode ceiling, from the unit's side.
What Apple Runs Where
Apple's own two answers are consistent with the table. For an 8B model on a Mac, its Core ML article says the quiet part outright: "specifically target the GPU, as the models like the Llama-3.1-8B-Instruct are usually constrained by memory bandwidth, and the GPU offers the best combination of compute FLOPS and memory bandwidth" — and reports about 33 tokens per second for a 4-bit build on an M1 Max, which at 4.2 GB against 400 GB/s is 35% of that chip's ceiling, a fraction the lab's MLX ladder beat with the 9B and the 27B on every Mac. For the ~3 billion parameter on-device foundation model, Apple describes "efficient Key-Value (KV) cache update on our neural engines" and 30 tokens per second on an iPhone 15 Pro "before employing token speculation techniques" — a model sized and laid out for the narrow door, on a device where the door's energy cost matters more than the bus. Big models to the GPU, a small one to the Neural Engine, speculation on top: the vendor's split matches the physics track's, and the Ollama lesson's trick appears in Apple's sentence too.