~16 min · journey, ollama, speculative-decoding, mtp, gguf, measured
Level 0Spec-Sheet Skimmer
0 XP0/91 lessons0/19 achievements
0/100 XP to next level100 XP to go0% complete
"A runtime's tokens per second is what you feel. The physics track's tokens per second is what one pass through the weights yields. They are the same number only until the runtime gets clever."
One Daemon, Two Engines
Ollama is the door most people use: a daemon, a registry, ollama run. On a Mac the 0.34 daemon has two engines behind that verb. One is a llama.cpp-derived llama-server subprocess reading GGUF (the previous track's community runtime, with Metal); the other is Ollama's own MLX engine, a Go runner over MLX's C API that its log announces as "MLX engine initialized … device=gpu" and that stores a model one blob per tensor. A tag chooses: qwen3.8:27b-mlx is the same 27B architecture the lab measured (64 layers, 16 full-attention and 48 linear), quantized NVFP4 — 4-bit values with an 8-bit scale per 16, 4.5 bits per weight, the same bits as the lab's 4-bit group-64 checkpoints — and reading the blob sizes by the physics track's rules gives 14.45 GB per token, against the lab's 14.42. qwen3.8:latest is a Q4_K_M GGUF of 16.8 GB. Both live in the one pool; ollama ps says 100% GPU for either, which on a Mac means every layer is on the GPU inside the working set — the same column on a discrete-GPU machine shows the CPU/GPU split that is the VRAM wall.
A Number the Physics Track Says Is Impossible
The first --verbose run on office reported 43.2 tokens per second for the MLX-engine 27B; API runs of 512 tokens of prose came in at 47.6–50.6, wall-clocked from the stream. The physics track's ceiling for 14.45 GB per token at office's measured 638 GB/s is 44. mlx-lm reached 32.6. A runtime cannot beat the ceiling by reading the weights — so it is not reading them once per token. The runner's own log says how: speculative decode stats iterations=225 drafted=458 accepted=287 acceptance=0.63. The checkpoint ships a multi-token-prediction head (mtp.*, 0.24 GB — the idea DeepSeek-V3 trained for), and Ollama's MLX runner uses it to draft several tokens cheaply, then verifies them in one pass through the full weights. Five hundred and twelve tokens took 217–225 passes: 2.3 tokens per pass, 45–50 ms per pass. Per pass, that is slower than mlx-lm's 30.7 ms — 290–320 GB/s of effective bandwidth against 470 — because a verify pass is a small batch, and the batch lesson said a batch costs compute where a single token costs bytes. The win is entirely tokens per pass. Ask it to repeat a sentence forty times and acceptance goes to 1.00, 4.9 tokens per pass, 90 tokens per second; ask for random hexadecimal and it still accepts 0.85, at 77–82.
The Same Trick on the Other Engine
The GGUF engine drafts with the same head — its log says "creating MTP draft context against the target model", acceptance 0.867, mean draft length 4.47 — and delivers 27.5 tokens per second. llama-bench on the very same blob, single token, no drafting: 26.4 ± 0.4. The same trick that bought the MLX engine half again bought this one four per cent; why its verify pass costs so much more is not something this quest measured. Two engines, one pool, one head, opposite outcomes — and neither eval rate is the number the decode-ceiling lesson defines.
Ollama is the family's convenience tier, not its physics tier. The coding agent has a local leg over Ollama and hands local models a compiled mini protocol because they cannot hold the whole vault; the video-memory engine's summaries run on Ollama on server or worker, defaulting to the cloud pin; Pippa's sibling door enumerates Ollama models in three tiers, local, server and cloud. And the memory search that Part 4 is about left Ollama for oMLX when an Ollama change moved the ground under the embedding model — the door most people use is also the door that moves. The next lesson is the path that breaks.
Code
Two engines, one pool — and the log line that explains the impossible number·bash
# office, 2026-09-15, Ollama 0.34.0 (Homebrew, launchd). The -mlx tag selects Ollama's MLX engine.
ollama show qwen3.8:27b-mlx | head -8 # architecture qwen3_5, 27.8B, quantization nvfp4
ollama show qwen3.8:latest | head -8 # architecture qwen35, 27.3B, quantization Q4_K_M
ollama run qwen3.8:27b-mlx --verbose "Explain unified memory in 150 words." >/dev/null
# load duration: 3.64s
# prompt eval count: 21 token(s)
# prompt eval rate: 13.59 tokens/s <- the first forward touching every page (previous lesson)
# eval count: 305 token(s)
# eval rate: 43.23 tokens/s <- above the physics track's 44 ceiling? read on
ollama ps
# NAME ID SIZE PROCESSOR CONTEXT UNTIL
# qwen3.8:27b-mlx 5642e97495e1 18 GB 100% GPU 131072 Forever
# the runner's own accounting (Ollama.app writes ~/.ollama/logs/server.log; this launchd install logs to ~/Library/Logs)
grep "speculative decode stats" ~/Library/Logs/ollama.err.log | tail -1
# msg="speculative decode stats" iterations=225 drafted=458 accepted=287 acceptance=0.63 avg_draft=2.04 max_draft=4
# -> 512 tokens in 225 passes through the weights: 2.3 tokens per pass, 45 ms per pass (217-225 over four runs)
# the GGUF engine drafts with the same head, and the community benchmark on the same blob does not:
grep -E "MTP draft|draft acceptance" ~/Library/Logs/ollama.err.log | tail -2
# common_speculative_init_result: creating MTP draft context against the target model ...
# slot print_timing: draft acceptance = 0.86695 ( 2906 accepted / 3352 generated), mean len = 4.47
./llama.cpp/build/bin/llama-bench -m ~/.ollama/models/blobs/sha256-<the qwen3.8:latest blob> -p 512 -n 128 -ngl 99
# | qwen35 27B Q4_K - Medium | 15.65 GiB | 27.32 B | MTL,BLAS | tg128 | 26.39 ± 0.44 |
ollama_wall.py — never trust a rate you did not wall-clock·python
#!/usr/bin/env python3
"""Stream from the Ollama API and time the tokens yourself. eval_count / eval_duration
is the daemon's own rate; first-chunk-to-done is the wall clock. They should agree.
What neither tells you is how many passes through the weights produced them —
that is in the runner's log."""
import json, time, urllib.request
def run(model: str, prompt: str, n: int = 512) -> None:
body = {"model": model, "prompt": prompt, "stream": True, "think": False,
"options": {"num_predict": n, "temperature": 0, "seed": 1}}
req = urllib.request.Request("http://127.0.0.1:11434/api/generate", data=json.dumps(body).encode(),
headers={"Content-Type": "application/json"})
first = None
with urllib.request.urlopen(req, timeout=900) as resp:
for line in resp:
d = json.loads(line)
if d.get("done"):
last = d
break
first = first or time.perf_counter()
wall = time.perf_counter() - first
ec, ed = last["eval_count"], last["eval_duration"] / 1e9
print(f"{model}: {ec} tokens reported {ec/ed:.1f} tok/s wall {ec/wall:.1f} tok/s "
f"(load {last['load_duration']/1e9:.2f}s, prompt {last['prompt_eval_duration']/1e9:.2f}s)")
for p in ("Explain unified memory in 400 words.",
"Repeat this sentence exactly 40 times, one per line: The quick brown fox jumps over the lazy dog."):
run("qwen3.8:27b-mlx", p)
# office, 2026-09-15, four prose runs: reported 47.57 / 47.60 / 49.33 / 50.62; wall 47.79 / 47.98 / 49.63 / 50.93
# repetition (384 tokens): wall 90.2 and 91.0 tok/s -- acceptance 1.00, 4.9 tokens per pass in the runner log
# the ceiling for 14.45 GB per token at 638 GB/s is 44: every number above it is more than one token per pass
Run a model through Ollama with --verbose twice, then run ollama_wall.py on it. Find your daemon's log and look for speculative decode stats. If they are there, compute tokens per pass and milliseconds per pass and put both on your card next to the eval rate; if they are not, your eval rate is a per-pass rate and belongs on the card as one. Then compare against the ceiling for that model's bytes per token on your Mac.
Hint
Passes per second × bytes per token is the bandwidth the runtime actually pulled; tokens per pass is the speculation gain. A rate above the ceiling with no speculation stats in the log means either your bytes-per-token is wrong (vision tower or embedding counted — the physics track's confession) or the log is not the runner's.
Progress
Progress is local-only — sign in to sync across devices.