One Inference Hub, Many Clients

"One Mac holds the models. Every other Mac holds a URL. That is the whole architecture, and the counters say it works."

What the Hub Serves

Server is a 512 GB Studio that runs three inference daemons and holds every model the family's apps share. Its MLX endpoint, read live through its own API, lists five models: two rerankers, the bge-m3 embedding model in fp16, a 31B chat model in bf16 and a document converter. Its Ollama daemon held one model at the moment of reading — the same embedding model in Ollama's format, kept warm for the soul-memory steward, which still uses it. Its image engine runs PyTorch and diffusers on MPS, the journey track's big-op case, for the drawing app, the file workbench and Pippa's image describe. The MLX endpoint's own counters, read the same minute: 143,626 requests, 329,536,723 prompt tokens, zero completion tokens, zero cached tokens. Every request the family has ever sent it was prefill — an embedding or a rerank — and the chat model in its list is a tier clients can choose, not one they have.

Why the Clients Are Thin

The pattern is one GPU host and many clients that hold no shared weights (the local tier lesson four describes — an Ollama daemon on the asking Mac — is the deliberate exception, chosen per surface). The image engine's own house rules make the strictest version explicit: server is the only inference host, and clients "fail closed" — a laptop that cannot reach the hub does not fall back to its own GPU, it reports the hub down. That sounds harsh and is the right trade: one copy of each model, one place to update it, one machine whose working set the GPU track's claimant discipline must protect, and no laptop quietly running a stale checkpoint. Pippa's own search, the corpus engine and the prose editor's co-writing all reach the same endpoint. The interface lesson priced the network crossing at a few milliseconds a request; for a query that reads 1,024 numbers back it is nothing, and for a chat stream it is less than a token's decode time. The hub is fast enough because the pool is large enough, and the pool is large enough because one machine has it.

The Number That Says Which Workload It Is

Zero completion tokens over 143,626 requests is the most informative line in the counters. The physics track's two stages — prefill compute-bound, decode bandwidth-bound — become a deployment fact: this hub's load is all prefill, at an aggregate rate the ledger derives near 1,229 prompt tokens per second of summed prefill time, with concurrency making that an overcount of wall time. The MLX server's SSD cache — its ability to reload a matching prefix's KV blocks from disk instead of recomputing — has nothing to reuse on such a load, and its cache directory held zero bytes when read. The household's ruling that the MLX server's caching makes parts of it faster than Ollama is true for generation with repeated prefixes and not yet exercised by a hub that only embeds; the memory-search lesson says both halves. What the hub proves is simpler: a 512 GB pool serving a family's retrieval for months at a time, on a desktop, silently, with one model warm in each daemon.

Code

hub_models.py — what the hub serves, from its own APIs and counters·python

#!/usr/bin/env python3
"""One inference hub, many clients: what the hub serves. Two daemons on one Mac, read
through their own APIs. The hub's address comes from the environment, never from a
quest; run this on the hub itself or point HUB_MLX / HUB_OLLAMA at it."""
import json, os, urllib.request
mlx = os.environ.get("HUB_MLX", "http://127.0.0.1:8000")          # the MLX server's OpenAI-compatible base
oll = os.environ.get("HUB_OLLAMA", "http://127.0.0.1:11434")      # the Ollama daemon
models = json.load(urllib.request.urlopen(mlx + "/v1/models"))["data"]
print("MLX endpoint serves:", [m["id"] for m in models])
ps = json.load(urllib.request.urlopen(oll + "/api/ps"))["models"]
print("Ollama has loaded:  ", [(m["name"], f"{m['size']/1e9:.1f} GB") for m in ps])
stats = json.load(open(os.path.expanduser("~/.omlx/stats.json")))            # the MLX server's own counters
print("MLX totals:         ", {k: stats[k] for k in ("total_requests", "total_prompt_tokens", "total_completion_tokens", "total_cached_tokens")})
# server, 2026-09-15:
# MLX endpoint serves: ['Qwen3-Reranker-0.6B-mlx-8Bit', 'Qwen3-Reranker-4B-mxfp8', 'bge-m3-mlx-fp16', 'gemma-4-31b-it-bf16', 'MarkItDown']
# Ollama has loaded:   [('bge-m3:latest', '0.7 GB')]
# MLX totals:          {'total_requests': 143626, 'total_prompt_tokens': 329536723, 'total_completion_tokens': 0, 'total_cached_tokens': 0}

Exercise

On the Mac you run models on, list what its daemons serve and read any counters they keep. Write on your card the ratio of prompt tokens to completion tokens — it tells you which stage your hub lives in — and, from that, whether its next upgrade should buy bandwidth or GPU cores.

Hint

A retrieval hub is prefill: cores. A chat hub is decode: bandwidth and capacity. A hub that does both at once is the batch lesson's territory, and the counters will show a completion-to-prompt ratio between the two.