How This Quest Labels a Claim

"Neutral is not a tone of voice. It is a column in the table."

Why a Label and Not a Promise

The brief for this quest asked for neutrality — no fan's bias, the real advantages and the real costs. A promise to be neutral is worthless; every vendor page promises it. What works is structural: every load-bearing claim in this quest carries one of four labels, visibly, so that you can see what kind of thing it is before you decide whether to believe it. Callouts carry the label as their type; number tables carry an Evidence column. The four:

Label	What it means	What must come with it	Example
physics	arithmetic anyone can redo from public inputs	the inputs and the formula	819 GB/s ÷ 14.4 GB per token = 57 tok/s ceiling
vendor-claim	a manufacturer's number, with its baseline and stage	the exact quote, the baseline unit, what stage it measures	"up to 4x faster LLM prompt processing than M3 Ultra" — prefill, LM Studio, M3 Ultra baseline
measured	read on a named fleet Mac, on a date	alias, macOS build, framework version, model, quantization, context length, date	Qwen3.5-9B 4-bit, office, macOS 26.6.2, MLX 0.32.2 / mlx-lm 0.31.3, 209-token prompt, 2026-09-15: 95.1 tok/s decode
our-judgment	the house position, labelled as one	the reasoning, and the fact that it is a position	the 512 GB ceiling is an engineering limit before it is a price limit

Other callout types appear too — principle, warning, tip, war-story, brain-trap (where a coding assistant gets the hardware wrong), and pippa-confession (where the author did). None of those is an evidence class; they are teaching devices. Only the four above tell you how a fact was obtained.

The Worked Example

Apple's Mac Studio release of 2026-08-25 says the M5 Ultra delivers "up to 4x faster" LLM prompt processing than the M3 Ultra in LM Studio. Here is how the labels take that sentence apart.

vendor-claim: the number is 4x, the stage is prompt processing — prefill — and the baseline is an M3 Ultra. Apple's footnote gives the exact unit and the test month.
physics: prefill is compute-bound, so a chip with far more matrix compute (the M5 GPU's Neural Accelerators) can plausibly prefill several times faster. Decode is bandwidth-bound: the ceiling is memory bandwidth divided by the bytes read per token. The M5 Ultra's 1.2 TB/s over the M3 Ultra's 819 GB/s is a ratio of about 1.47.
our-judgment: therefore the honest expectation for decode on the same model is around 1.5x, not 4x. This is arithmetic on vendor inputs, and no one in this household has measured an M5 Ultra, so it is labelled as a judgment and marked unmeasured.
measured: nothing yet. When an M5 Ultra is measured with the lab script, that row gets this label and the judgment row gets retired or confirmed.

Notice that nothing in that chain accuses Apple of anything. The 4x is probably true for what it measures. The label system simply refuses to let a prefill number stand in for a decode number, which is the single most common way hardware marketing gets read wrong.

The Stamp a Measurement Must Carry

The lab track's script stamps every result with the machine alias, chip, macOS version and build, MLX and mlx-lm versions, model, quantization, prompt length and a UTC timestamp — the code block shows the function. The rule the quest applies to itself: a number without that stamp is not a measurement, and is not quoted as one. It is also the rule to apply to every benchmark you read elsewhere, and most of them fail it.

Code

labels.py — a claim is a value plus the kind of evidence behind it·python

#!/usr/bin/env python3
"""Four evidence classes. A Claim without a complete stamp for its class
refuses to print as that class — the same rule this quest applies to itself."""
from dataclasses import dataclass, field

REQUIRED = {
    "physics":      ["inputs", "formula"],
    "vendor-claim": ["quote", "source", "baseline", "stage"],
    "measured":     ["alias", "macos", "framework", "model", "quantization", "context_tokens", "date"],
    "our-judgment": ["reasoning"],
}


@dataclass
class Claim:
    text: str
    label: str
    stamp: dict = field(default_factory=dict)

    def render(self) -> str:
        missing = [k for k in REQUIRED[self.label] if k not in self.stamp]
        if missing:
            return f"[UNLABELLED] {self.text}  (missing {', '.join(missing)} for '{self.label}')"
        return f"[{self.label.upper()}] {self.text}  {self.stamp}"


claims = [
    Claim("M5 Ultra: up to 4x faster LLM prompt processing than M3 Ultra", "vendor-claim",
          {"quote": "up to 4x faster", "source": "Apple newsroom 2026-08-25",
           "baseline": "Mac Studio M3 Ultra", "stage": "prefill (LM Studio)"}),
    Claim("Decode ceiling ratio M5 Ultra / M3 Ultra ≈ 1.47", "physics",
          {"inputs": "1229 GB/s, 819 GB/s", "formula": "bandwidth ratio"}),
    Claim("Expect ~1.5x decode on the same model, unmeasured", "our-judgment",
          {"reasoning": "decode is bandwidth-bound; no M5 Ultra in the fleet"}),
    Claim("Qwen3.5-9B 4-bit decodes at 95.1 tok/s", "measured",
          {"alias": "office", "macos": "26.6.2 (25G83)", "framework": "mlx 0.32.2 / mlx-lm 0.31.3",
           "model": "mlx-community/Qwen3.5-9B-4bit", "quantization": "4-bit g64 affine",
           "context_tokens": 209, "date": "2026-09-15"}),
    Claim("The 9B model decodes at 95 tok/s on an M3 Ultra", "measured", {"alias": "office"}),
]
for c in claims:
    print(c.render())

stamp() — what the lab script attaches to every result·python

import platform, subprocess
from datetime import datetime, timezone
import mlx.core as mx
import mlx_lm


def stamp(alias: str) -> dict:
    info = mx.device_info()   # mx.metal.device_info() is deprecated in mlx 0.32
    build = subprocess.run(["sw_vers", "-buildVersion"], capture_output=True, text=True).stdout.strip()
    return {
        "alias": alias,
        "device": info["device_name"],
        "memory_gb": round(info["memory_size"] / 1e9, 1),
        "recommended_working_set_gb": round(info["max_recommended_working_set_size"] / 1e9, 1),
        "macos": f"{platform.mac_ver()[0]} ({build})",
        "mlx": mx.__version__,
        "mlx_lm": mlx_lm.__version__,
        "date": datetime.now(timezone.utc).isoformat(timespec="seconds"),
    }


print(stamp("office"))
# {'alias': 'office', 'device': 'Apple M3 Ultra', 'memory_gb': 549.8,
#  'recommended_working_set_gb': 498.2, 'macos': '26.6.2 (25G83)',
#  'mlx': '0.32.2', 'mlx_lm': '0.31.3', 'date': '2026-09-15T04:31:52+00:00'}

Exercise

Run labels.py and note which claim prints as UNLABELLED and why. Then add three claims of your own about your Mac — one of each of physics, vendor-claim and our-judgment — with complete stamps, and one measured claim that you leave deliberately incomplete. Which single missing field would make a measurement least reusable by someone else, and why?

Hint

The framework version is the usual answer: a decode number from mlx-lm 0.20 and one from 0.31 on the same Mac can differ by more than the gap between two Macs, and nothing in the number tells you which you are looking at. Date is a proxy for it, but only a proxy.