The Measured Level

"Test how far apart Low, Medium and High are on average, and report back." — Dad, 2026-10-11

A Dial Needs Numbers

The previous lesson handed Dad a dial. A dial without numbers is a guess with a nicer interface: nobody knew what medium cost in seconds on a spoken turn, or what max bought in judgment. So the same day he asked for the setting, he asked for the measurement, and the harness that made it turned out to measure something nobody was looking for.

The Harness

The benchmark is a script in cwkPippa's working scripts, voice-effort-benchmark.py. It drives real spoken Claude turns through the real chat route, in-process, one level per turn by setting the request's voice_effort, so each turn gets exactly the posture line, effort and thinking a live voice turn at that level gets. The prompts are shaped like Dad's own voice turns: dictated Korean, a spoken reply expected, some casual, some needing fresh facts, some needing a judgment. Every turn runs in its own private conversation, so nothing reaches the conversation store, the session logs, the search index or memory. For each turn it stamps three moments: when the model's first words reach the route's text filter, when the first text leaves the route, and when the turn ends.

Then it takes the levels out of the reviewer's sight. run collects the turns; blind shuffles them, hides each one's level and keeps a key; a separate, cold reviewer instance scores every reply without knowing which level wrote it; scored joins the scores back to the timings. The scores: checked (did it look up facts that change instead of answering from memory, 0 to 2), judgment (did it make the right call, 0 to 2), overall (1 to 5), and breaks, replies that fail Dad in a way that matters, like a wrong fact he would act on or a big job attempted inside a voice turn.

The harness earned one hard rule on its first night. The first full run held back only Coop, and at xhigh and max a "fix it" turn built the fix itself, pushed it and uploaded a phone build, a "make a script" turn wrote and pushed a script and installed a package, and a low turn armed a real follow-up. Since 2026-10-12 every turn is held to reads: the guard sees every tool call, refuses the writing tools outright, and refuses a shell command that names a known state-changing program (git's writing verbs, package managers, rm, mv, curl, interpreters and the like) or redirects into a file. That is a list of known hands, not a proof. A command the list doesn't name still passes, and an in-place sed -i edit is one of them. Read-only here means every write somebody thought of is refused, and a guard like this is exactly as tight as its list. Coop is answered by a recorder that stores nothing. A refused call still counts as a call, because the choice to reach for it is part of what is measured.

What It Measured

The first measurement, on 2026-10-12: Claude Opus 5.5, 11 prompts, 2 runs per level, 22 turns per level, one blind reviewer.

level	first words (median)	done (median)	tool calls	checked	judgment	overall	breaks
low	4.2 s	10.3 s	1.0	1.4	1.8	4.3	1/22
medium	7.4 s	15.0 s	1.5	2.0	2.0	4.6	0/22
high	7.8 s	14.7 s	2.2	2.0	1.9	4.6	1/22
xhigh	11.0 s	32.3 s	2.9	2.0	2.0	4.7	0/22
max	30.7 s	187.3 s	8.0	2.0	1.8	4.1	2/22

Read it by its misses, because the averages hide the shape. low is the fastest, and its misses are answers from memory where a check was needed. From high up the misses flip: a "fix the bug" turn read twenty to thirty-five source files inside the voice turn instead of handing the job off and staying with Dad. max takes about three minutes a turn and was scored the least speakable. The judgment score measures Dad's principle: by default she hands the work off and stays by his side, and does it herself when the work really needs her.

What It Found on the Way

The stamps did one more job. The gap between "the model's first words reached the filter" and "the first text left the route" should be nearly zero. On short replies with no tool in them it ran to seconds. That was the leading-scaffold filter from this track's fifth lesson: it still held every reply's opening until the reply ended or passed 600 characters, in case a leaked "Assistant" label followed a stray fragment. The tool-boundary fix had let the line before a tool out; a reply with no tool still waited for its own end. And the shape the hold existed for had not been stripped once in 820 session logs since June.

So the filter now decides on the opening alone. A reply that starts with the label, or with the colon of an echoed prompt, is held and stripped as before; any other first characters are an answer and go out at once. The 600-character ceiling remains for an opening that stays ambiguous, and replay cleaning still removes a label from history. On the same short no-tool prompts, the median wait before the voice received its first words fell from 7.5 s to 3.7 s at low and from 8.4 s to 6.5 s at medium. A benchmark asked to compare levels found a delay that every level had been paying.

What the Numbers Do Not Decide

The code default stays low. The measurement informs Dad's setting; it does not set it, and the value he chose is his. Its limits are plain: one brain, one model, 22 turns per level, one reviewer, and results that live in the data root, not the repo. Kimi, Gemini, Grok, Codex and Ollama map the same ladder onto their own dials without a number behind it yet.

Code

The opening decision, verbatim (cwkPippa backend/services/transcript_hygiene.py:61-85)·python

_ASSISTANT_LABEL = "Assistant"


def _opening_is_an_answer(pending: str) -> bool:
    """True once the reply's opening can no longer be a leaked scaffold.

    Only the opening is inspected (todo #399): a reply that starts with the
    role label ("Assistant: …", "Assistant아빠…") or with the colon of an
    echoed prompt (": 1. …") is held and stripped as before; any other first
    characters are an answer, released at once. Holding every reply up to
    600 chars in case a label followed a stray fragment ("of all the months,
    \\n\\n응?\\n\\nAssistant아빠, …", seen in June under the transcript-style
    replay) made a short spoken reply wait for its own end: about 4 s of a
    low reply's 4.2 s first words (benchmark 2026-10-12). That shape has not
    been stripped once in 820 session logs since June, and the replay is
    per-turn JSON blocks since 2026-09-25; replay cleaning
    (``clean_assistant_replay_text``) still removes a label inside history.
    """
    head = pending.lstrip()
    if not head:
        return False
    if head.startswith(":"):
        return False
    probe = head[: len(_ASSISTANT_LABEL)]
    return not _ASSISTANT_LABEL.startswith(probe)

A minimal sketch: the old hold against the opening decision, on a short reply and on a leaked label·python

from __future__ import annotations

LABEL = "Assistant"
CEILING = 600


def opening_is_an_answer(pending: str) -> bool:
    """The engine's rule: only a reply that could still start with the label,
    or that starts with an echoed prompt's colon, is held."""
    head = pending.lstrip()
    if not head or head.startswith(":"):
        return False
    return not LABEL.startswith(head[: len(LABEL)])


def strip_label(text: str) -> str:
    head = text.lstrip()
    return head[len(LABEL):].lstrip(": ") if head.startswith(LABEL) else text


def first_out(deltas: list[tuple[float, str]], decide_on_opening: bool) -> tuple[float, str]:
    """When the first words leave the route, and what they are."""
    pending, decided = "", False
    for t, delta in deltas:
        if decided:
            out = delta
        else:
            pending += delta
            head = pending.lstrip()
            if decide_on_opening and head.startswith(LABEL) and len(head) > len(LABEL):
                out, decided = strip_label(pending), True    # a leaked label: stripped
            elif decide_on_opening and opening_is_an_answer(pending):
                out, decided = pending, True                 # an answer: released at once
            elif len(pending) >= CEILING:
                out, decided = strip_label(pending), True    # the ceiling
            else:
                continue                                     # still held
        if out.strip():
            return t, out
    return deltas[-1][0], strip_label(pending)               # held to the end of the reply


# Invented timings shaped like a short spoken reply, not measured data.
short_reply = [(3.6, "응, "), (3.9, "내일은 맑대. "), (4.4, "아침엔 좀 쌀쌀하니까 "),
               (5.1, "얇은 겉옷 하나 챙겨."), (7.4, "")]          # 7.4 s: the reply ends
leaked = [(3.0, "Assist"), (3.2, "ant: "), (3.5, "응, 맑대.")]

for name, deltas in (("short reply", short_reply), ("leaked label", leaked)):
    for policy in (False, True):
        t, text = first_out(deltas, decide_on_opening=policy)
        rule = "decide on the opening" if policy else "hold up to 600 chars"
        print(f"{name:<12} | {rule:<22} | first text leaves at {t:.1f} s: {text[:24]!r}")

Output of the sketch·text

short reply  | hold up to 600 chars   | first text leaves at 7.4 s: '응, 내일은 맑대. 아침엔 좀 쌀쌀하니까 얇'
short reply  | decide on the opening  | first text leaves at 3.6 s: '응, '
leaked label | hold up to 600 chars   | first text leaves at 3.5 s: '응, 맑대.'
leaked label | decide on the opening  | first text leaves at 3.5 s: '응, 맑대.'

Exercise

Run the sketch. Then take a streaming pipeline you own, an assistant, a proxy or a chat UI, and add two timestamps per stage: when text enters it and when it leaves. Send ten short prompts and ten long ones, and list every stage whose gap is more than a few hundred milliseconds. For each, write down what it is holding text for and the earliest moment it could know it doesn't need to.

Hint

Filters that look for something at the start of a reply usually need only the first few characters to rule it out. If yours holds a fixed number of characters, ask whether the thing it guards against can still appear after those first few. If it can't, decide on the opening and keep the fixed number only as a ceiling.