Priced by a Real Run

A Move Every Check Approved

On the day the family separated code, execution and state (track 6), generic model calls moved. Until then a sibling that wanted a plain rewrite, a schema-shaped judgment or a model list sent the request to the assistant's own engine, which ran it in process. After the move, each app ran the kit's worker itself.

Every check approved it. The kit's tests passed, with exact wire tests for each extracted provider call. The drift check was green. Each consumer's suite was green. And no live model call was issued as part of the deployment — deliberately, because model calls cost money and the move was supposed to be a relocation, not a change of behavior.

What the Consumers' Own Records Said Within Minutes

Three providers' credentials could not be read. The client that fetches credentials kept its own list of which providers exist, and the owner's list had more. The only consumer test had faked the client, so the client's own rule had never run under test.
A judging job collapsed. The memory workshop's judging step had been succeeding on roughly 185 of 197 calls. Through the worker it succeeded on 2 of 54, because the worker validated each answer strictly against its JSON schema, where the old runner had returned the text and let the caller parse it.
A news-triage stage failed every pass. The worker translated the caller's output-token limit into a local model's prediction limit with thinking turned on; the thinking used up the budget, and every answer stopped at the length limit before it began.

None of these was visible to a fixture. Each was visible, within minutes, in the run journals the consuming apps already kept.

The Rule That Followed

The owner's ruling was about direction, not about any one bug: the worker replaces calls the apps used to send to the assistant, so it must behave the way the assistant's in-process execution behaved. A stricter rule in the worker is a broken feature in the app. The repairs were done in parity batches, each item checked against the old code rather than against what seemed tidier: no strict schema validation, the caller's text handed back for the caller to parse; an effort level the brain does not offer falls back to the family default instead of failing; a missing local-model tier means the cloud tier, as it always had; failures carry the exception kind and a short scrubbed message instead of a bare class name.

The cutover had one more lesson waiting two days later. A new model pin required a newer command-line tool than the one bundled inside the worker's SDK, so every worker call to that brain failed while the assistant, which had always pointed at the installed tool, kept working. The worker now runs the installed tool too — another place where "behave like the code you replaced" meant copying a detail nobody had thought of as behavior.

Price an execution move by real runs, read back from the consumer's own records. "No live inference as a deployment check" was the right cost rule and the wrong acceptance rule. Accepting a cutover of execution needs one real run per changed execution path, read from the journal of the app that made it, plus the real client driven through a mock transport rather than a faked client. A green fixture suite is evidence about the fixtures.

How This Differs From the Rest of This Track

The earlier lessons priced conversions of a UI component: what moved, what was deleted, what the shared piece had to grow. Those costs were visible in a diff and a browser. An execution move has a different invoice. Its costs live in behavior that nobody wrote down because it was never a choice — how strict the parser was, which tool was on the path, what an absent field meant — and the only place they show up is the output of a real call, in the place where that output is used.

Code

Acceptance for an execution cutover: one real run per path, read where it lands·python

import httpx

# Every execution path the cutover changed: (consumer app, operation,
# brain). Not "every test" - every PATH that now runs differently.
CHANGED_PATHS = [
    ("journal",  "rewrite",   "claude"),
    ("memory",   "judge",     "claude"),      # schema-shaped answer
    ("news",     "triage",    "local-high"),  # thinking + token limit
    ("health",   "interpret", "kimi"),        # credential read
]


def accept_cutover(trigger_one_real_run, read_consumer_journal):
    """One real call per changed path, then read the CONSUMER's own
    record of it. The producer's log says the call was made; only the
    consumer's journal says the result was usable."""
    failures = []
    for app, op, brain in CHANGED_PATHS:
        run_id = trigger_one_real_run(app, op, brain)
        row = read_consumer_journal(app, run_id)
        if row["status"] != "ok":
            failures.append((app, op, brain, row.get("error")))
    assert not failures, failures


# And in the unit suite: drive the REAL client through a mock
# transport, so the client's own rules run under test.
def test_real_client_reads_every_owner_provider():
    def owner(request: httpx.Request) -> httpx.Response:
        return httpx.Response(200, json={"key": "redacted"})
    client = CredentialClient(                     # a sketch of the shape
        base="http://owner.invalid",
        transport=httpx.MockTransport(owner))      # real client, fake owner
    for name in ("kimi", "glm", "gemini"):        # the owner decides
        assert client.credentials(name)           # which providers exist

Exercise

Pick a piece of work in your system that recently moved from one executor to another — a job queue, a service, a library call replacing a remote call. List every execution path the move changed. For each, find the place where the result is actually consumed and check one real run there. Then look at the tests that approved the move: which of them faked the very component that changed?

Hint

Look for behavior nobody would call a feature: how strict a parser was, what an absent field defaulted to, which binary was on the path, how a timeout was counted. Those are the details a rewrite drops, and they only show up in real output.

A Move Every Check Approved

What the Consumers' Own Records Said Within Minutes

The Rule That Followed

How This Differs From the Rest of This Track

Code

External links

Exercise

Progress

Comments 0