Lesson 03 of 04 · published

Tests That Fake the Brain

~14 min · pytest, test-doubles, monkeypatch, drift-check, testclient

Level 0Mise en Place

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Four things a test must not touch

A Sizzle engine on the service host depends on four things outside its own process. cwkPippa answers every reading and every conversation turn. The family brain authority behind Firelink says which brain each function uses. The data directory holds the journal, the mirror and the photos. The Mac's host profile can move the data root and the Mac each member is placed on. It cannot move a port: the profile accepts no port key, and engine_port reads the registry's port table directly. A test that touches any of them is testing the Mac it runs on. backend/tests/conftest.py replaces all four, and it makes the first replacement before anything else is imported.

Its first statement sets CWK_HOST_PROFILE to an empty string. The engine's settings read the family registry when they are imported, and the registry reads the host profile, so the variable must be set before backend is imported at all. That is why the imports under it carry # noqa: E402. With the profile pinned empty, every registry answer is the no-profile default, on every Mac.

FakePippa: the brain as five functions

The engine reaches cwkPippa only through backend/pippa.py. That module is the seam, and FakePippa replaces five of its functions with monkeypatch.setattr: read_plate, ensure_conversation, upload, ask and last_answer. Three choices make it more than a stub.

It answers in the real wire shape. read_plate returns the reading as a fenced JSON block, the way a model reply arrives, so the engine's parser and the rubric's validation run for real in every test that reads a plate. The fixture is a generic gambas al ajillo reading written in Korean, with one principle and two siblings.
It records what it was sent. inputs, image_batches, reading_picks and asked let a test assert on what Pippa would have seen: that a correction rode into the next read, that every photo went along, which brain the slot resolved to.
It enforces the contract. read_plate fails the test if it is ever called without photos, because a reading always carries the plate's photos.

last_answer returns None when it is asked for an exact match. That is the case where cwkPippa holds no answer for an interrupted turn, so a restart in a test settles the turn as failed instead of inventing one.

FakeCentral: a fake at the HTTP seam

The brain authority is faked one layer lower. FakeCentral comes from the kit (kit_brain_control_testing). The fixture builds a real BrainControlClient whose http_get and http_send are the fake's methods, with its cache under tmp_path. The engine's startup declaration then registers Sizzle and its three slots against a blank authority, through the same client code production runs. Only the transport is pretend.

A real engine for every engine test

make_client builds create_app(tmp_path / "sizzle-db", serve_webui=False) and enters a FastAPI TestClient. Entering runs the lifespan, so every test that takes make_client gets the whole startup from lesson 1: the mirror rebuilt from an empty journal, the slots declared, the worker started, the turns resumed. Not every test takes it. The parser tests, the host-profile tests, the drift test and two helper tests in the threads file call their functions directly and boot nothing, because what they check needs no engine. And because the engine is a function of its data directory, a restart can be tested. The rubric test captures a plate, records a reading under an older rubric, exits the client, then starts a second engine on the same directory and checks that the plate was read exactly once more, under the current rubric.

Suite	Count on the current main	How it runs
Engine (pytest, 8 files)	40 tests from 34 functions; the parser tests are parametrized	`python -m pytest backend/tests` in the `cwk-sizzle` env
iOS hosted unit (XCTest)	25	`scripts/test-mobile.sh` on a dedicated simulator
iOS UI (XCUITest)	2; one skips unless `SIZZLE_BENCH_ENGINE` is set	the same run
Web	no test runner	`tsc -b && vite build` is the check

The drift check with teeth

One of the 40 is different. test_sibling_kit_drift.py is itself vendored from a kit template. It runs the kit's sync/sibling_kit_sync.py --check --repo cwkSizzle --repo-root against the checkout it sits in and asserts exit 0. The kit compares every file its manifest vendors into Sizzle, after stripping the provenance header, and a script that lost its executable bit counts as drift too. That set is 36 files and much wider than Python: the kit_*.py modules, the TypeScript and CSS under frontend/src/kit/, the shell scripts from the last lesson and the TestFlight pipeline, and one Swift file, PippaBrainCatalog.swift.

The teeth are in the missing-kit branch. With no cwkSiblingKit checkout beside the repo, a fleet Mac (one with a non-empty ~/.config/fleet-label) fails the test, because a skipped check there is a green that checked nothing. Off the fleet it skips and says why. Exit 1 reads as drift. Any other exit reads as a check that ended without a verdict, and the message names a kit checkout older than the copies as one cause.

What the phone suite does not do

The architecture doc says the SizzleMobile suite calls KitDriftCheck.run(repo:) from CwkKitTesting. For this app that is false. No Swift file in the repository names KitDriftCheck, and no commit has ever added it under clients/. The iOS tests import CwkKitTesting for ScriptedHTTPTransport, which feeds the engine client scripted replies. Only pytest runs the drift check. The Swift copy is still compared, but only when someone runs the engine suite. A phone-only change verified with scripts/test-mobile.sh alone never hears about Swift drift. sibling-kit-quest's a-check-with-teeth track is about exactly this question: whose suite runs the check.

Principle: Fake at the seam you own, in the shape the real thing answers. A fake that returns the model's wire format keeps your parser under test. A fake that returns already-parsed objects tests little beyond itself.

Code

cwkSizzle, backend/tests/conftest.py: FakePippa and the fixture that patches it in (verbatim, lines 70-107)·python

class FakePippa:
    def __init__(self) -> None:
        self.reading: dict[str, Any] = GAMBAS
        self.inputs: list[str] = []
        self.image_batches: list[list[bytes]] = []
        self.reading_picks: list[dict[str, Any]] = []
        self.asked: list[dict[str, Any]] = []
        self.answer = "좋아, 들기름 버전으로 남기자.\n```sizzle-actions\n[]\n```"

    async def read_plate(self, instruction: str, input_text: str, images: list[bytes], **picks: Any) -> str:
        self.inputs.append(input_text)
        self.image_batches.append(images)
        self.reading_picks.append(picks)
        assert images, "a reading always carries the plate's photos"
        return "```json\n" + json.dumps(self.reading, ensure_ascii=False) + "\n```"

    async def ensure_conversation(self, key: str) -> str:
        return "conv-" + key.replace(":", "-")

    async def upload(self, files: list[tuple[str, bytes, str]]) -> list[dict[str, Any]]:
        return [{"id": f"up{i}", "name": name} for i, (name, _data, _type) in enumerate(files)]

    async def ask(self, conversation_id: str, message: str, **kwargs: Any) -> dict[str, Any]:
        self.asked.append({"conversation_id": conversation_id, "message": message, **kwargs})
        return {"result": self.answer, "error": None}

    async def last_answer(self, conversation_id: str, request_id: str, *, exact: bool = False) -> dict[str, Any] | None:
        if exact:
            return None
        return {"content": self.answer, "emotion": "happy"}


@pytest.fixture
def fake_pippa(monkeypatch: pytest.MonkeyPatch) -> FakePippa:
    fake = FakePippa()
    for name in ("read_plate", "ensure_conversation", "upload", "ask", "last_answer"):
        monkeypatch.setattr(pippa, name, getattr(fake, name))
    return fake

cwkSizzle, backend/tests/conftest.py: FakeCentral at the kit client's HTTP seams, and a real engine per test (verbatim, lines 110-136)·python

@pytest.fixture
def fake_central(tmp_path, monkeypatch):
    """A blank family brain authority at the kit client's HTTP seams: the
    engine's startup declaration registers Sizzle and its three slots."""
    fake = FakeCentral(brain_control.APP_ID)
    client = BrainControlClient(
        app_id=brain_control.APP_ID, control_base="http://central.test", data_dir=tmp_path / "brain",
        slots=brain_control.SLOTS, http_get=fake.get, http_send=fake.send,
    )
    monkeypatch.setattr(brain_control._bound, "client", client)
    monkeypatch.setattr(brain_control._bound, "adopted", False)
    return fake


@pytest.fixture
def make_client(tmp_path, fake_pippa, fake_central) -> Callable[[], TestClient]:
    clients: list[TestClient] = []

    def factory() -> TestClient:
        client = TestClient(create_app(tmp_path / "sizzle-db", serve_webui=False))
        client.__enter__()
        clients.append(client)
        return client

    yield factory
    for client in clients:
        client.__exit__(None, None, None)

cwkSizzle, backend/tests/test_sibling_kit_drift.py: fail on the fleet, skip off it, assert the kit's verdict (verbatim, lines 73-118)·python

def test_vendored_kit_files_match_kit_source():
    main = _main_checkout()
    kit_root = main.parent / "cwkSiblingKit"
    sync = kit_root / "sync" / "sibling_kit_sync.py"
    if not sync.is_file():
        label = _fleet_label()
        if label:
            pytest.fail(
                f"this is fleet Mac {label!r} and cwkSiblingKit is not checked out "
                f"beside this repo at {kit_root} — clone it there; the drift check "
                "cannot run without it"
            )
        pytest.skip(
            "not a fleet Mac (no ~/.config/fleet-label) and no cwkSiblingKit beside "
            f"this repo at {kit_root} — drift not checked here"
        )
    result = subprocess.run(
        [
            sys.executable,
            str(sync),
            "--check",
            "--repo",
            "cwkSizzle",
            "--repo-root",
            str(CHECKOUT),
        ],
        capture_output=True,
        text=True,
        timeout=60,
    )
    if result.returncode == 1:
        verdict = "sibling-kit drift detected — edit the kit, not the vendored copy:\n"
    else:
        verdict = (
            f"the kit's drift check ended without a verdict (exit {result.returncode}). "
            f"One cause is a kit checkout at {kit_root} older than this repo's copies; "
            "its own words:\n"
        )
    worktree = ""
    if main != CHECKOUT:
        worktree = (
            f"\n{CHECKOUT} is a git worktree of {main}; its own files were compared, "
            "not the main checkout's. A copy that is only older than the kit's comes "
            "with a merge of the main branch, where the kit's deploys are committed."
        )
    assert result.returncode == 0, verdict + result.stdout + result.stderr + worktree

External links

Exercise

Find the module in your own app that talks to a language model and make it the only place that does. Write a fake that replaces its functions in tests, returns replies in the exact text shape the real model sends (fences, extra prose and all), and records every input it receives. Then write one test that boots your real app on a temporary directory, triggers a model call, and asserts on what the fake was sent, and a second test that restarts the app on the same directory and checks that queued work resumes.

Hint

Add one assertion inside the fake that encodes a contract the real call must always meet, the way read_plate refuses an empty photo list. Then make the fake's reply include a stray line after the JSON: if your parser breaks, the test just caught a real failure before a model did.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousInstalled by a Kit Template, Placed by the Registry Next →Four TestFlight Facts and One Visible Version

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.