The Daily Canary

The Paths Nobody Uses

Some of Soul Vault's operations run thousands of times a day: reading a note, loading identity. Others run a few times a year: creating a soul, creating a guest, editing a soul's vault, removing one. Rare paths are where bugs live longest, because nothing exercises them. A creation path broken by an unrelated change in September would be discovered the next time Dad welcomed a new soul, perhaps months later, at exactly the moment he wanted it to work.

So Soul Vault runs its rare paths every day, on the live pointer and the backend it names, as a brain-free scheduled job (soul_vault_canary, part of cwkPippa's heartbeat). No model is involved; it is code exercising code.

What One Run Does

preflight: the pointer reads, the backend opens, and no fixture is left over. A leftover from a crashed earlier run is purged first and reported.
create_soul, create_guest: a system-test soul and a system-test guest, made through the public lifecycle calls.
invisible_to_ordinary_lists: neither fixture appears in an ordinary soul list.
load_identity, load_guest_identity: each loads as identity.
edit_soul: an edit with compare-and-swap, read back, including a stale revision that must be refused as revision_conflict.
identity_note_stays: a delete of each fixture's identity note must be refused (contract v7). If one is ever accepted, the note is put back and the run is red, naming it.
own_vault_only: where the store keeps history, the fixture soul also gets a guest vault of its own slug (what promotion leaves behind), loses its instructions, and its name must fail rather than load the other vault (next lesson).
delete_and_prove_absent: both fixtures removed and proven gone.
real_soul_identity: the real soul, Pippa, loads from the production tree, so a broken production tree can never stay green behind a healthy fixture.
every_soul_whole: every other soul and guest loads its own identity too. One that lost its identity note or its instructions turns the run red by name.
identity_probe (optional): the probe, compared with its baseline, red on any new red (a fail that was not a fail there, or a surface that passed there and cannot answer now).

Fixtures That Cannot Touch Dad's Tree

System-test souls live in their own tree under Soul Vault's state root, never in Dad's vault tree, and never in an ordinary soul list on any Mac (a store lists them only to a caller that asks for them by flag). Their deletion is itself a step inside a finally block, so a failing step cannot skip it. If the deletion itself fails, the run goes red by name, and the next run's preflight purges any leftover and reports it. Cleanup is attempted every time and checked every time; it is not assumed. This is what makes it safe to run destructive checks on the live system every day.

A Canary That Is Not Blind

A monitor that is always green proves nothing until you have seen it go red. The canary has a built-in sensitivity test. Two injections deliberately break a run: alias resolves the test soul to the wrong vault, and binding reads a broken pointer. Each injected run must come out red, at the expected step. sensitivity() runs both and fails if either stays green. That is a small piece of mutation testing pointed at the monitor itself.

Receipts and Alerts

Every run writes canary/latest.json (schema cwk.soulvault-canary.v1) plus a dated copy. Watchfire's detector reads it and alerts on a red run, and also on a receipt older than its deadline (26 hours): a canary that stopped running is as alarming as one that failed. The path was proved end to end on the live store during the rare-path trial (before contract v7 made such a delete impossible): a throwaway soul's identity note was removed through the write path, the canary went red naming the soul and the note, the alert reached Dad's Telegram, the note was written back, and two clean passes resolved the issue.

Code

The canary's shape (trimmed from soulvault/canary.py)·python

SCHEMA = "cwk.soulvault-canary.v1"
STALE_AFTER_S = 26 * 3600          # a missing run alerts like a failed one

def run(open_vault, *, inject=None, probe=None):
    steps, vault = [], None
    with _step(steps, "preflight"):
        vault = open_vault()        # pointer reads, backend opens; leftovers purged and reported
    if vault is not None:
        try:
            with _step(steps, "create_soul"): ...
            with _step(steps, "create_guest"): ...
            with _step(steps, "invisible_to_ordinary_lists"): ...
            with _step(steps, "load_identity"): ...
            with _step(steps, "load_guest_identity"): ...
            with _step(steps, "edit_soul"): ...            # incl. a revision_conflict
            with _step(steps, "identity_note_stays"): ...  # v7: deletes must be refused
            if vault.capabilities().durable_history:
                with _step(steps, "own_vault_only"): ...
        finally:
            with _step(steps, "delete_and_prove_absent"):
                ...  # delete each fixture by its vault and prove it gone; a failure here
                     # is a red step, and the next run's preflight purges what was left
        with _step(steps, "real_soul_identity"): ...
        with _step(steps, "every_soul_whole"): ...
    if probe is not None:
        with _step(steps, "identity_probe"): ...       # red on any new red against the baseline
    return {"schema": SCHEMA, "ok": bool(steps) and all(s.ok for s in steps), "injected": inject,
            "failed": [f"{s.name}: {s.cause}" for s in steps if not s.ok]}

Proving the canary is not blind (sketch of sensitivity())·python

def sensitivity():
    expected = {"alias": "load_identity:", "binding": "preflight:"}
    results = {name: run(inject=name) for name in expected}
    blind = [name for name, r in results.items()
             if r["ok"] or not any(f.startswith(expected[name]) for f in r["failed"])]
    return {"ok": not blind, "blind_to": blind}

Exercise

Write a daily canary for the rarest operations of a system you run (create, rename, delete, restore). Use fixtures that live outside real data and are removed in a finally block, check the real production data once per run, write a receipt with a schema and a timestamp, alert on red and on a stale receipt, and add two injected faults that must turn the run red at a named step.

Hint

Write the sensitivity test before you trust the canary. If an injected fault does not turn it red, the canary is not checking what you think, and every green day until then was meaningless.