Skip to content
C.W.K.
Stream
More →
Lesson 02 of 04 · published

One Line Into Master Chief

~15 min · idempotency, locks, authentication, conversations, telegram

Level 0Loose Note
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

The Door on Pippa's Side

Coop's notifier does not talk to Telegram. It talks to Pippa. cwkPippa exposes a small family of engine doors under /api/coop/engine/ (bind, wake, status and notify), and the last one is how a line from the engine reaches Dad. Its body is strict: text (up to 4,000 characters), a kind, and an optional outbox_id. Any other field is rejected. The door carries the message; it does not own it. The engine's outbox row stays the durable record, which is why a refusal here costs nothing but a delay.

Who May Knock

A door that can ring Dad's phone and write into his conversation is worth guarding, and the guard is layered:

  • POST from loopback only. The engine runs beside Pippa on office and dials loopback. The accepted peers default to the two loopback addresses; a tailnet address was once a guess and is not a peer.
  • No browser on the line. A request carrying an Origin, a Cookie, an X-Forwarded-For or a Pippa session header is refused before anything else is checked. A page open in Dad's browser cannot ride his login into this door, and nothing proxied can either.
  • A service key, compared in constant time. The key travels in a header and is checked with hmac.compare_digest, so the time a wrong guess takes says nothing about how close it was. On Coop's side, the client refuses to send at all if its key file is readable by anyone but its owner. (Where the key lives and what it is stays out of this course.)
  • The family's security posture. Pippa asks the family authority for its current state; a blocked or locked posture shuts the door, and an unreadable one answers 503 rather than guessing.
  • Dad's context. The route runs only when the current user is Dad.

Two Destinations, in a Fixed Order

A call without an outbox_id has nothing to deduplicate by, so it goes to Telegram alone. Every call from the notifier carries one, and then the order is fixed:

  1. Master Chief first. master_chief.report writes the text into Dad's Master Chief conversation, keyed by the source coop-notify:<outbox id>. The door waits at most two seconds for it to land.
  2. Telegram second, once. Under a lock held per outbox row, the door checks a small table, coop_notify_sent. If this row was already sent, it answers already_sent and rings nothing. Otherwise it sends; a refused send answers 503 notification_channel_unavailable, which the engine turns into a deferral; a successful send is marked in the table.

Master Chief is Dad's one continuous conversation with Pippa: the same thread whether he opens the WebUI, the Pippa app or Telegram, never trashed and never filed away. It has its own quest, which covers that conversation itself. What matters here is that a Coop line lands there as a stored assistant row marked turn_kind coop: a system line, not Pippa's own turn. The WebUI and the phone draw it apart from her replies, with a badge where her avatar would be and no Listen control, and voice mode never reads it aloud or takes it for her answer (Coop todo #406, 2026-10-12). It is still part of the conversation's stored history, so the next time Dad talks to Pippa, the report is already in the history she reads. Delegated work reports back into the room where Dad actually is (Dad, 2026-10-08, Coop todo #348).

Durable Before Anything Waits

The interesting part is how report writes into a conversation that may be busy. Dad could be mid-turn with Pippa on his phone when a Coop outcome arrives, and a report must never interleave with a live turn. So report does three things in a deliberate order:

  • It INSERT OR IGNOREs the line into chief_report_outbox, keyed chief-report: plus the source id. From this moment the line survives anything.
  • It starts (or nudges) the one drainer for that Chief. The drainer takes the Chief's turn lock, appends every undelivered row in order as an assistant message whose external request id is that key, marked as a scheduled-task origin with no tools and with the turn_kind its source key names (coop for a coop-notify: line), and records the message id.
  • It waits, but only as long as its caller allows. The notify door allows two seconds, a cron job five. Past that, the call returns and the row lands when the Chief is free.

Before inserting anything, report checks whether a message with that key already exists and returns it if so. At startup, resume_reports drains whatever a restart left waiting. A report is now idempotent, bounded for its caller, and restart-proof.

The Bug That Rang Twice

It was not always so. The notify door first shipped on 2026-09-30 as Telegram only. Master Chief delivery was added on 2026-10-08, and a cold review the same day (finding #6) found what that addition had done. A report used to wait on the Chief's turn lock inside its caller, with no bound. Dad in a long turn held the notify request open past the engine's 15-second client timeout. The engine saw a failure and retried. Each retry added another waiter. When the Chief came free, every waiter sent its Telegram, and one event rang Dad's phone once per retry. The same shape kept a cron job "running" through a long Chief turn so its next fire was skipped, and a restart lost the report outright, because it lived only in a waiting coroutine.

The fix, commit 5536821e, is everything in the previous section plus the sent-table and lock in this one: write first, wait briefly, ring once per row.

Once, Honestly

Read the door's last lines carefully and you will see that "once" has a precise shape. The sent mark is written after Telegram accepts the message. Retries after the mark ring nothing; concurrent retries inside the process wait on the per-row lock and then find the mark. What remains is a crash between the send and the mark, and the engine's next retry would ring again. That is a choice, not an oversight: Coop's architecture promises at-least-once delivery with deduplication per destination, never global exactly-once. For an outcome Dad needs to know about, a rare second ring is cheaper than a silent first one. Lesson 4 meets a mechanism that chose the other way, and says why.

Do the idempotent step first and the non-idempotent step once. The Chief line has a natural key and lands at most once however often it is asked; Telegram has no key at all. Putting the keyed, durable write before the unkeyed send means a failed send and its retries can never duplicate the part that has a memory.

Code

cwkPippa routes/coop_chief.py, the notify door (trimmed excerpt)·python
@router.post('/engine/notify')
async def notify(body:Notify,request:Request):
    refusal=await authorized(request)
    if refusal is not None:return refusal
    ...
    if not body.outbox_id:
        # Nothing to deduplicate by; Telegram alone carries it.
        if not await send_telegram_message(body.text):
            raise HTTPException(503,'notification_channel_unavailable')
        return {'delivered':True,'channel':'telegram','outbox_id':None,'chief_message_id':None}
    # The Chief line is durable in its outbox before this waits, and the wait
    # is bounded: a busy Chief held this request past the engine's 15 s
    # timeout, every retry added a waiter, and each re-sent Telegram once the
    # Chief was free (cold review #6). It lands when the Chief is free.
    store=sibling_binder.store
    chief_message=await report_to_chief(text=body.text,source_id='coop-notify:'+body.outbox_id,store=store,
                                        wait=NOTIFY_CHIEF_WAIT_SECONDS)
    lock_key=(id(asyncio.get_running_loop()),body.outbox_id)
    async with _notify_locks.setdefault(lock_key,asyncio.Lock()):
        if await _notify_sent(store,body.outbox_id):
            _notify_locks.pop(lock_key,None)
            return {'delivered':True,'channel':'telegram','outbox_id':body.outbox_id,
                    'chief_message_id':chief_message,'already_sent':True}
        if not await send_telegram_message(body.text):
            raise HTTPException(503,'notification_channel_unavailable')
        await _mark_notify_sent(store,body.outbox_id)
    _notify_locks.pop(lock_key,None)
    return {'delivered':True,'channel':'telegram','outbox_id':body.outbox_id,'chief_message_id':chief_message}

# One Telegram line per Coop outbox row, however often the engine retries.
NOTIFY_CHIEF_WAIT_SECONDS=2.0
cwkPippa services/master_chief.py, report() (trimmed excerpt)·python
async def report(*, text, source_id, source_conversation_id=None, store=None, wait=None):
    """The report is durable before anything waits: it enters the outbox, and the
    outbox drains into the Chief whenever no turn holds it, after a restart
    too. ``wait`` bounds how long this call waits for the row (None: until it
    lands); past it the row lands later and this returns None."""
    ...
    chief = await resolve(store, as_owner=True)
    key = "chief-report:" + source_id
    landed = await _landed(store, chief.id, key)
    if landed:
        return landed
    async with aiosqlite.connect(store.db_path, timeout=5) as db:
        await db.execute(_OUTBOX_DDL)
        await db.execute(
            "INSERT OR IGNORE INTO chief_report_outbox (source_key, chief_id, text, source_conversation_id, created_at) "
            "VALUES (?, ?, ?, ?, datetime('now'))",
            (key, chief.id, text, source_conversation_id),
        )
        await db.commit()
    drainer = _drain_reports(store, chief.id)   # the one drainer for this Chief, under its turn lock
    try:
        if wait is None:
            await asyncio.shield(drainer)
        else:
            await asyncio.wait_for(asyncio.shield(drainer), timeout=max(wait, 0.0))
    except asyncio.TimeoutError:
        return None
    return await _landed(store, chief.id, key)

External links

Exercise

Pick an endpoint in your own system that does two things for one request: writes something durable and sends something external (an email, a push, a chat message). Write down, for each step, whether it has a natural idempotency key. Then simulate the client timing out and retrying three times while the first request is still running. Count how many external sends happen. Reorder and key the steps until the answer is one, and write down the single crash window that remains.
Hint
Give the durable write a key derived from the caller's stable id (an outbox row id, a message id) and use INSERT OR IGNORE or a unique constraint. For the external send, keep a sent-marker keyed the same way and guard it with a per-key lock. The crash window between 'sent' and 'marked' never fully closes; decide out loud which side of it you prefer to fail on.

Progress

Progress is local-only — sign in to sync across devices.
← PreviousNotifications Are Rows Without a SessionNext →One Facade, Many Doors
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.