Skip to content
C.W.K.
Stream
Lesson 04 of 05 · published

Blockers Loud, Success Quiet

~15 min · followups, alerting, delivery, noise

Level 0Visitor
0 XP0/33 lessons0/11 achievements
0/100 XP to next level100 XP to go0% complete

Three Volumes

An armed check speaks at three volumes, and the choice of volume is the whole design. Dad delegates a lot. If every check that passed rang his phone, the phone would ring all day for things that went fine, and the one ring that mattered would arrive already ignored. So the rule is lopsided on purpose: pending is silent, success is a quiet line, a blocker is loud.

What the probe sawTelegramMaster ChiefThe watch
Pending (busy, idle, queued, a transient outage)nothingnothingkeeps checking; every poll still recorded
First blockeralert at oncea system line, after the alertkeeps checking until success or the deadline
The same blocker again, or another onenothingnothingkeeps checking
Successnothingone line: success_messagedone
Deadline without successalerta system line: the deadline blockerdone, never retried

Success posts one line into Master Chief under an idempotent source key, followup: plus the check's id, through the same durable master_chief.report the last lessons followed. No external alert: the alert state is set to not_required. Dad reads it the next time he talks to Pippa, from whatever doorway, and the Chief already has it in her history.

Why the Loud Part Goes First

For a blocker the order is Telegram, then the Chief. The code says why in one comment: Urgent alerts precede the possibly busy Chief conversation lock. Dad might be in the middle of a long turn with the Chief. The report would wait for that turn to end, and an alert that waits for a busy conversation is not urgent anymore. So the alert goes out first, and the Chief line is attempted with a two-second bound. If the Chief is busy, the attempt simply returns; the line is already durable in the Chief's outbox and lands when she is free, and the next tick asks again under the same key, which can only find it or land it once.

The First Blocker Has Its Own Receipt

A blocker does not end the watch. The worker might clear it in five minutes, or Dad might answer the question it raised. So the check keeps polling, and that creates a risk: polling every 30 seconds against a blocked worker would alert Dad every 30 seconds. The fix is a separate record. The first blocked observation is stored once, in its own blocker column (COALESCE: never overwritten), with its own delivery state and its own Chief source key, followup: plus the id plus :blocked. Once it is delivered it is never delivered again, while the observation log still records every poll and every changed reason for the status action.

If the worker then finishes, success arrives as its own terminal report, and it says exactly success_message. Until 2026-10-10 the code appended an English “Recovered from the earlier blocked state.” to it. Commit 7a404b4e removed that: success_message is what Dad reads. The follow-ups doc still describes the old suffix; the code is the truth.

A Deadline Is Never Success

When the deadline passes without success, the watch becomes blocked with the reason deadline_without_completion_evidence. It is never counted as success because nothing went wrong, and it never retries the work it was watching. If the last probe happened to land on a momentary outage, the most recent real blocker is kept, with the time it was seen, rather than being erased by that blip. Rechecks are scheduled no later than the deadline, even when the normal interval would overshoot it, so the verdict arrives on time. Could not be verified is a true sentence; done would not be.

Reserve, Then Send

Telegram has no idempotency receipt. Ask it twice and Dad's phone rings twice. So before the deliverer calls it, it saves alert_state = 'sending'; afterwards it saves sent, or puts it back to pending if the channel refused, to try again. If the process dies between those two saves, the next run finds sending and records uncertain instead of sending again. The Chief line is still delivered, so Dad still has the news in his conversation; status shows the uncertainty apart from the work's own outcome. Compare Coop's notify door from Lesson 2: it marks Telegram sent after the send, so a crash between the two can ring twice, and the Coop Quest's Track 7 explains why that door accepts it. Two places, two honest choices: an outcome line is better twice than never, and a follow-up alert has the Chief line as its backstop.

What Dad Reads

A blocker alert is the armed blocked_message, written in Dad's language at arm time, plus the worker's own for_dad lines, and nothing else. On a deadline with no fresh lines, it borrows the last real blocker's. The English record (report body, reason codes, delivery errors) stays in the observation for status and never reaches Telegram or the Chief. That was learned the hard way: the #373 blocker reached Telegram and Master Chief as a wall of English, the worker's whole report body plus a reason code after the Korean lead. Commit 7a404b4e (2026-10-10, todo #375) cut it to the lead plus for_dad. And since 2026-10-12 the Chief line is born a followup system line, which Telegram's sync never mirrors, so a blocker rings once: from the alert, not again from the transcript.

Volume is a promise about what deserves attention. Keep success quiet and a ring keeps its meaning. Give the first blocker its own receipt and a stuck worker cannot turn into a siren. And never let a deadline say done.

Code

Delivering the first blocker or the terminal outcome (services/followups.py, verbatim excerpt)·python
async def _deliver(row, delivery, text, source_id, *, blocked, interim=False):
    """Deliver either the first blocker or the terminal outcome durably."""
    async def save(**fields):
        delivery.update(fields)
        update = {"blocker": _json(delivery)} if interim else fields
        await asyncio.to_thread(_update_delivery, row["id"], **update)

    if delivery.get("report_id"):
        return
    try:
        if blocked and delivery["alert_state"] == "pending":
            await save(alert_state="sending")
            delivered = await notify(text)
            await save(alert_state="sent" if delivered else "pending")
        if blocked and delivery["alert_state"] == "pending":
            raise RuntimeError("Blocker notification channel is unavailable")
        if blocked and delivery["alert_state"] == "sending":
            # A process died after reserving the send: no blind duplicate alert.
            await save(alert_state="uncertain")
        # Urgent alerts precede the possibly busy Chief conversation lock.
        async with asyncio.timeout(2):
            mid = await report(text, source_id, row["conversation_id"])
        if not mid:
            raise RuntimeError("Origin conversation no longer accepts Chief delivery")
        await save(report_id=mid, delivery_error=None)
    except TimeoutError:
        return
    except Exception as exc:
        log.warning("Follow-up %s delivery failed: %s", row["id"], type(exc).__name__)
        await save(delivery_error=type(exc).__name__)
What Dad reads of a blocker (services/followups.py, verbatim excerpt)·python
def _blocked_text(spec, observation):
    """What Dad reads of a blocker: the armed lead and the worker's own lines.

    The lead is written in Dad's language at arm time and the worker's
    for_dad lines are its plain report to him. The English record (report
    body, reason codes, delivery errors) stays in the observation for the
    status action and never reaches Telegram or the Chief (todo #375).
    """
    evidence = observation.get("evidence") or {}
    lines = evidence.get("for_dad")
    if not lines and observation["reason"] == "deadline_without_completion_evidence":
        lines = ((observation.get("last_blocker") or {}).get("evidence") or {}).get("for_dad")
    return spec["blocked_message"] + ("\n" + lines if lines else "")

External links

Exercise

List every notification that reached your phone from your own systems in the last week: CI, uptime monitors, bots, scripts. Sort each into one of three bins: needed me now, worth reading later, should have been silent. Then change one source so that its successes go to a quiet log you read on your own time, and its failures still reach you at once, once per incident.
Hint
For the once-per-incident part, store the first failure with its own key and a delivery state, and check it before every send. A key and a sent flag are not enough on their own: decide what a crash between the send and the save means. Save 'sending' before the send and 'sent' after it. On restart, a leftover 'sending' is your choice between at most once (record it as uncertain and do not resend, as the Chief's follow-ups do) and at least once (resend and accept a rare duplicate). A channel with no idempotency key cannot give you exactly once. Keep polling after the first failure, but write the later observations to a log instead of to your phone.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.