Skip to content
C.W.K.
Lesson 06 of 07 · published

One Ring per Work Item

~15 min · notifications, coop, telegram, idempotency

Level 0Visitor
0 XP0/44 lessons0/11 achievements
0/100 XP to next level100 XP to go0% complete

One Event, Up to Four Rings

Coop tells Dad about delegated work in plain lines, and until 2026-10-23 it often told him the same thing several times. A worker replied; a second later Coop sent the same words again as blocked; a minute after that the Coop chief parked the item for Dad, or escalated it. Each line was correct on its own, each had its own outbox row, and each reached both Master Chief and Dad's phone. In the week to 2026-10-22, of 486 Coop lines about a work item, 201 were followed within 120 s by a newer line about the same item, and 124 of those repeated it word for word. Beside them sat 185 lines of session plumbing, and flapping chats were the loudest of those: the Chief held 99 "worker session disconnected" lines since 2026-10-07, one todo's worker announced seven times in three nights, and five in a single minute during the 2026-10-22 move.

The morning the Chief's half was fixed, Dad asked for the phone too:

"텔레그램 쪽 중복 알림도 하나로 합쳐."
"Merge the duplicate alerts on the Telegram side into one, too."
— 아빠 (Dad), 2026-10-23, translated from Korean

This lesson is the machinery behind that sentence: plumbing kept out, item lines settled, the newest kept, and a held line delivered by Pippa herself.

Plumbing Never Enters

The first rule is a filter. Some of Coop's lines are about a worker session, not about work: a session came up, dropped, exited or was enrolled, an invitation expired or was declined, a follow-up could not be delivered. Those stay in Coop and on Telegram, and the Chief reads them with the coop tool when she is asked. chief_doors.COOP_INFO_REASONS lists them, keyed by the reason Coop sends and valued by the head its notifier writes, so coop_info_only recognizes such a line by Coop's own reason, or by its head when the engine sends none:

Coop's reasonThe line's head
remote_session_readyremote session ready
session_disconnectedworker session disconnected
session_exitedworker session exited
take_adoptedchat enrolled
assignment_expiredassignment setup expired
invitation_declinedinvitation declined
invitation_expiredinvitation not accepted
followup_undeliveredfollow-up not delivered

Settle, Then Supersede

The second rule is a window. The door reads the item from the line's head ([Coop] … : todo #N, through coop_item) and hands the Chief's outbox a group, coop:todo #N, and a settle of 120 s (COOP_SETTLE_SECONDS). report() writes the row with a not_before that far ahead and returns at once, because nobody should wait two minutes for a line. When a new row arrives for the same group, every older row of it still owed is marked superseded_by the new key. The drainer lands only rows whose window has passed and that nothing replaced, and it checks once more after its select, since a newer line can arrive in between.

Three details make it safe rather than merely quiet. Only a new row supersedes: Coop's engine retries a notify until it is answered, and a retry of an older line must never take the place of what came after it. A retry of a superseded row returns nothing and never lands. And a superseded row is not deleted. It stays in the outbox with superseded_by set, a record of what was said and what replaced it, and Coop keeps every line as well. Only the Chief's replay keeps just the last.

Telegram settles the same way (3097f620). A line about an item is held in coop_notify_sent as held, with its text, its group and its not_before, and a newer line for the item marks a held one superseded. A line about no item still goes at once, reserved and sent as before.

A Held Line Is Pippa's to Deliver

Holding a line changes who owns it. The door answers Coop's engine at once with settling, and the engine settles its outbox row: it will not retry. So Pippa must. One sender per database, _telegram_sender, wakes when the earliest held line is due. It reserves each due row as sending before it calls Telegram, so a superseding line that arrived a moment earlier wins, and a restart mid-send can never send the same row twice. A delivered row becomes sent. A failed send goes back to held with its not_before moved 60 s on (TELEGRAM_RETRY_SECONDS), and after 30 attempts (TELEGRAM_MAX_ATTEMPTS) it is failed and logged as an error. A read timeout counts as a failed send here too, though Telegram may already have shown the line, so that case can ring again a minute later. At startup resume_held_notifications turns any row a restart left as sending into uncertain, never to be sent again, and delivers what still waits. The Master Chief copy is durable either way, which is why at most once is the right side to err on after a crash, the same choice Track 5, lesson 4, "Blockers Loud, Success Quiet", made for follow-ups. It shares that lesson's gap too: a send whose answer never came back is retried, never counted as uncertain.

A Belt for Repeats

Settling folds lines about one item that arrive close together. A chat that flaps every few hours is another shape: the same line, far apart. Its real fix was Coop's own (cwkCoop d917c51, #599, M3): a loss already announced, with no report or note since, is not announced again, and an adapter decides nothing until a failure has lasted 60 s. Pippa's door wears a belt as well (7a993593). An attention line identical, whitespace aside, to one Telegram carried in the last 30 minutes is answered as delivered, with repeat_of naming the earlier row in that answer, recorded under its own row as repeat (the row keeps the line's hash, not which row it repeated), and sent nowhere, Master Chief included. Past 30 minutes the same line is news again.

What It Bought, and What It Costs

Replayed over the week before the gate (Track 2, lesson 6, "The Chief Is a Sanctuary", has the whole table), the Chief's 671 Coop lines become 285, none of them plumbing. In tokens that is about 1% of a Dad turn, because compaction already kept those lines small; the gain is in the reading. For Dad's phone the gain is the point: one event, one ring, and the ring carries the latest word on the item, usually the one that asks something of him (the park, the escalation), not the reply before it. The price is time and machinery. A result about a work item now reaches the phone up to two minutes after Coop wrote it, and Pippa now owns a retry loop and a small state machine (held, sending, sent, superseded, failed, uncertain, repeat) that the door used to leave to Coop's engine.

Group by what the reader cares about, then keep the newest. One notification per event is the system's view of the world. One per work item, after a short settle, is the reader's.

Code

Verbatim excerpt, routes/coop_chief.py notify(), as of 2026-10-27: plumbing stays out, the Chief's line settles, and the Telegram line is held·python
    # The Chief is a sanctuary (todo #624): a worker session's plumbing (ready,
    # disconnected, enrolled) stays in Coop and on Telegram, and the Chief
    # reads it with the coop tool when asked.
    chief_message=None
    if not chief_doors.coop_info_only(body.text,body.reason):
        # The Chief line is durable in its outbox before this waits, and the
        # wait is bounded: a busy Chief held this request past the engine's
        # 15 s timeout, every retry added a waiter, and each re-sent Telegram
        # once the Chief was free (cold review #6). It lands when the Chief is
        # free, and a line about a work item settles first: a newer line for
        # the same item within the window takes its place (one event used to
        # reach the Chief as reply, blocked, then parked or escalated).
        item=chief_doors.coop_item(body.text)
        chief_message=await report_to_chief(text=body.text,source_id='coop-notify:'+body.outbox_id,store=store,
                                            wait=NOTIFY_CHIEF_WAIT_SECONDS,group='coop:'+item if item else None,
                                            settle=chief_doors.COOP_SETTLE_SECONDS)
    answer={'delivered':True,'channel':'telegram','outbox_id':body.outbox_id,'chief_message_id':chief_message}
    if chief_task:
        answer['chief_task']=chief_task
    item=chief_doors.coop_item(body.text)
    settle=chief_doors.COOP_SETTLE_SECONDS
    if item and settle>0:
        state=await _hold_notify(store,body.outbox_id,body.text,'coop:'+item,settle)
        if state=='held':
            return {**answer,'settling':True}
        return {**answer,'already_sent':state in ('sent','repeat'),'state':state}
A minimal sketch you can run, not the engine (stdlib, Python 3.9+): lines about one item settle, the newest replaces the ones still settling, and a retry of a replaced line never lands·python
"""A minimal sketch: lines about one work item settle for a window, the newest
takes the place of the ones still settling, and a retry of a replaced line
never lands. Not the engine: one table, a fake clock, no Telegram."""
import sqlite3

SETTLE = 120.0
db = sqlite3.connect(":memory:")
db.execute("""CREATE TABLE outbox (key TEXT PRIMARY KEY, item TEXT, text TEXT,
              not_before REAL, superseded_by TEXT, landed REAL)""")
OWED = "landed IS NULL AND superseded_by IS NULL"


def report(key, item, text, now):
    """Hold a line until it settles; only a NEW row supersedes older ones."""
    inserted = db.execute("INSERT OR IGNORE INTO outbox (key, item, text, not_before) VALUES (?, ?, ?, ?)",
                          (key, item, text, now + SETTLE)).rowcount
    if inserted:
        db.execute(f"UPDATE outbox SET superseded_by=? WHERE item=? AND key!=? AND {OWED}", (key, item, key))
        return "settling"
    (by,) = db.execute("SELECT superseded_by FROM outbox WHERE key=?", (key,)).fetchone()
    return f"retry of a line replaced by {by}: never lands" if by else "retry: nothing changes"


def drain(now):
    """Land every line whose window has passed and that nothing replaced."""
    due = db.execute(f"SELECT key, text FROM outbox WHERE {OWED} AND not_before<=? ORDER BY not_before",
                     (now,)).fetchall()
    for key, text in due:
        db.execute("UPDATE outbox SET landed=? WHERE key=?", (now, key))
        print(f"t={now:5.0f}s  RING  {text}")


print(report("c1", "todo #9", "[Coop] worker reply: todo #9 ...", now=0))
print(report("c2", "todo #9", "[Coop] blocked: todo #9 ...", now=1))
print(report("c3", "todo #9", "[Coop] parked for Dad: todo #9 ...", now=62))
print(report("c1", "todo #9", "[Coop] worker reply: todo #9 ...", now=70))   # the engine retries c1
print(report("d1", "todo #12", "[Coop] done: todo #12 ...", now=80))
for t in (60, 121, 182, 200):
    drain(t)

# Output:
# settling
# settling
# settling
# retry of a line replaced by c2: never lands
# settling
# t=  182s  RING  [Coop] parked for Dad: todo #9 ...
# t=  200s  RING  [Coop] done: todo #12 ...

Exercise

Export a week of the notifications one of your systems sent you, and group them by the thing they were about (a ticket, a build, a deploy), not by the event that produced them. Count, per item, how many arrived within two minutes of another one about the same item. Then put a settle window in front of the sender: hold each line for that window, let a newer line about the same item replace a held one, and keep the replaced lines in a record instead of deleting them.
Hint
Decide who retries a held line before you build it. If your sender answers 'accepted' to the upstream system while the line is still held, the upstream stops retrying, so the holder must own delivery: reserve before sending, retry on failure with a cap, and after a restart treat a half-sent line as uncertain rather than sending it again.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.