A Dropped Phone Does Not Kill the Turn

The Phone Will Drop

Dad's turns come from a phone in a pocket, a watch on a wrist, Telegram over a mobile network, a laptop that goes to sleep. A real Chief turn can run for minutes: tools, delegation, a long answer. If the server ran the turn inside the request that carried it, every dropped connection would cancel the turn halfway: the reply lost, state half written, the model's work paid for and thrown away. And the shared room makes it worse, because the next device to open the Chief would find a question with no answer.

Split the Turn From the Connection

The Chief route separates two jobs that a naive streaming handler fuses. A produce() coroutine owns the turn: it waits for the turn lock, runs the brain's original chat handler, drains that stream into an asyncio.Queue, and the reply is persisted on the way. It is started with spawn_detached, which is create_task plus a strong reference held until the task completes (the Python docs warn that an unreferenced task can be garbage-collected mid-flight), and it is registered in the app's background_streams set so a graceful shutdown waits for it. A consume() generator owns the connection: it reads the queue and yields each chunk to the client, and after every 10 seconds of silence it yields : chief waiting, a Server-Sent Events comment line that clients ignore but that keeps proxies and sockets from deciding the stream is dead while the turn waits in line or the model thinks.

When the phone drops, only consume() dies. produce() keeps draining to the end and the answer lands in the conversation. The engine's test does exactly that: it cancels the client's request mid-turn, lets the model finish, and asserts the last row reads "finished offline". The cold review of 2026-10-08 checked this shape and found it sound.

Ask Later, Without Asking Twice

A client that lost its stream needs to ask what happened without starting the turn again. So every client mints a request id before it sends: Telegram uses tg-<chat>-<message>, the WebUI webui-<uuid>, the Pippa app pm-turn-<uuid>, the watch pw-msg-<uuid>. The engine accepts 8 to 80 letters, digits, dashes and underscores, and a unique index on the messages table means one id can own one turn. The Pippa app's outbox drain is the reference client: before sending, it asks the engine whether it already has this turn. completed replays the stored answer, pending is polled, and only an unknown id starts a model call.

Two states made that honest, both from the cold review. A turn waiting for the lock had no row yet, so turn status answered 404, which every client reads as "never arrived, send it again"; a client that resent while the first copy still waited ran the turn twice (finding #2). Now a waiting turn's id is held in a queue map and reads as pending, queued: true. And a resend that races its own first send, once admitted, finds the row already there and answers a status event with state already_accepted. A status, not an error: the native outbox treats an error event as permanent and quarantines the turn, and that is exactly what it had done to turns that were already answered (finding #8).

Withdrawn Is Not Interrupted

A waiting turn can end without running in two ways, and the client must know which. When Dad presses Stop, cancel_waiting bumps the conversation's stop revision and cancels every turn still waiting (worker reports never register as waiting, so Stop never withdraws one). The producer catches the cancellation, sees the revision moved, and sends an error with state withdrawn: "Stopped before it began: this turn was withdrawn and will not run." Do not resend. When the server is going down instead, the shutdown first calls withdraw_waiting_for_shutdown, the revision has not moved, and the producer sends a status with state interrupted: "Interrupted before it began; this turn was not run. Send it again." An outbox client does exactly that once the server is back.

That shutdown path exists because of finding #2: a restart cut running turns about a second into the backend's advertised 30-second grace, because the launcher had its own fixed 3-second deadline, and queued turns simply vanished. The fix (f3bd5968) made the launcher honour the service's real exit timeout, withdrew waiting turns first so none starts a model call the grace would cut, taught Telegram to ask turn status after a cut and resend under the same tg- id only if the engine never took it, and made the WebUI return the words to its queue, held. Two smaller items followed under #13: the stop revision now lives in a chief_stop_revisions table loaded at startup, because a counter reset to 0 by a restart read as a remote Stop on every client and held every queue; and a steer ends only the running turn instead of withdrawing Dad's other devices' queued ones.

Name every way a request can end, and make the name tell the client what to do next. Completed, pending, already accepted, withdrawn, interrupted: five answers, each with exactly one correct client reaction. An error that means "try again" and an error that means "never" are not the same error.

Every Error Says Something

The last item is the smallest and the most embarrassing. The Chief route's own error events used the key message, while every client reads text. The WebUI showed "Error: undefined". Now every error and status event the route itself emits carries text, and the wire texts sit as constants at the top of the module, under a comment naming the finding that put them there.

Code

Engine excerpt (trimmed from the Chief route): the producer's two endings, and the consumer's heartbeat·python

except asyncio.CancelledError:
    if master_chief.stop_revision(chief.id) != stop_seen:
        # Dad's Stop withdrew it while it waited its turn.
        await queue.put(_event({"type": "error", "state": "withdrawn", "text": WITHDRAWN_TEXT}))
    else:
        # Not a Stop: the server is going down. A notice, not
        # an error, so a client that keeps an outbox sends it
        # again once the server is back.
        await queue.put(_event({"type": "status", "state": "interrupted", "text": INTERRUPTED_TEXT}))
    raise

# ...

producer = spawn_detached(produce())

async def consume():
    while True:
        try:
            chunk = await asyncio.wait_for(queue.get(), timeout=10)
        except asyncio.TimeoutError:
            yield ": chief waiting\n\n"
            continue
        if chunk is None:
            break
        yield chunk

A minimal sketch you can run: a detached producer persists the answer after the client's stream is gone·python

from __future__ import annotations

import asyncio

saved: dict[str, str] = {}               # request id -> persisted answer
_keep: set[asyncio.Task] = set()         # strong references to detached work


def spawn_detached(coro) -> asyncio.Task:
    task = asyncio.create_task(coro)
    _keep.add(task)
    task.add_done_callback(_keep.discard)
    return task


async def handle(request_id: str, text: str):
    """Answer one turn. Returns the generator the client reads."""
    queue: asyncio.Queue = asyncio.Queue()

    async def produce():                  # owns the turn: runs to the end
        try:
            answer = ""
            for word in ("this", "reply", "outlives", "the", "phone"):
                await asyncio.sleep(0.1)  # the model, slowly
                answer += word + " "
                await queue.put(word)
            saved[request_id] = answer.strip()  # persisted whoever listens
        finally:
            await queue.put(None)

    spawn_detached(produce())

    async def consume():                  # owns only the connection
        while True:
            try:
                item = await asyncio.wait_for(queue.get(), timeout=10)
            except asyncio.TimeoutError:
                yield ": waiting"         # a heartbeat keeps proxies patient
                continue
            if item is None:
                return
            yield item

    return consume()


async def phone() -> None:
    stream = await handle("pm-turn-0001", "a long question")
    async for word in stream:
        print("phone got:", word)
        if word == "reply":
            print("-- the phone drops --")
            return                        # the consumer dies here


async def main() -> None:
    await phone()
    await asyncio.sleep(0.5)              # later, the phone comes back
    print("turn status:", saved.get("pm-turn-0001", "unknown id"))


asyncio.run(main())

Exercise

Run the sketch and confirm the answer survives the drop. Then extend it: give handle() a check that returns "already accepted" when the request id is already in saved or still being produced, add a turn_status(request_id) that answers completed, pending or unknown, and write a client that, after a drop, asks status first and resends under the same id only on unknown. Count model calls across a drop-and-retry; the right number is one.

Hint

You need a second map for turns that are accepted but not finished, filled before produce() starts and cleared in its finally block. Without it, a status check during the turn answers unknown, and the client's resend runs the turn twice: cold review finding #2 in miniature.

The Phone Will Drop

Split the Turn From the Connection

Ask Later, Without Asking Twice

Withdrawn Is Not Interrupted

Every Error Says Something

Code

External links

Exercise

Progress

Comments 0