One Turn at a Time

Two Devices, One History

A shared room invites collisions. Dad sends from Telegram while a WebUI reply is still streaming. The Pippa app comes back online and drains a message he typed in a dead zone. A browser tab and the phone both hold a draft. Every one of those arrives as its own request, and each request, left alone, would read the history as it is at that instant, pick the newest row as its parent, and call a model. Two requests doing that at once produce a tangle: user rows chained to user rows, replies attached to the wrong question, and a model answering a message without seeing the one sent a second earlier. In a conversation that is meant to last forever, a tangle is not a glitch; it is a permanent fork.

A Fair Lock, Held Through Persistence

The answer is master_chief.turn(), an async context manager every admitted Chief turn enters before the brain's handler runs. It keeps one asyncio.Lock per conversation (keyed by the resolved database path and the conversation id), and Python's lock is fair: when several coroutines wait, the one that started waiting first proceeds first. So turns from every device are admitted in arrival order. The lock is held from admission through the end of the reply and its persistence: the route's producer enters it, calls the original chat handler, drains the whole streamed reply, and only then lets go. The next turn therefore starts from a history that already contains the previous answer.

Inside the lock it enters one more gate, chief_turn, which predates Master Chief: it waits, in 0.1-second steps, while the conversation has a stream in flight, a scheduled (heartbeat) turn running, another chief turn, or a chat pipeline between admission and its stream. Cron turns and Coop's own chief coordinator already used that exclusion. By joining it, a Chief turn, a scheduled turn and a Coop wake can never overlap on one conversation. The cold review of 2026-10-08 checked admission order and the parent chain against the live room and found it sound: no dangling parents, no user row with more than one assistant child.

New Turns Hang Off the Newest Reply

The client's idea of the parent is not trusted. A phone cannot know what Telegram sent a minute ago, so for every new turn the route simply drops the parent_id the client posted. Once admitted, the chat handler asks the store for the newest live row (the newest message compaction has not taken out of the replay) and hangs the turn there. The engine's test does exactly this: it sends a second message with a parent of "stale" while the first is still answering, and asserts the second lands as a child of the first reply.

Only the Latest Message Can Be Edited

Pippa's conversations are trees: editing a message creates a sibling branch. In an ordinary chat that is a feature. In the Chief it was cold review finding #7. Every doorway continues from the newest row on any branch, so one WebUI edit of an older message made that branch the newest: the phone and Telegram moved onto it, and every turn after the edited point, days of them, dropped out of the model's path. The fix (7920a88f) is edit_refusal: an edit runs only if its parent is the latest user message's parent, otherwise it is refused with "In Master Chief only the latest message can be edited: every device continues from it, so an older one stays as it was." The check runs twice, before admission (a 409) and again after the lock is acquired (an edit_refused error event), because a turn queued ahead can change which message is the latest. The WebUI shows the edit control on that one message only.

The same finding had a second half. An edit used to carry the edited message's own context, so a WebUI edit of a Telegram turn went back out to Telegram, and a phone turn's edit filed the browser's clock under the phone. Now an edit is made where Dad edits: the WebUI strips the origin fields from the context it sends, and the route drops any origin naming a doorway that cannot edit (Telegram, the Pippa app, PippaGo, scheduled tasks).

When many writers share one sequence, the server picks the position. Clients may say what they want to add. Only the component that sees every arrival can say where it goes.

The Running Turn Keeps Its Tools

Lesson 2 promised that host tools follow the admitted turn. Here is the mechanism. Once turn() holds the lock, it sets a ContextVar, _origin, to this turn's conversation and origin, and resets it on the way out. Each asyncio task runs in its own copy of the context, and a message waiting for the lock is a different task, so nothing it carries can reach into the running turn's value. A tool callback that arrives through a separate HTTP request does not inherit the variable at all; it falls back to the origin persisted on the admitted user message.

Code

Engine excerpt (verbatim, services/master_chief.py): FIFO admission, held through persistence·python

@asynccontextmanager
async def turn(conversation_id: str, *, store: ConversationStore | None = None,
               origin_conversation_id: str | None = None, cancellable: bool = False):
    """FIFO admission before history/parent selection, held through persistence.

    asyncio.Lock is fair. App bindings are task-local, so an arriving message
    cannot replace the active turn's tools while it waits for admission.
    """
    store = store or ConversationStore()
    key = (str(Path(store.db_path).resolve()), conversation_id)
    lock = _turn_locks.setdefault(key, asyncio.Lock())
    task = asyncio.current_task()
    waiting = _waiting.setdefault(conversation_id, set()) if cancellable else set()
    if cancellable:
        waiting.add(task)
    from app.streaming import chief_turn
    try:
        async with AsyncExitStack() as stack:
            await stack.enter_async_context(lock)
            await stack.enter_async_context(chief_turn(conversation_id))
            waiting.discard(task)
            token = _origin.set((conversation_id, origin_conversation_id))
            try:
                yield
            finally:
                _origin.reset(token)
    finally:
        waiting.discard(task)
        if cancellable and not waiting:
            _waiting.pop(conversation_id, None)

A minimal sketch you can run: three devices, one fair lock, the server picks the parent (set LOCKED = False to see the tangle)·python

from __future__ import annotations

import asyncio

LOCKED = True                  # set to False to watch the room tangle

rows: list[dict] = []          # the conversation: {"id", "role", "text", "parent"}
locks: dict[str, asyncio.Lock] = {}


class NoLock:
    async def __aenter__(self):
        return self

    async def __aexit__(self, *exc):
        return False


def newest_id() -> str | None:
    return rows[-1]["id"] if rows else None


async def model(text: str) -> str:
    await asyncio.sleep(0.2 if text == "from the phone" else 0.01)  # one slow reply
    return "reply to " + text


async def turn(conversation: str, text: str, client_parent: str | None) -> None:
    lock = locks.setdefault(conversation, asyncio.Lock()) if LOCKED else NoLock()
    async with lock:                          # fair: waiters enter in arrival order
        parent = newest_id()                  # the client's parent is ignored
        user = {"id": f"u{len(rows)}", "role": "user", "text": text, "parent": parent}
        rows.append(user)
        answer = await model(text)            # held through the reply...
        rows.append({"id": f"a{len(rows)}", "role": "assistant", "text": answer,
                     "parent": user["id"]})   # ...and through persistence


async def main() -> None:
    await asyncio.gather(
        turn("chief", "from the phone", client_parent=None),
        turn("chief", "from telegram", client_parent="stale"),
        turn("chief", "from the browser", client_parent="stale"),
    )
    for row in rows:
        print(f'{row["id"]:>3} <- {str(row["parent"]):>4}  {row["role"]:<9} {row["text"]}')


asyncio.run(main())

Exercise

Run the sketch twice, once with LOCKED = True and once with LOCKED = False, and draw both trees on paper: which user rows point at user rows, which reply answers which question. Then find the place in your own project where several clients append to one ordered thing (a chat, an activity log, a ledger) and write down who chooses each new item's position today. If it is the client, move that choice to the server behind a per-thread lock and add a test that sends two items with the same stale parent.

Hint

A per-thread lock only works within one process. If your writers live in several processes or machines, the same rule needs a database-level answer instead: a single writer, or a conditional insert that fails when the parent is no longer the newest row.

Two Devices, One History

A Fair Lock, Held Through Persistence

New Turns Hang Off the Newest Reply

Only the Latest Message Can Be Edited

The Running Turn Keeps Its Tools

Code

External links

Exercise

Progress

Comments 0