Compaction Between Turns

The Replay Is the Expensive Grower

Every brain builds its turn from stored history: the system prompt, then the conversation's rows replayed in order. In a conversation that never ends, that replay is the grower that costs the most, because it is paid again on every turn. On the Chief's second day, Claude's input per turn had already gone from 146k tokens to about 314k. Something has to take older rows out of what the model is handed. That is compaction: a run of older rows is replaced, in the replay only, by one summary row. Where the rows themselves go is lesson 5; this lesson is about when compaction runs, and why the answer is "not while anyone is waiting".

Two Lines per Brain

Pressure is the latest token count a provider reported for the conversation, divided by the target brain's context window (lesson 4 explains why "target" matters). Each brain has two thresholds set in Admin: a warning line, 50% by default, and an emergency line, 80% by default; a nonsense pair falls back to 50 and 80. Every brain's chat route calls the guard after it has saved Dad's message and before it rebuilds the replay for the provider:

Pressure	An ordinary conversation	Master Chief
below warning	nothing	nothing
warning to emergency	nothing	schedule a background pass and answer the turn normally
at or above emergency	a model summary now, keeping the last 6 rows	a model-free digest now (lesson 4), and schedule the background pass to upgrade it

A private conversation never compacts at all; it lives for minutes, and its summary would be one more copy of its words.

One Task per Conversation

Crossing the warning line does not start a summary each time. The scheduler keeps one task per conversation: if a pass is already running, it is marked to go once more when it finishes, and nothing else starts. Ten turns above the line produce one pass and, at most, one follow-up.

Summarize Unlocked, Write Locked

A background pass does five things, in this order:

It first upgrades any emergency digest still standing in for its rows (lesson 4).
It keeps the newest 40 rows verbatim (CHIEF_KEEP_LAST; an ordinary conversation keeps 6), and it gives up unless there are at least 20 more beyond them. Below that a utility-model read is not worth it; the next crossing waits for more turns.
It summarizes everything older than those 40 rows with the utility model, holding no lock. The rows are cut into chunks of about 18,000 characters (a chunk closes before the row that would push it past that, so a single row longer than the limit becomes a larger chunk of its own); four chunks are summarized at once; each chunk's prompt carries only its own identifiers and tool notes; and each chunk's summary is cached on disk under a hash of the chunk's text, so a restart resumes instead of starting over. A chunk the model cannot summarize becomes a bounded list of its turns, never the chunk copied verbatim.
It takes the Chief's turn lock, the same first-come, first-served lock every doorway's turn waits on, only to write. Inside it, it re-reads the live replay and checks that it still begins with exactly the rows it summarized. If an emergency digest or an edit moved it, the pass gives up; the next crossing summarizes the replay as it is then, and the cached chunks make that cheap.
It writes the compaction (rows flagged, one summary row, a marker in the log) and releases the lock, then flags the rows' search vectors outside it.

Meanwhile every turn that arrives runs on the replay as it stands. Nobody waits for a summary.

Why It Is Built This Way: Cold Review Finding #1

The first version compacted at the emergency line, inside the Chief's turn lock, before the turn was marked as streaming. The cold review measured what that meant on the live Chief: 418 messages would become 63 chunks, every chunk prompt repeated about 45 KB of tool and file notes, and at the 80% trigger the reviewer projected roughly 200 to 250 utility-model calls, one after another, each allowed two minutes. For all of that time every doorway, every cron report and every Coop notification waited. Stop could not reach it, because Stop acts on a streaming turn and this one had not started streaming. The backend had restarted about 30 times in three days, and a restart threw the half-done summary away, so the next turn began again: in the reviewer's words, compaction might never finish. And when the summarizer failed, the fallback kept each chunk verbatim, and the formatted text was 6.4 times the size of the replay it was meant to shrink. Commit 2154ed97 (2026-10-08) is the rewrite you just read.

Be clear about what this costs, because it is not free. A pass can lose its race and be thrown away. Every pass spends utility-model calls. And a summary is lossy by nature: the model sees what the summarizer chose to keep, which is why the summarizing prompt insists on Dad's decisions, open tasks, exact names and identifiers, and why the rows stay findable after they leave (lesson 5).

Do the slow work without the lock; take the lock only to commit, and first check that what you read is still true. This is optimistic concurrency. It turns "everyone waits for the slowest job" into "the slowest job sometimes redoes its work", and in a room every doorway shares, that is the right trade.

Code

Engine excerpt (faithfully trimmed, the background pass): summarize unlocked, re-check under the lock, then write·python

async def run(store: Any, conversation_id: str, *, brain: str, max_tokens: int | None = None) -> dict[str, int]:
    """One background pass: upgrade every digest still standing in for its
    rows, then take the replay beyond the kept window out behind one model
    summary. Returns what it did."""
    from services import context_compaction_guard as guard

    done = {"upgraded": 0, "compacted": 0}
    for row in await store.get_messages(conversation_id):
        if guard.is_digest_row(row) and await upgrade_digest(store, conversation_id, row.id, brain=brain):
            done["upgraded"] += 1

    messages = await store.get_messages(conversation_id)
    keep = guard.CHIEF_KEEP_LAST
    if len(messages) < keep + MIN_BATCH:
        return done
    to_summarize = messages[:-keep]
    ...
    summary = await guard._build_runtime_summary(
        to_summarize, brain=brain, before_pct=before_pct, cache_dir=guard._cache_dir_for(store),
    )

    from services import master_chief

    async with master_chief.turn(conversation_id, store=store):
        live = await store.get_messages(conversation_id)
        prefix = [m.id for m in to_summarize]
        if [m.id for m in live[:len(prefix)]] != prefix:
            # The replay moved underneath (an emergency digest, an edit). The
            # next crossing summarizes it as it is then; the chunk summaries
            # stay cached for it.
            logger.info("Master Chief compaction for %s found the replay moved; left for the next pass", conversation_id)
            return done
        await guard.apply_compaction(...)  # rows flagged, summary row, JSONL marker
    await guard.flag_compacted_vectors(conversation_id, prefix)

A minimal sketch you can run: turns keep flowing while a slow summary runs, and an emergency digest that lands meanwhile makes the pass give up·python

# A minimal sketch: summarize with no lock held, take the turn lock only to
# write, and give up if the replay moved underneath.
import asyncio

KEEP = 3


async def summarize(batch: list[str]) -> str:
    await asyncio.sleep(0.3)  # the slow utility-model call
    return f"[summary of {batch[0]}..{batch[-1]}]"


async def scenario(emergency_meanwhile: bool) -> None:
    rows = [f"m{i}" for i in range(10)]
    lock = asyncio.Lock()  # the conversation's turn lock

    async def turn(text: str) -> None:
        async with lock:
            rows.append(text)
            await asyncio.sleep(0.05)  # the brain answers
            rows.append("re: " + text)
        print(f"  turn answered: {text}")

    async def emergency_digest() -> None:
        async with lock:  # a turn met the emergency line and wrote a digest first
            rows[:5] = ["[digest of m0..m4]"]
        print("  emergency digest written at the start of a turn")

    async def background_pass() -> None:
        to_summarize = rows[:-KEEP]                # read, no lock
        summary = await summarize(to_summarize)    # no lock: doorways stay open
        async with lock:                           # the lock only to write
            if rows[:len(to_summarize)] != to_summarize:
                print("  replay moved underneath; left for the next pass")
                return
            rows[:len(to_summarize)] = [summary]
            print(f"  compacted {len(to_summarize)} rows")

    jobs = [background_pass(), turn("from the phone"), turn("from telegram")]
    if emergency_meanwhile:
        jobs.insert(1, emergency_digest())
    await asyncio.gather(*jobs)
    print("  replay now:", rows)


for flag in (False, True):
    print(f"emergency meanwhile = {flag}")
    asyncio.run(scenario(flag))

Exercise

Find a slow operation in your own code that runs while holding a lock or a transaction that other requests need: a cache rebuild, a report, a summary, a migration step. Restructure it into four steps: read a snapshot, do the slow work with no lock, take the lock, and verify that the part of the snapshot you depend on is unchanged before writing. Then add a test that changes the data during the slow step and asserts the write is abandoned and retried later.

Hint

Decide exactly what 'unchanged' means before you code it. The Chief's pass compares identities (the ids of the rows it summarized), not contents or counts: a count can match by accident, a list of ids cannot.

The Replay Is the Expensive Grower

Two Lines per Brain

One Task per Conversation

Summarize Unlocked, Write Locked

Why It Is Built This Way: Cold Review Finding #1

Code

External links

Exercise

Progress

Comments 0