The Emergency Digest

When the Background Pass Is Too Late

The warning line schedules a background pass, and most of the time that is enough. Sometimes it is not. A long tool-heavy turn can jump the pressure past the emergency line in one step; a pass can lose its race and be thrown away; the utility model can be down; or Dad can switch the next turn to a brain with a much smaller window. Then a turn arrives at or above the emergency line, and the replay has to shrink before the provider sees it.

An ordinary conversation simply runs a model summary right there. Master Chief cannot, because blocking a room every doorway shares on a provider call is exactly what cold review finding #1 was about. So the Chief gets something that needs no model at all.

What a Digest Is

A digest is pure CPU work. For each older row it writes one line: the time to the minute, who spoke, the row's id, the opening of its text (up to 240 characters, with a count of what was cut) and the names of any tools it ran. The whole list has a budget of 60,000 characters. When there are many rows, each line's excerpt shrinks to fit (never below 60 characters), and if even that overflows, the newest rows are listed and the rest are counted, with a line saying they are in the transcript and in recall. Underneath come up to 120 preserved identifiers and up to 40 tool and file notes, each list saying how many it left out. A digest names turns; it does not carry them.

How Many Rows to Keep: Halving From 40

The planner starts with a kept window of 40 rows (or one less than the whole replay, when that is smaller), builds the digest for everything older, and estimates the replay that would result: what it was, minus the rows leaving, plus the digest. The system prompt stays in that estimate; the first version estimated "summary plus kept rows" alone and dropped the system prompt, which is most of a fresh Chief turn. If the estimate is under the emergency line (the window times the emergency percentage), it stops. If not, it halves the window: 40, 20, 10, 5, 2. It never goes below 2, and at 2 it stops whether or not the estimate is under the line: if the system prompt and the last two rows are already over it, no digest can fix that, and the turn goes ahead on the smallest replay a digest can make. All of this runs in a thread, off the event loop, and then the turn replays the digest plus the kept rows.

One correction to the design doc, which says the window halves "until the estimate fits the target brain's window". The code aims under the emergency line, 80% of the window by default, and stops at two kept rows even when it has not got there. Teach the code: landing under the line, when it can, leaves headroom, so the very next turn does not arrive straight back in an emergency. And "until it fits" was never a promise the planner could keep.

Counting Korean Honestly

None of this calls a tokenizer; on the hot path the engine estimates from characters. The first version divided the length by four, which is a fair guess for English and a bad one for a conversation held mostly in Korean. The code comment says it plainly: plain len/4 read Korean text at a third of its size. Now each Hangul syllable or CJK character counts as one token and everything else four characters to one. That deliberately errs high. Overcounting means compacting a little early; undercounting means a replay the estimator calls fine and the provider refuses.

Measured Against the Brain That Will Read It

Master Chief is one conversation that every brain replays, so the latest usage snapshot is often another brain's. The first version divided by the snapshot's window. A Claude snapshot of 314K tokens against Claude's 1M window read as 31%, so nothing compacted; but the next turn might go to Codex, whose window is 272K, and the replay was already larger than that. Cold review finding #3: Codex, and every brain except Claude, could never be saved by compaction. Now a snapshot taken by another brain lends its token count, never its window; the pressure is always the target brain's. Two more holes closed the same day: after a restart (snapshots live in memory) the newest replayed Claude row's stored prompt size stands in, and a turn a provider refuses for its size records the conversation as full for that brain, so the next turn compacts first instead of failing the same way forever.

Then the Upgrade

The digest is a stand-in, and the background pass replaces it. Before anything else, the pass finds each digest row still in the replay, fetches the rows it stands for (the compaction recorded them), and summarizes them with the utility model, holding no lock. Then, under the turn lock, it checks the row is still a digest and rewrites its text in place: the row keeps its id and its position, only its words change, and the log gets a marker saying the digest was upgraded. The cheap answer arrived on time; the good answer arrives when it is ready.

Never make a shared turn wait on a remote call. Answer now with something cheap, bounded and honest about what it left out, and improve it later out of everyone's way. And measure the pressure against whoever reads next, never whoever read last.

Code

Engine excerpt (verbatim, the compaction guard): the Korean-aware estimate and the halving planner·python

def _estimate_tokens(text: str) -> int:
    """Rough, and conservative for Korean: a Hangul syllable or CJK
    character counts as one token, everything else four characters to one.
    Plain len/4 read Korean text at a third of its size (cold review #1)."""
    if not text:
        return 0
    cjk = len(_CJK_RE.findall(text))
    return max(1, cjk + (len(text) - cjk) // 4)


def _plan_digest(
    messages: list[Any],
    *,
    keep_max: int,
    before_total: int,
    max_tokens: int,
    emergency_pct: int,
    brain: str,
    before_pct: float,
) -> tuple[int, str, int]:
    """The largest kept window, up to ``keep_max`` and halving to two, whose
    estimated replay after a digest compaction is under the emergency line;
    with that digest and the estimate."""
    line = max_tokens * emergency_pct / 100 if max_tokens > 0 else 0
    tokens = [_message_tokens(m) for m in messages]
    keep = max(2, min(keep_max, len(messages) - 1))
    while True:
        summary = build_digest_summary(messages[:-keep], brain=brain, before_pct=before_pct)
        after = _after_estimate(before_total, sum(tokens[:-keep]), summary, messages[-keep:])
        if keep <= 2 or not line or after < line:
            return keep, summary, after
        keep = max(2, keep // 2)

A minimal sketch you can run: len/4 against the engine's estimate on one Korean line, and a kept window halving under Codex's emergency line·python

# A minimal sketch of the emergency digest's two calculations: a
# Korean-aware token estimate, and the kept window halving from 40.
import re

CJK = re.compile(r"[ᄀ-ᇿ぀-ヿ㄰-㆏㐀-䶿一-鿿가-힯]")


def estimate_tokens(text: str) -> int:
    """Each Hangul syllable or CJK character is one token; the rest, four characters to one."""
    if not text:
        return 0
    cjk = len(CJK.findall(text))
    return max(1, cjk + (len(text) - cjk) // 4)


line = "아빠, 오늘 회의 끝나면 그 리뷰 결과부터 같이 보자."
print(f"len/4: {len(line) // 4} tokens   estimate: {estimate_tokens(line)} tokens")

# The digest plan, with token counts standing in for rows.
WINDOW = 272_000           # the TARGET brain's window (Codex), not the last brain's
EMERGENCY_PCT = 80
SYSTEM_PROMPT = 34_000
DIGEST_LINE = 70           # roughly one digest line per older row
rows = [215] * 560 + [4_000] * 40      # older turns, then a tool-heavy recent stretch
before_total = SYSTEM_PROMPT + sum(rows)
emergency_line = WINDOW * EMERGENCY_PCT / 100
print(f"before: {before_total:,} tokens = {before_total / WINDOW:.0%} of the window; line at {emergency_line:,.0f}")

keep = max(2, min(40, len(rows) - 1))
while True:
    removed = sum(rows[:-keep])
    digest = DIGEST_LINE * (len(rows) - keep)
    after = before_total - removed + digest     # the system prompt stays in the estimate
    print(f"  keep {keep:>2}: after ~{after:,} tokens")
    if keep <= 2 or after < emergency_line:
        break
    keep = max(2, keep // 2)
print(f"kept window: {keep} rows")

Exercise

If your project sends conversation history to a model, measure your own estimator. Take ten real messages in each language your users write in, get the true token count from your provider's counting endpoint or its usage report, and compare it with whatever estimate your code uses (len/4 is the usual one). Write down the ratio per language. Then decide, and record in a comment, which direction your estimate should err and by how much.

Hint

Use real text, not lorem ipsum: code blocks, URLs and mixed-script sentences behave differently from prose. If one language is off by a factor of two or more, an estimator that is right on average is still wrong for the users who write in it.

When the Background Pass Is Too Late

What a Digest Is

How Many Rows to Keep: Halving From 40

Counting Korean Honestly

Measured Against the Brain That Will Read It

Then the Upgrade

Code

External links

Exercise

Progress

Comments 0