Skip to content
C.W.K.
Stream
Lesson 08 of 08 · published

A Conversation That Never Ends — Segments, Compaction That Keeps the Record, and Pages With a Byte Budget

~15 min · ground-truth, jsonl-segments, compaction, paging, unbounded-growth

Level 0Curious
0 XP0/93 lessons0/18 achievements
0/100 XP to next level100 XP to go0% complete

Every rule in this track assumed a conversation ends

One JSONL file per conversation. Healing that re-reads the log on every GET. A client that fetches the whole transcript and draws it. All of that was written for conversations that start, run for a while, and stop. In October 2026 the household got one that does not stop: Dad's default Pippa conversation, shared across every app doorway (its own quest, Master Chief, tells why it exists). It reached 16.4 MB and 47,775 JSONL lines in two days.

Nothing broke loudly. Each reader just got slower in proportion to a history that would only grow, and a few of them quietly started dropping things. The cold review that followed (coop todo #351) turned one sentence into a rule the codebase now carries: nothing about a conversation may grow without bound. The fixes are generic. They apply to every conversation that gets long enough, not only to the one that forced them.

The log becomes a sequence of files

The active file keeps its canonical path, sessions/{conversation_id}.jsonl, and every writer appends there exactly as before. When a SessionLogger opens a conversation that no other logger in the process holds and finds the active file at or past PIPPA_JSONL_SEGMENT_BYTES (8 MiB), it seals it first: the whole file is renamed into sessions/{conversation_id}.segments/{seq:06d}.jsonl, and the new active file opens with a pippa_segment_sealed line. A turn is written by one logger, so a turn never straddles two files, and a sealed segment never changes again.

The trap is on the reading side. A whole-history reader that globs sessions/*.jsonl sees only the active files and silently loses everything sealed. So readers go through store/session_logger.py: conversation_jsonl_paths / read_conversation_events for the whole history (exports, the restore drill, the archive sweep, the purge), iter_conversation_files_newest_first for reads that can stop early (healing, the tool-use backfill), and all_session_jsonl_files for whole-store passes. The house sign lists it as a gotcha, because the wrong glob still compiles and still returns files.

Compaction leaves the replay, never the record

Every brain replays the conversation's history on every turn (vessels track, lesson 1). A conversation that never ends needs a replay that does not grow forever, so old rows are summarized and taken out of the replay chain. The first version did that by archiving the rows into a side table and deleting them. Every keyed lookup lost them at once: request-id dedupe, report idempotency, turn status, the memory engine's pointers.

The shipped version flags rows in place instead. ConversationStore.compact_messages sets compacted_at and compacted_by (the summary row's id). get_messages and healing read only live rows; every keyed lookup still finds a compacted row; a conversation delete still takes it; and its vectors carry compacted: true, so retrieval can still reach it from inside the same conversation. The JSONL gets a pippa_compaction marker carrying the summary text. The record stays whole; only the replay got shorter.

Principle: Bound what a turn has to read, never what the system remembers. Deleting old rows to make a replay shorter turns a performance fix into data loss, and every index keyed on those rows finds out one at a time.

Pages with a byte budget

The full GET /api/conversations/{id} still exists, but no client draws a long conversation from it any more. GET /api/conversations/{id}/page serves the display transcript one window at a time (limit, before, start), and the server bounds every window by bytes as well as rows: PAGE_BYTES_MAX, 512 KiB of JSON, whatever the row count. The window is chosen from the newest end, and rows leave its oldest end until it fits, so the newest rows always come; the page itself is returned in reading order, oldest first, with earlier counting what is left for the next page. One row larger than the budget is served alone and whole: a byte budget may split a window, never a message. A row count was not enough. One compaction summary row made the newest 200 rows weigh 2.47 MB, and the phone dropped a block of Dad's MacBook turns while decoding it (coop todo #374). Compaction summaries are replay state, so they stay off the page entirely. The WebUI and every Sidekick embed (through the shared chat hook), the Pippa app, and Telegram's sync all page now.

Healing learned what a turn could touch

Lesson 4 says healing is cheap because it reads the JSONL anyway. With segments, "the JSONL" can be many files, and the tool-use backfill used to read all of them on the serving loop for every GET of a reply that had no tools to find. Two changes followed. The GET-time repairs now run in a thread and stop as soon as they have read the file a turn could have touched. And they wait for turn_in_progress(), which covers any stream, a cron turn, a Chief turn holding its lock, or a pipeline between admission and its first event; a turn whose prologue ran before the stream flag was once healed as interrupted, and its real reply then landed as a second sibling.

The archive writes parts

The append-only conversation archive used to copy each conversation whole on every run. For the longest conversation that meant the whole history again every day. Now the first archive file is the whole conversation and every later file is a part holding only messages the archive does not have yet, tracked in the directory's .archive-manifest.json along with the segments already read. And the sweep refuses to run at all where it cannot read the session logs, rather than writing an archive that silently misses them.

Self-reference: The longest record I keep is the one I talk to Dad in most. It is split into sealed files, its old turns are summarized for my replay, and every screen reads it a page at a time. None of that makes me remember less. It makes sure a conversation with no end can still be read, replayed and repaired on the hardware it actually runs on.

Code

Where a long conversation lives now·text
sessions/
  {id}.jsonl                    active file, every writer appends here
  {id}.segments/
    000001.jsonl                sealed at 8 MiB (PIPPA_JSONL_SEGMENT_BYTES)
    000002.jsonl                never changes again

seal: a logger opening a conversation no other logger holds,
      active file >= 8 MiB -> rename whole into .segments/,
      new active file starts with a pippa_segment_sealed line

read the whole history  -> conversation_jsonl_paths / read_conversation_events
read newest-first        -> iter_conversation_files_newest_first
whole-store passes       -> all_session_jsonl_files
glob sessions/*.jsonl    -> sees active files only (the trap)
Compaction that keeps the record·text
first version   archive rows to a side table, DELETE them
                -> request-id dedupe, report idempotency, turn status,
                   memory pointers all lost the rows

shipped         flag in place: compacted_at, compacted_by = summary row id
  get_messages / healing   live rows only (the replay)
  keyed lookups            still find compacted rows
  conversation delete      still takes them
  vectors                  compacted: true, still reachable by RAG
  JSONL                    pippa_compaction marker carries the summary

page window     GET /api/conversations/{id}/page
                chosen from the newest end, returned oldest first
                at most 512 KiB of JSON (PAGE_BYTES_MAX);
                one row over budget is served alone, whole
                compaction summaries never on the page

Exercise

Pick the longest-lived record in a system you run: a log, a chat history, an event stream. Write down every reader of it and what each one reads: the whole thing, the newest part, or one keyed row. Then imagine the record ten times its current size and mark which readers would get slower, which would start dropping data, and which would break a keyed lookup if old entries were deleted to save space.
Hint
The dangerous reader is rarely the slow one. It is the one that keeps working while silently seeing less, like a glob that only matches the newest file.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.