Segments, Not One File

Written Before It Is Shown

Under every Pippa conversation except a private one sits one append-only JSONL log. (A private conversation keeps no log at all: its logger opens no file and drops every event.) The reason the log exists is a sentence Dad said, not a reliability metric: "Do not present broken chat to your dad. That hurts." Every event of a turn (text and thinking deltas, tool calls and their results, usage reports, errors, the marker when Dad presses Stop) is written to that log before it is shown. Each line is written and flushed to the operating system at once, so a crashed process loses nothing; an fsync follows every structural event, and any line written a quarter of a second or more after the last one, so even a power cut costs at most a fraction of a second of streamed text.

The Pippa Quest's track on Pippa's Truth teaches the healing that reads this log. This lesson is about what happens to it when the conversation never ends.

Encrypted at Rest, One Line at a Time

The log is encrypted at rest, and the unit of encryption is the line, not the file. That choice falls straight out of append-only: encrypting the file as a whole would mean rewriting it on every message, while an encrypted line can be appended like any other, and the reader tells an encrypted line from an old plain one by its first character. Be honest about the threat model, because the code is. It is peace of mind against accidental visibility (a file preview, a backup copy landing somewhere new), not a defense against someone who holds both the disk and the key.

It also has a cost you should notice now: every read of the log is a decrypt. Reading a conversation's whole history is not a cheap file read, and it gets more expensive every day the conversation lives. That is why the tool-history repair that decrypted the Chief's whole log for a reply was a cold-review finding, not a footnote.

One File Becomes a Sequence

Since coop #351 the log is a sequence of files. The active file keeps its canonical name, {id}.jsonl, and every writer appends to it exactly as before. When a logger opens a conversation that no other logger in the same process holds, and the active file has reached the segment size (8 MiB by default), it first seals it: the whole file is renamed into {id}.segments/ under the next six-digit number, and the new active file begins with a pippa_segment_sealed line naming where the history before it went. Sealed segments are never written again. If the rename fails, the active file stays where it was; the log never loses a line.

Two properties follow from where the seal happens, and both are worth stealing:

A turn never straddles two files. One logger writes one turn, and sealing only happens when a logger opens with nobody else writing. The boundary between segments is always a boundary between turns.
Segments are 8 MiB or larger, not exactly 8. The size is checked only at open, so a turn that crosses the threshold finishes in the same file. The Chief's first segment is 16,436,942 bytes, because its file was already that big when sealing shipped; the next two are about 8.48 and 8.52 MB. The design doc says "sealed 8 MiB segments"; the code says "has reached", and the code is what runs.

The seal is a rename, not a copy. On one filesystem a rename is atomic: at every instant the bytes are either the active file or a sealed segment, never half of each.

The Trap: A Glob Sees Only the Active Files

Once a log is segmented, the naive reader is wrong in the quietest way possible. A script that globs sessions/*.jsonl still finds a file for every conversation, still parses, still returns events, and for the Chief it reads only the last few days. Nothing errors. So every reader goes through the store's segment-aware helpers: one returns every file of a conversation oldest first (a legacy session-keyed file, the sealed segments, then the active file); one reads the whole history for the jobs where the whole history is the point (exports, the restore drill, the archive sweep, the purge); one walks newest first and lets a turn-shaped reader stop early (healing, the tool-history backfill); and one lists every file in the store, segments included, for whole-store passes. The architecture doc names the glob as the trap its paragraph exists to prevent.

The Honest Caveat: Ground Truth Is Not Every Row

"Ground truth" holds for turns that streamed through a brain adapter, and that is not every row. The first restore drill, in July, found SQLite rows with no JSONL line behind them: healing placeholders, duplicate rows from a cron fallback (fixed at the source), imported conversations (deliberately written without message ids), and branch or council children copied from another conversation. So a rebuild reads both stores, and the archive reads their union. Pippa also keeps some primary state that never had a JSONL source at all. And until the cold review, a compaction summary existed only in SQLite; now the compaction marker in the log carries its text. The invariant is strong, and its holes are written down. That is the version worth copying.

Split a growing log where nothing is in flight. Choose the boundary you already own (here, a turn), move the old file whole with an atomic rename, and make one helper the only door readers use. Then segmenting changes nothing for the writer and nothing for a reader that goes through the door.

Code

Engine excerpt (verbatim, the session logger): sealing the active file into the next segment·python

def seal_active_segment(sessions_dir: Path, conversation_id: str, *, threshold: int | None = None) -> Path | None:
    """Seal the active file into the next segment when it has reached the
    threshold. Returns the sealed path, or None when nothing was sealed.

    The caller guarantees no writer holds the conversation (SessionLogger
    calls this under its process lock, with no logger open for it). The
    rename is atomic on one filesystem; the next writer creates a fresh
    active file.
    """
    active = conv_jsonl_path(sessions_dir, conversation_id)
    try:
        size = active.stat().st_size
    except FileNotFoundError:
        return None
    if size < (segment_bytes() if threshold is None else threshold):
        return None
    directory = segments_dir(sessions_dir, conversation_id)
    directory.mkdir(parents=True, exist_ok=True)
    existing = sealed_segment_paths(sessions_dir, conversation_id)
    last = int(_SEGMENT_NAME_RE.match(existing[-1].name).group(1)) if existing else 0
    target = directory / f"{last + 1:06d}.jsonl"
    if target.exists():
        raise FileExistsError(target)
    os.rename(active, target)
    logger.info("Sealed JSONL segment %s (%d bytes) for %s", target.name, size, conversation_id)
    return target

A minimal sketch you can run: seal-on-open segments, turns that never split, and the glob that sees almost nothing·python

# A minimal sketch of a segmented, append-only conversation log.
import json
import os
import tempfile
from pathlib import Path

SEGMENT_BYTES = 200  # tiny so the demo seals; the engine's default is 8 MiB


def active_path(root: Path, cid: str) -> Path:
    return root / f"{cid}.jsonl"


def sealed_paths(root: Path, cid: str) -> list[Path]:
    directory = root / f"{cid}.segments"
    return sorted(directory.glob("*.jsonl")) if directory.is_dir() else []


def open_logger(root: Path, cid: str):
    """Seal only when a logger opens, never mid-turn: one logger writes one whole turn."""
    active = active_path(root, cid)
    if active.exists() and active.stat().st_size >= SEGMENT_BYTES:
        directory = root / f"{cid}.segments"
        directory.mkdir(exist_ok=True)
        target = directory / f"{len(sealed_paths(root, cid)) + 1:06d}.jsonl"
        os.rename(active, target)  # atomic on one filesystem; the file moves whole
    return open(active, "a", encoding="utf-8")


def write_turn(root: Path, cid: str, turn: int) -> None:
    with open_logger(root, cid) as log:
        for kind in ("user", "text_delta", "text_delta", "done"):
            log.write(json.dumps({"turn": turn, "type": kind}) + "\n")
            log.flush()  # on disk before it would be shown


def conversation_paths(root: Path, cid: str) -> list[Path]:
    """Oldest first: the sealed segments, then the active file."""
    active = active_path(root, cid)
    return sealed_paths(root, cid) + ([active] if active.exists() else [])


with tempfile.TemporaryDirectory() as tmp:
    root = Path(tmp)
    for turn in range(1, 8):
        write_turn(root, "chief", turn)

    naive = sum(len(p.read_text().splitlines()) for p in root.glob("*.jsonl"))
    events = [json.loads(line) for p in conversation_paths(root, "chief") for line in p.read_text().splitlines()]
    print(f"a glob of the directory sees {naive} lines")
    print(f"the segment-aware reader sees {len(events)} lines, turns {sorted({e['turn'] for e in events})}")
    for p in conversation_paths(root, "chief"):
        turns = sorted({json.loads(line)["turn"] for line in p.read_text().splitlines()})
        print(f"  {p.relative_to(root)}: {p.stat().st_size} bytes, turns {turns}")

Exercise

Find an append-only log in your own project (an event log, a chat log, an audit trail). Add seal-on-open: when the writer opens it and the file is past a threshold, rename it whole into a numbered segment beside it. Then write the one reader helper that returns every file oldest first, search your code for every place that opens or globs the log directly, and route each through the helper. Finish with a test that writes past the threshold several times and asserts the helper sees every event and no unit of work is split across files.

Hint

Set the threshold tiny in the test (a few hundred bytes) so it seals often. The search is the part people skip: grep for the log's directory name and its file extension, not just for the helper's old name.

Written Before It Is Shown

Encrypted at Rest, One Line at a Time

One File Becomes a Sequence

The Trap: A Glob Sees Only the Active Files

The Honest Caveat: Ground Truth Is Not Every Row

Code

External links

Exercise

Progress

Comments 0