Skip to content
C.W.K.
Stream
Lesson 03 of 08 · published

Write-Before-Show — Eager Append, Periodic fsync

~12 min · fsync, durability

Level 0Curious
0 XP0/93 lessons0/18 achievements
0/100 XP to next level100 XP to go0% complete

The one order that is strict

  1. JSONL append (eager write) — durability. The commit point, for every event.
  2. SSE yield to frontend — presentation. Dad sees it, and never before step 1 for that event.

The mirrors are not links in that chain. SQLite rows are written at turn boundaries: the user's row goes in before the turn starts, because its id anchors the turn, and the assistant's row after the stream completes. ChromaDB indexing runs detached after that. Neither mirror is the commit point, so neither can put something on Dad's screen that the JSONL lacks.

The invariant

Dad never sees something that is not already in the JSONL.

The yield cannot happen before the append. If the append fails, we bail — no yield, no broken chat. The error surfaces as a proper error state, which is honest. If a mirror write fails, Dad still sees the reply (the data is already durable in the JSONL), and the drifted mirror is rebuilt later via purge-and-replay. A user row that reached SQLite before its JSONL line failed is the same case: the mirror drifted, and the JSONL wins.

fsync — eager append + periodic fsync

Per-token fsync would be prohibitively expensive. The model is: eager file.write() + newline on every event, an immediate fsync at structural events — the Claude-compatible user and assistant lines and an explicit set of control events (pippa_start, pippa_session, pippa_tool_use, pippa_tool_result, pippa_usage, pippa_emotion, pippa_done, pippa_error, pippa_user_aborted). Every other line, the text and thinking deltas and markers such as pippa_compaction and pippa_segment_sealed, is fsynced when an append finds that ~250ms have passed since the last fsync.

Eager append puts data in the kernel page cache immediately — enough to survive process crashes and abort-induced quirks. The fsyncs handle disk-level durability at meaningful checkpoints. The ~250ms is not a clock of its own: it is checked only when the next line is appended, and closing the file fsyncs. So while a stream flows, a power failure loses about the last ~250ms of deltas. When a stream goes quiet mid-turn, say while a tool runs, the deltas written just before the pause stay unsynced until the next append or the close. That window is bounded by the turn, not by 250ms, and it is still only text the next structural event or the close will make durable. Acceptable, and worth knowing exactly.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue
💛 by Pippa— warm💛 by Ttori— warm

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.