A promise nobody had rehearsed
This track leans on one sentence from lesson 2: SQLite and ChromaDB are mirrors, and the mirrors can be rebuilt from the JSONL. Lesson 3 leans on it again: a failed mirror write is survivable, because purge-and-replay rebuilds the mirror later. For months nobody ran that rebuild. The drill that finally did, scripts/restore-drill.py (2026-07-25), opens with its reason to exist:
A doctrine nobody exercises is a hypothesis, and the only bad time to discover it was wrong is during an actual recovery.
Its fidelity pass derives from every conversation's JSONL what SQLite should hold and diffs that against what SQLite does hold. Its replay pass rebuilds a few conversations into a scratch database under /tmp through the real ConversationStore API, then deletes it. Read-only against every real store, it runs quarterly by cron and by hand whenever anyone wants.
The first answer, and a count that kept rising
The first run found SQLite rows with no JSONL line behind them: 22 healing placeholders, 323 duplicates from a cron fallback chain that re-posted one message to three brains (fixed at the source that day), imported conversations whose lines carry no message ids by contract, and branch and council children copied rather than streamed. A rebuild from the JSONL alone drops those rows, so a rebuild has to read both stores.
Then the count kept rising: 277, 396 and 505 rows on the full drills of 2026-08-15, 08-28 and 10-01, and on 10-01 another 33 rows whose text differed between the stores. The drill printed one line over all of it, a purge-and-replay would lose these, and nobody could say what the rows were. An alarm nobody can act on is worth as much as silence. The drill's code now says so where it counts findings:
Known shapes are reported, not counted: a known shape that reads as data at risk is an alarm nobody acts on.
Every row, classified by the writer that left it
On 2026-10-18 (todo #478) every row was traced to the code that wrote it. Over 1,276 conversations, 749 rows had no line: 523 in conversations with a log, 226 in the 13 with none.
| Class | Rows | What wrote it | Another record? |
|---|---|---|---|
fork_copy | 584 | Branch and Soul Archive copies keep the source's timestamp; a fork wrote no log until its first live turn. | Yes, the source. |
cut_turn | 80 | 71 healing rows, and 9 route backstops that saved a turn which ended without done. | The deltas rebuild the text; the id is lost. |
fork_copy_source_gone | 68 | Copies whose source was deleted, or was a private conversation that ended. | No. |
never_logged | 10 | Three prompt-refine seeds; seven cron prompts whose turn never started. | No. |
council_stub | 7 | The retired [Council Context …] pick stub, last written 2026-05-17. | The council round's record. |
typo_edit (drift) | 33 | Dad's "original" target of a transcript typo fix: SQLite rewritten, the JSONL never. | Yes, the canonical overlay. |
Two cron rows had a cause on record: the Soul Stream wakes of 2026-10-06 and 10-15, where the route saved the prompt and answered 500 before the turn wrote its first line. Only fork_copy_source_gone, never_logged and unexplained have no second record, so only they are findings and set the exit code; every other class is reported by name. The full drill after the change read sqlite_only=78 (68 + 10), known_rows=671, known_drift=33, nothing unexplained.
One marker, never a turn line
Classification makes the alarm readable; it does not stop new rows. So every writer that puts a row into SQLite outside a streamed turn now leaves one line in that conversation's log carrying the row itself: pippa_message_row (store/session_logger.py). writer says why the row has no turn of its own (heal, fork, seed, unstarted_turn, backfill); fields carries its other columns when the writer has them.
A user or assistant line, the shape every reader already knows, would have broken healing. A user line opens a turn window, and lesson 4's healing answers an open window with an interruption row. An assistant line closes whatever window it lands in, hiding a cut turn from healing; on 2026-10-17 a Master Chief report did land inside a cut turn's window, and the drill learned to count only an assistant line whose parent is that turn's own user message. So the marker is a type no turn reader looks at. The drill derives it like a row, a later marker for the same id replaces the earlier one (a heal in place), and the replay maps ids over the whole file because a marker can follow the lines that already name its row.
The marker is not structural: record_message_rows appends a batch through one logger, so no row pays an fsync of its own. The close fsyncs once, and lesson 3's ~250ms check adds one for each quarter second the appends take, so a fork of a few hundred rows costs a few fsyncs, not hundreds. It is built not to fail the request that wrote the row, because the row is already in SQLite: a logger that will not open and an append that raises are caught and logged, and the drill names what such a failure leaves behind. One failure still gets out. The close runs in a finally with no guard of its own, so an fsync error there reaches the caller.
The gap no single writer could close was a user row saved before its turn began: every chat route saves Dad's message, prepares the turn, and only then writes the user line. All seven chat routes are now @unstarted_turn.guarded. The route marks the row saved, then logged once the line is written; if the route raises or the stream ends in between, the guard records the row with writer unstarted_turn. The Claude, Gemini and Ollama backstops now write a cut turn's assistant line, as the other four routes always had. A backfill script writes the marker for the 749 older rows; it is a dry run by default, and applying it waits for Dad.
Tables that were never mirrors
The re-measure named something this track never said: not everything in the conversation database is a mirror. transcript_message_edits, with the emotion and replacement-rule tables beside it, is primary state. Dad's canonical typo fixes live there and nowhere else, so purging conversations.db whole and replaying it from the JSONL would take every one. The security_* tables had set that precedent quietly, and the Family Brain Control Plane's brain_control_* tables made it load-bearing: every family app's brain assignments are an original record. Both now live in the family authority store, and the drill checks on every run that they exist, because a "rebuilt" store without them has lost the PIN posture and the brain allocation however clean its conversation rows look. Purge-and-replay stays correct for conversation rows and is never applied to a primary-state file whole.
Drift on purpose
Some differences are decisions, and the drill must know each by name or it cries wolf. typo_edit is accepted only when the message's active canonical overlay holds exactly the SQLite text; the JSONL stays the forensic twin of what was said. quote_escape arrived on 2026-10-26. The Claude replay had carried prior turns as a JSON array, every quotation mark reached the model as \", and Opus 5.5 wrote it back into its prose: 798 of 6,307 stored replies and summaries by that day (the vessels track tells the replay half). A guard under the write now makes a stray \" outside code plain where a reply or summary is stored, while the JSONL keeps exactly what the model wrote. The drill accepts the difference when the logged text, unescaped, equals the stored one.
That class taught one more thing within hours. The first unescape also turned \\" into \", on every pass, so 19 compaction summaries quoting a shell command never reached a fixpoint. It now replaces only a lone \" and is idempotent, so the replay cleaner, the guard, the repair and the drill agree on every pass. A known-drift class is only as stable as the function that defines it.