Skip to content
C.W.K.
Stream
Lesson 04 of 04 · published

Ground Truth and Its Mirror

~12 min · storage, architecture, records, design

Level 0Wet Clay
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

You Want Two Shapes and You Can Only Trust One

An append-only file per subject is a beautiful record: the single writer serializes every append, so no half-written record of somebody else's can corrupt it; it merges without conflict because two subjects never touch the same file; it is readable by anything, and its history is its content. It is also useless for the questions you actually ask — how many passes of this kind ran last month, which units have no verification, what did this system produce across the corpus.

A relational store answers all of those instantly and is a much worse record. So you keep both, and the entire design rests on one decision made in advance: the file is the record, the store is a derived mirror.

Why Declaring It Beforehand Is the Whole Trick

Two stores holding the same facts will disagree eventually. A write that succeeded in one and failed in the other, a schema migration that dropped a column, a rebuild that ran against a stale copy. The disagreement is not the emergency — the emergency is discovering that nobody knows which one to believe, in the middle of the incident, with a decision needed now.

Declaring the authority in advance converts that emergency into an operation. Rebuild the mirror from the record. Ten minutes, no judgment calls, no data loss possible, because the mirror never held anything the record does not.

The rule that follows: nothing is ever written to the mirror that is not derivable from the record. The first time somebody adds a convenient column that exists only in the store, the mirror has become a second record and the rebuild is now destructive.

The Layout Choices That Make This Cheap

One file per subject rather than one big file, so a corrupted file damages one subject's history and no more. Note what that does not buy: the unit of work is the delegation, and several delegations against one subject share a file, so parallel work still lands in the same place and the append has to be safe on its own. Newline-delimited records, so appending is a single write and reading is streamable. And a rebuild that is idempotent, so it can be run at any time by anyone who is unsure — a repair operation people are afraid of is one that does not get run.

All three buy the same property: when something goes wrong, the blast radius is decided in advance. A corrupted file costs one unit, a disagreeing mirror costs one rebuild, a half-finished write costs the last line. The storage design worth choosing is usually not the fastest one but the one whose worst case is smallest.

When you keep the same facts in two places, write down which one is authoritative before you need to know. The decision is nearly free today and nearly impossible during an incident, because by then both stores have plausible-looking data and somebody is waiting.

Code

The record, the mirror, and the operation that reconciles them·bash
# GROUND TRUTH - one newline-delimited file per SUBJECT.
# Appending is one write, and a corrupted file costs one
# subject's history rather than everything. Note what it does
# NOT buy: the unit of work is the delegation, and several
# delegations against one subject share a file - so the append
# still has to be safe on its own.

  <data-dir>/logs/<subject>.jsonl
    {"ts":"...","event":"claimed","by":"...","pipeline":"create"}
    {"ts":"...","event":"stage_done","stage":"sweep","note":"..."}
    {"ts":"...","event":"review_done","verdict":"blocked", ...}
    {"ts":"...","event":"verified","passed":true, ...}
    {"ts":"...","event":"landed","commit":"...", ...}

# MIRROR - a relational store, rebuilt from the files. Answers
# the questions the files cannot: counts, joins, "which units
# have no verification", "what did this system produce".

# THE OPERATION - available to anyone, any time, idempotent.
  $ <tool> admin rebuild-log-mirror
  read 1,284 records from 82 files -> mirror rebuilt

# Because the mirror holds NOTHING that is not derivable from
# the files, this is never destructive and never needs a
# judgment call. The moment somebody adds a column that exists
# only in the mirror, that stops being true - and the rebuild
# quietly becomes a data-loss operation.

External links

Exercise

Find two places in your systems that hold the same facts — a cache and its source, a search index and a database, a reporting table and the transactions behind it. For each pair, write down which one is authoritative and how you would rebuild the other. If either answer takes more than a sentence, you have found the pair that will produce your next confusing incident.
Hint
The tell is a field that exists in only one of the two. It got added because it was convenient there, and its presence means the derived store is no longer purely derived — so the rebuild you were relying on has quietly become a data-loss operation that nobody has tested.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.