Append-Only, and Yet a Correction Deletes

"Append-only means a correction REMOVES what it invalidates; it never leaves it beside the replacement."

The apparent contradiction

The storage invariant says snapshots are append-only: metric history rows carry their series, date, value, source and fetch time, and are never rewritten. Then the same invariant says a correction deletes. Read quickly, those two sentences fight.

They do not, and resolving the tension is the most useful thing in this lesson, because plenty of teams get one half and lose the other.

What append-only is actually protecting

Append-only exists to stop silent in-place mutation of history. The failure it prevents is a value that used to be one thing and is now another, with nothing anywhere recording that it changed. That is the thing that makes a historical series untrustworthy: not that it was ever wrong, but that you can no longer tell whether what you are looking at is what was recorded.

So the rule is: you do not edit a row's value in place. New readings arrive as new rows. History accumulates rather than being revised.

What it is not

Append-only is not a rule that says every row ever written must remain forever, regardless of whether it is true. That reading turns the invariant into a machine for preserving known falsehoods — and in a store whose read model takes the newest row per series, a preserved falsehood is not an inert historical artifact. It is the answer the product gives.

Which is precisely what happened. The forward-dated rows were not "history we should keep for the record"; they described sessions that never occurred, and they sat at the front of the read model. Keeping them would have meant choosing an abstract purity over the product being correct.

Distinguish a wrong reading from a superseded one. A reading that was correct and has been revised by the source is history — keep both, and let the dates order them. A row that describes an event that did not occur is not history, it is an error, and an error left in place is served, not archived. Append-only protects the first case; it has nothing to say about the second.

What the repair looks like in practice

The repair for this incident was a named, dated script — not an interactive session with the database. That distinction matters more than it looks.

A script has three properties an ad-hoc fix does not: it states in code exactly which rows are being removed and on what criterion, it can be read by someone else before it runs, and it stays in the repository afterward as the record of what was done. When a later reader asks "why does this series have a gap in early August", the answer is a file with a name and a date, sitting in version control.

Make the deletion criterion narrower than you think you need. Delete by the specific defect — these gauges, this market, these exact dates, this source string — never by a broad range that happens to contain them. A too-wide delete during a data repair is one of the few mistakes in this domain that is genuinely unrecoverable, and it is made under exactly the conditions that encourage haste.

Code

Three states, three different correct actions·text

CASE 1 -- a NEW reading for the same series
  action:  append a new row with its own data_date
  history: both rows remain; dates order them
  why:     the series moved. That is what a series does.

CASE 2 -- the SOURCE revised a past value
  ideal:   keep both -- "what we knew then" is itself a fact,
           and quarterly macro data is revised routinely
  ACTUAL:  THIS store cannot. Its primary key is
           (gauge, market, series, data_date) and the write is
           INSERT OR REPLACE, so a revision for the same series
           and date overwrites the original IN PLACE -- the one
           thing append-only was supposed to prevent.
  why it is tolerable here: the gauges are published series whose
           vendors are themselves the record of revisions.
  what it would cost to fix: `fetched_at` or `source` in the key,
           and then every read model needs a rule for which of
           several rows for one date it means.
  -> Know which of these you have. "Append-only" describes a
     WRITE discipline; whether history is actually retained is a
     property of the KEY, and the two are easy to conflate.

CASE 3 -- the row describes something that NEVER HAPPENED
  action:  DELETE, by a narrow criterion, in a named script
  history: nothing is lost, because nothing true was there
  why:     it is not history, it is an error -- and in a
           newest-row read model, an error is the ANSWER

# The mistake is treating case 3 as case 2 out of respect for
# append-only. Purity that serves a false number is not integrity.

Exercise

Write down your own three cases for a time series you own: new reading, upstream revision, and impossible row. Decide the action for each before you need it. Then check what your system currently does with case three — most systems have no answer, which means in practice they keep the impossible row and serve it.

Hint

Case two is the one people forget to design and then discover under pressure: when an upstream revises a past value, do you want the revision only, or the pair? For macro statistics the pair is genuinely valuable, because 'what was published at the time' explains decisions that were made then.

The apparent contradiction

What append-only is actually protecting

What it is not

What the repair looks like in practice

Code

External links

Exercise

Progress

Comments 0