The Index Is Derived

"If it can be rebuilt, never repair it. If it can't be rebuilt, it isn't derived — and you should know that."

Two Kinds of State, and You Must Know Which Is Which

Every row in Recall is one of two things, and the whole maintenance story depends on being able to say which:

Durable state — the things that would be genuinely lost if deleted: video records, immutable runs, releases, corrections, job history. This is the truth. It's authored once, versioned, and never regenerable from anything else.
Derived state — projections computed from the durable truth: the full-text index, quality metrics, (someday) a vector index. Delete any of it and nothing is lost, because it can be recomputed.

The classification isn't bookkeeping trivia. It determines the repair procedure, and getting it wrong is how systems rot.

Derived State Has Exactly One Repair: Rebuild

Say the search index drifts out of sync with the releases — a crash mid-projection, a bug in an old version, whatever. The instinct of every experienced engineer is to reach for a reconciliation script: diff the index against the releases, find the differences, patch them. It feels surgical and efficient. It's the wrong move, and Recall forbids it outright.

Here's why. That patching script is a second implementation of the projection — a new piece of code that decides what the index should contain, living alongside the real projection code that already decides that. Two implementations of one truth, and they will diverge, because nobody updates the reconciler when the projection changes. Now you have a repair tool that quietly writes slightly-wrong rows and calls it fixed. Patching is where bugs live.

The correct move is simpler and boring: purge the affected derived rows and replay them from the durable source. The projection code already exists and is the only thing that knows what the index should be. Run it again. No diffing, no second implementation, no divergence. This is exactly what a materialized view is: a cache you refresh, not a table you hand-edit. And it's the same discipline the sibling brain follows — when its derived mirrors drift, it purges and replays from its own append-only truth, never reconciles.

The Test That Keeps You Honest

So here's the question to run on every piece of state you own: if I deleted this entirely, could I rebuild it from something durable?

Yes → it's derived. Never patch it. Its only repair is purge-and-replay, and you should be able to run that on a whim.
No → it's durable, whether you meant it to be or not. If you've been treating it as a disposable cache, you have undeclared truth sitting in something you'd delete without a second thought.

That second case is the dangerous one, and it's shockingly common: a computed field nobody can recompute, a cache holding the only copy of something, an index containing rows the source no longer has. Find those and force the choice — either make the thing genuinely durable and versioned, or make it genuinely rebuildable. What you cannot do is leave it in between, because in-between state is truth you'll lose the first time someone treats it as a cache.

Code

Purge and replay. Never diff-and-patch.·python

# DURABLE (the truth — never regenerable from anything else)
#   videos, transcript_runs, transcript_releases, corrections, jobs

# DERIVED (projections — always rebuildable)
#   transcript_segments, transcript_segment_fts, quality_assessments

# WRONG: a reconciler is a SECOND implementation of the projection.
def reconcile_index():
    for release in all_releases():
        if index_differs(release):
            patch_index_rows(release)   # diverges from the real
                                        # projection the day it changes

# RIGHT: the projection code is the only thing that knows the answer.
def repair(release):
    purge_derived_rows(release)         # throw the projection away
    project_segments_and_fts(release)   # run the REAL code again

# Test every row you own:
#   deletable + rebuildable -> derived. purge and replay.
#   not rebuildable         -> durable. you just found undeclared truth.

Exercise

List the major pieces of state in a system you own and label each durable or derived. For each 'derived' one, ask whether you could actually purge and rebuild it right now — is the rebuild path written, tested, and runnable on demand? For each thing you labeled a cache, run the deletion test: if it vanished, would anything be unrecoverable? Anything that fails that test is undeclared truth; decide right now whether to make it durable or make it rebuildable.

Hint

Look for reconciliation scripts first — each one is a confession that a projection can't simply be rebuilt, plus a second implementation of that projection waiting to diverge. Then look for caches with no rebuild path; those are the ones that quietly hold the only copy. The healthy end state: every derived thing has a one-command rebuild, and everything else is explicitly durable.