Derived vs Authoritative

"There should only ever be one answer to 'which copy is real?' The moment there are two, you've built a bug."

The Question Every System Must Answer

Point at any piece of data in a system and ask: is this the truth, or a copy of the truth? A store that holds the truth is authoritative. A store that holds a copy — one you could throw away and regenerate — is derived. Every well-built system knows, for every byte it holds, which of the two it is. Every fragile system has places where nobody can say.

Moving Files Destroys the Answer

Now watch what the gather-by-moving trap does to this. Before the move, the original is authoritative — full stop. After you copy it into a gather-folder, there are two files with the same content and equal claim to being real. Edit one; the other is now wrong, but nothing marks it as the copy. You've manufactured ambiguity. Six months later you find two versions of the same essay and you genuinely cannot tell which one you meant to keep. The move didn't gather your writing; it dissolved its authority.

Declaring Keeps the Line Bright

The registry approach never crosses the line. The originals stay authoritative because they're never copied — only pointed at. Everything Lantern builds from them (the chunk table, the FTS index, the vectors) is unambiguously derived: none of it is truth, all of it is rebuildable, and if any of it disagrees with an original, the original wins by definition. There is never a moment where two things both claim to be the real copy.

Never patch derived state — rebuild it. When a derived store (an index, a cache, a denormalized table) drifts out of sync with its source, the fix is not to reconcile it byte by byte. The fix is to purge it and regenerate from the authoritative source. Patching derived state is where subtle, permanent corruption is born.

This Is Bigger Than Lantern

Caches, search indexes, materialized views, denormalized tables, thumbnails, compiled assets — all derived. The discipline is identical everywhere: mark them as derived, never let them become a second source of truth, and recover them by rebuilding, not repairing. Lantern's whole "index is disposable, originals are sacred" posture is just this one idea, held without exceptions.

Code

Recover by rebuilding from the source of truth·python

# Recovery model: purge-and-rebuild, never patch-and-reconcile.

def heal_corpus(corpus_id: str) -> None:
    # WRONG (the temptation): crawl the index, diff against the files,
    # and surgically patch the rows that look stale. This is where
    # corruption hides — one missed edge case and the index lies forever.
    #
    # RIGHT: the index is derived, so throw it away and rebuild it.
    delete_all_chunks(corpus_id)          # drop the derived state
    for doc in walk_declared_roots(corpus_id):
        chunks = chunk(read(doc))          # re-derive from the authoritative file
        insert_chunks(corpus_id, doc, chunks)
    # The originals were never touched. The index is now provably correct.

Exercise

Find a place in your own setup where you keep two copies of the same thing (synced notes, a backup folder, an exported version). For each, decide right now: which one is authoritative and which is derived? If you hesitate on any pair — if both feel equally 'real' — you've found a latent bug. Write down how you'd rebuild the derived one from the authoritative one if it vanished.

Hint

The test: if you deleted one copy, could you regenerate it exactly from the other? If yes, that one is derived and safe to lose. If you can't tell which one to delete, that's the ambiguity Lantern is designed to never create.

The Question Every System Must Answer

Moving Files Destroys the Answer

Declaring Keeps the Line Bright

This Is Bigger Than Lantern

Code

External links

Exercise

Progress

Comments 0