Rebuild Without Breaking Citations

"Throw the whole index in the fire. Rebuild it from scratch. Every citation you ever made still lands. That's not magic — it's arithmetic."

The Apparent Contradiction

Two things you've now been told seem to fight each other. First: the index is derived and disposable — if it's ever suspect, purge it and rebuild, never patch it. Second: citations must be durable — a saved citation has to keep resolving forever. But a citation points into the index. How can you throw away the index every time it misbehaves and still have citations that never break? Those sound mutually exclusive.

Content-Addressing Dissolves the Contradiction

They only conflict if ids are assigned. Because Lantern's chunk ids are computed from content, a rebuild is not a re-labeling — it's a re-derivation that arrives at exactly the same answer. Purge the entire index, walk the same sources under the same frozen profiles, and every chunk gets recomputed to the byte-identical id it had before. The citation that pointed at 9f3ce1a7… still points at 9f3ce1a7…, and that id still resolves to the same slice of the same document. Disposable index and durable citations are both true because of content-addressing, not despite it.

Recovery Is Purge-and-Replay, Always

This is why Lantern's recovery story has no repair step. If the search index is corrupt, out of sync, or just suspect, you don't crawl it looking for rows to fix. You delete it and rebuild from the authoritative sources. There is no diff-and-patch logic to get subtly wrong, no reconciliation pass that might miss a case. Purge, replay, done — and because ids are content-addressed, the rebuilt index is not just correct, it's identical to what a healthy build would have produced, citations and all.

Deterministic derivation makes disposability free. When a derived store is rebuilt by a pure, deterministic function of its sources, you can throw it away without fear: the rebuild reproduces exactly what was there. Combine that with content-addressed ids and even the references INTO the derived store survive its destruction. Determinism is what lets 'just rebuild it' be a complete recovery plan.

Two Kinds of State, Cleanly Named

So the full picture has exactly two categories. Derived state — the FTS index, the vectors, the chunk table — is disposable, rebuilt by deterministic replay, and safe to purge on any suspicion. Precious state — the captured snapshots, and later the evidence layer's judgments — is append-only and preserved, because nothing can regenerate it. Every store in the engine is consciously one or the other. Knowing which is which, for every byte you hold, is the difference between a system you can confidently recover and one you're afraid to touch.

Code

Purge, replay, and the old citation still resolves·python

# A citation saved long ago, on a machine since retired:
saved = {"chunk_id": "9f3ce1a7b2c4d8e0", "doc_relpath": "2026/investing/patience.md"}

# Disaster: the index is corrupt. The recovery is not repair — it's replay.
purge_entire_index()                      # throw ALL derived state in the fire
for doc in walk_declared_roots():         # same sources...
    for start, end in chunk_spans(doc, profile="md-para-v1"):   # ...same frozen profile
        cid = chunk_id(doc.sha256, start, end, "md-para-v1")     # ...same computed id
        insert(cid, doc, start, end)

fetch(saved["chunk_id"])   # -> resolves. The rebuild recomputed the SAME id.
# The index was disposable AND the citation survived. Both, because of the math.

Exercise

Take a derived store you maintain (a search index, a cache, a generated report). Right now, do you recover it by patching, or by rebuilding from source? If patching, imagine the subtle bug hiding in your reconcile logic. Then sketch the rebuild-from-source path — and check whether any saved references into that store would survive the rebuild. If they wouldn't, that's the content-addressing gap.

Hint

Two properties make 'just rebuild it' a complete plan: the rebuild is deterministic (same sources always yield the same output), and any references into the store are content-addressed (so they resolve to the same thing after the rebuild). Missing either one is why people are scared to purge.

Rebuild Without Breaking Citations

The Apparent Contradiction

Content-Addressing Dissolves the Contradiction

Recovery Is Purge-and-Replay, Always

Two Kinds of State, Cleanly Named

Code

External links

Exercise

Progress

Comments 0