Snapshots Are Forever

"Everything else in the engine, I can rebuild from scratch. The snapshots I cannot. So the snapshots get a different set of rules."

The One Store That Isn't Disposable

By now the refrain is familiar: the index is derived, throw it away, rebuild it. But there's one store where that reflex would be catastrophic — the snapshot store, where captured corpora keep their converted text. If a source PDF is deleted and you've purged its snapshot too, that content is gone from the universe. So snapshots live under the opposite discipline from the index: they are preserved, not rebuilt.

Append-Only and Content-Addressed

A snapshot is stored by the hash of its content. The converted text of a document hashes to some sha256, and that becomes both its id and its address on disk. Two consequences fall out for free: identical content is stored once (same hash, same file), and a snapshot is never mutated. When a captured source changes, the engine doesn't overwrite the old snapshot — it appends a new one under a new hash. The superseded version stays perfectly readable. History accretes; it never gets edited in place.

Deleted Only by Arming the Charge

Because snapshots are precious, deletion can't be a side effect. Removing a corpus does not delete its snapshots. Re-indexing does not delete snapshots. The only way a snapshot dies is an explicit, armed action — a delete call that names the exact hash twice, once as a confirmation — and even that is refused with a conflict error while any live document still derives from it. You cannot delete a snapshot by accident, in bulk, or as a consequence of some other operation. You have to walk up and pull the pin yourself.

Give your precious store different rules from your disposable store. The failure of most systems is treating all state the same — one delete path, one recovery story. Separate them: derived state is cheap, purgeable, rebuilt on a whim; captured ground truth is append-only, content-addressed, and armed to delete. The two must not share a code path, or the cheap one's carelessness will eventually reach the precious one.

Converters Are Versioned Like Everything Else

One last immutability: the converter that produced a snapshot is versioned. Change a strip rule or a filter threshold and you mint a new converter id — never mutate the old one's behavior. Why? Because every existing snapshot was produced by a specific converter, and it should stay explicable by that converter forever. An old snapshot plus its recorded converter version is a complete, reproducible story of how that text came to be.

Code

Append-only snapshots, armed single deletion path·python

from pathlib import Path
import hashlib

class SnapshotStore:
    root = Path("~/lantern/snapshots").expanduser()

    def append(self, text: str) -> str:
        sha = hashlib.sha256(text.encode("utf-8")).hexdigest()
        path = self.root / sha[:2] / f"{sha}.md"      # addressed by content
        if not path.exists():                          # identical text: store once
            path.parent.mkdir(parents=True, exist_ok=True)
            path.write_text(text)                       # write-once, never mutate
        return sha

    def delete(self, sha: str, confirm: str) -> None:
        if confirm != sha:
            raise ValueError("arm the delete: pass the sha as confirmation")
        if live_documents_deriving_from(sha):
            raise Conflict("snapshot still cited by a live document")  # 409
        (self.root / sha[:2] / f"{sha}.md").unlink()    # the ONLY deletion path

Exercise

Look at how your own tools delete things. Find one place where deleting or changing one thing silently deletes or overwrites another (uninstalling an app that takes its data, a sync that mirrors a deletion). Now imagine that cascade reaching data you could never recreate. Design the rule that would have stopped it: what would 'you must arm this deletion explicitly' look like for that tool?

Hint

The pattern is: separate the precious store, make its only delete path require naming the exact thing twice, and refuse the delete while anything still depends on it. If deletion is ever a side effect, the accident is already scheduled.

The One Store That Isn't Disposable

Append-Only and Content-Addressed

Deleted Only by Arming the Charge

Converters Are Versioned Like Everything Else

Code

External links

Exercise

Progress

Comments 0