Every Index Is a Derived Mirror

"If the index disagrees with the files, the index is wrong. Purge it and replay — never patch it into agreement."

EV v2 · 2026-09-18: One documented refinement since this lesson: v4 keeps app-only attributes (pin/lock/workflow state) in sparse sidecar JSON under the app's own data root (meta/notes, meta/tags), not in frontmatter — portable semantic metadata still rides the file, while app-only state stays out of the portable bytes entirely. The rule survives by being split: frontmatter for what the document means, sidecars for what the app remembers, indexes still purge-and-replay.

Indexes Are Derived, Never Masters

Rekindle grows real databases — a note index, full-text search (the corpus side is Lantern's). None of them are the truth. Every one is a derived mirror of the Markdown files, and its entire contents could be thrown away and rebuilt from disk without losing anything. That status is decided up front, not discovered later: an index is a fast cache of what the files already say.

Purge and Replay, Never Diff and Patch

The discipline that keeps it honest is the recovery rule. If the index and the files ever disagree — an external edit, a crash mid-write, a bug — the answer is never "figure out the delta and patch the index into agreement." It's purge the derived rows and replay from the files. Diff-and-patch reconciliation is exactly where corruption lives: every patch is a small guess about what happened, and guesses accumulate. A rebuild has no guesses in it.

// Recovery is always this shape — never a reconcile.
fn rebuild_index(vault: &Path) -> Result<(), Error> {
    purge_index()?;                       // throw away ALL derived rows
    for file in markdown_files(vault) {   // replay from ground truth
        index_note(&read_to_string(&file)?)?;
    }
    Ok(())
}

This is cwkPippa's discipline, carried one domain over. There, the JSONL log is ground truth and SQLite plus the vector store are derived mirrors, rebuilt by purge-and-replay. Here, the Markdown files are ground truth and every index is the mirror. Same rule, same reason: one authority, rebuildable derivatives, no reconciliation logic anywhere.

Meaning Lives in Frontmatter

The rule extends to metadata, with one line drawn carefully. What the document means — its title, its tags, its date — lives in the file's frontmatter, not in a database. If tags lived only in a DB, that DB would become a second master (lose it and you've lost real data), and the whole file-canonical claim would be a half-truth. What the app remembers about a note — a pin, a lock, a workflow lane — is a different kind of thing, and v4 keeps it in small sparse sidecar files beside the library, on purpose (the note above). Lose a sidecar and every document is still whole; only the app's arrangement of it is gone. The index, either way, stays what it should be: a fast, disposable view of what the files already contain.

Code

Recovery is purge-and-replay, never diff-and-patch·rust

// If the index disagrees with disk, the index is wrong. Rebuild it.
fn rebuild_index(vault: &Path) -> Result<(), Error> {
    purge_index()?;                        // throw away ALL derived rows
    for file in markdown_files(vault) {    // replay from ground truth
        index_note(&std::fs::read_to_string(&file)?)?;
    }
    Ok(())
}

// There is deliberately NO reconcile_index(): no diff, no patch,
// no "figure out what changed and fix it up." Every patch is a guess,
// and guesses accumulate into corruption. A rebuild has no guesses.

Meaning rides the file; the index is disposable·markdown

---
title: The Rekindling
tags: [editor, cm6, architecture]
date: 2026-07-16
---

The file carries what it means. Lose the index and you lose
nothing; rebuild it from these files. Lose a database that owned
the tags, and you've lost real data — that's a second master.
(App-only state such as a pin or a workflow lane lives in a sparse
sidecar instead: losing it loses arrangement, never the document.)

Exercise

For a system you know that has a cache or index, ask the rebuild question: could you delete it entirely and regenerate it from the authoritative source with no data loss? If not, list exactly what would be lost — that list is the data your 'cache' secretly owns, which makes it a second master. Then decide where that data should actually live.

Hint

The usual culprits are user-set fields hiding in the cache: tags, ordering, 'last opened,' custom labels. If they only exist in the cache, they aren't cache. Sort each one: what the document means goes into the authoritative artifact (here, frontmatter); what the app merely remembers goes into its own small, clearly labelled app-state file. Either way the index becomes genuinely disposable.

Indexes Are Derived, Never Masters

Purge and Replay, Never Diff and Patch

Meaning Lives in Frontmatter

Code

External links

Exercise

Progress

Comments 0