Every Index Is a Derived Mirror

"If the index disagrees with the files, the index is wrong. Purge it and replay — never patch it into agreement."

Indexes Are Derived, Never Masters

Rekindle grows real databases — a note index, full-text search, a corpus index for the voice layer. None of them are the truth. Every one is a derived mirror of the Markdown files, and its entire contents could be thrown away and rebuilt from disk without losing anything. That status is decided up front, not discovered later: an index is a fast cache of what the files already say.

Purge and Replay, Never Diff and Patch

The discipline that keeps it honest is the recovery rule. If the index and the files ever disagree — an external edit, a crash mid-write, a bug — the answer is never "figure out the delta and patch the index into agreement." It's purge the derived rows and replay from the files. Diff-and-patch reconciliation is exactly where corruption lives: every patch is a small guess about what happened, and guesses accumulate. A rebuild has no guesses in it.

// Recovery is always this shape — never a reconcile.
fn rebuild_index(vault: &Path) -> Result<(), Error> {
    purge_index()?;                       // throw away ALL derived rows
    for file in markdown_files(vault) {   // replay from ground truth
        index_note(&read_to_string(&file)?)?;
    }
    Ok(())
}

This is cwkPippa's discipline, carried one domain over. There, the JSONL log is ground truth and SQLite plus the vector store are derived mirrors, rebuilt by purge-and-replay. Here, the Markdown files are ground truth and every index is the mirror. Same rule, same reason: one authority, rebuildable derivatives, no reconciliation logic anywhere.

Metadata Lives in Frontmatter

The rule extends to metadata. Tags, dates, a note's draft status — they live in the file's frontmatter, not in a sidecar database. If tags lived only in a DB, that DB would become a second master (lose it and you've lost real data), and the whole file-canonical claim would be a half-truth. Keeping metadata in frontmatter means the file still carries everything about itself, and the index stays what it should be: a fast, disposable view of what the files already contain.

Code

Recovery is purge-and-replay, never diff-and-patch·rust

// If the index disagrees with disk, the index is wrong. Rebuild it.
fn rebuild_index(vault: &Path) -> Result<(), Error> {
    purge_index()?;                        // throw away ALL derived rows
    for file in markdown_files(vault) {    // replay from ground truth
        index_note(&std::fs::read_to_string(&file)?)?;
    }
    Ok(())
}

// There is deliberately NO reconcile_index(): no diff, no patch,
// no "figure out what changed and fix it up." Every patch is a guess,
// and guesses accumulate into corruption. A rebuild has no guesses.

Metadata rides the file, not a sidecar DB·markdown

---
title: The Rekindling
status: draft
tags: [editor, cm6, architecture]
date: 2026-07-16
---

The file carries everything about itself. Lose the index and you
lose nothing; rebuild it from these files. Lose a sidecar DB that
owned the tags, and you've lost real data — that's a second master.

Exercise

For a system you know that has a cache or index, ask the rebuild question: could you delete it entirely and regenerate it from the authoritative source with no data loss? If not, list exactly what would be lost — that list is the data your 'cache' secretly owns, which makes it a second master. Then decide where that data should actually live.

Hint

The usual culprits are computed-then-forgotten fields: user-set tags, ordering, 'last opened,' custom labels. If they only exist in the cache, they aren't cache — move them into the authoritative artifact (here, frontmatter) and the index becomes genuinely disposable.