The Registry Is the Place

"The gathering isn't a folder. It's a list — and the list is the whole trick."

The Manifest Is the Map

The one place all your scattered writing becomes findable is not a directory — it's a small, git-tracked manifest. Each entry declares a corpus: an id, the roots where its files already live, the globs that decide which files count, and the chunking profile to use. That's it. The manifest holds addresses, never content. You can read the entire "gathered" corpus of a decade in a file you could print on one page.

The Walk

Registration points; ingestion walks. Given a corpus's declared roots and globs, the engine discovers every matching file, reads it, hashes its bytes into a doc_sha256, chunks the text, and writes those chunks to the index. The document's own id is content-addressed from its corpus and path, so the same file always lands in the same slot. Nothing is copied out of place; the walk is a reader, not a mover.

Idempotent Re-Ingest — the Cheapness That Makes It Livable

Re-ingesting is safe to run constantly, because it's idempotent. The engine hashes each file and compares it to what it indexed last time. Unchanged hash? Skip it — there is literally nothing to do. Changed hash? Replace that document's entire chunk set in one atomic transaction: delete all its old chunks, insert all its new ones, never a half-patched in-between. This is why "keep the corpus fresh" costs almost nothing: only what actually changed does any work.

The smallest unit of index mutation is a whole document. Never patch a document's chunks piecemeal. If one byte of the file changed, blow away all its chunks and rebuild them together. Atomic whole-document replacement means the index is never caught in a partially-updated state — a query either sees the old document or the new one, never a Frankenstein of both.

Availability Is Just a Field

Different machines hold different files — a laptop might have a subset, one workstation has everything. That reality doesn't need special plumbing; it's a field on the manifest entry. The registrar simply skips corpora whose declared roots don't exist on the current machine and indexes the ones that do. The machine with everything becomes the canonical index; the others index what they can. The map knows what the territory looks like from where you're standing.

Code

Hash, compare, skip-or-rebuild — the idempotent walk·python

import hashlib
from pathlib import Path

def reindex(corpus) -> dict:
    stats = {"unchanged": 0, "reindexed": 0}
    for path in discover(corpus.roots, corpus.include_globs, corpus.exclude_globs):
        raw = Path(path).read_bytes()
        doc_sha = hashlib.sha256(raw).hexdigest()

        if doc_sha == last_indexed_sha(corpus.id, path):
            stats["unchanged"] += 1          # same bytes — nothing to do
            continue

        # Changed (or new): replace the WHOLE chunk set atomically.
        with transaction():
            delete_chunks(corpus.id, path)
            for seq, ch in enumerate(chunk(raw.decode("utf-8"), corpus.profile)):
                insert_chunk(corpus.id, path, seq, ch)
        stats["reindexed"] += 1
    return stats  # cheap to run on every save — only real changes cost work

Exercise

Write a manifest, on paper, for your own writing. For each place it lives, give it an id, a root path, and a glob for which files count (e.g. only *.md, exclude drafts). Notice how small the whole thing is. Then ask: if you re-ran an indexer over this every time you saved a file, what would make that cheap enough to do constantly?

Hint

The answer is the content hash: if 'unchanged' can be detected without re-reading and re-processing the file, then re-indexing a thousand documents where two changed costs the work of two, not a thousand.

The Manifest Is the Map

The Walk

Idempotent Re-Ingest — the Cheapness That Makes It Livable

Availability Is Just a Field

Code

External links

Exercise

Progress

Comments 0