Referenced vs Captured

"Sometimes the file on disk is the writing. Sometimes the writing is trapped inside the file, and you have to let it out first."

Two Kinds of Ground Truth

Not all sources are equal. A markdown essay is its file — the bytes on disk are exactly the writing. But a PDF, a Word document, or a web page buried in HTML tags is different: the writing is inside the file, wrapped in a format you have to convert away before you can index it cleanly. Lantern names this split with two corpus classes: referenced and captured.

Referenced: The File Is the Document

A referenced corpus points at text files whose bytes are the ground truth. Markdown, plain text — read them, chunk them, index them, done. The doc_sha256 is the hash of the file itself. If the file is ever lost, it's lost; Lantern never claimed to preserve it, only to index it. The originals are authoritative and they stay put. This is the simple, common case.

Captured: The Conversion Is the Document

A captured corpus declares an adapter — a converter that turns a source into clean text. A PDF or DOCX goes through a document-to-markdown converter; HTML-polluted content goes through a prose extractor that strips tags. The converted text is what gets chunked and indexed, and — crucially — it's preserved in a snapshot store. Here the doc_sha256 is the hash of the snapshot (so offsets refer to the converted text), while a separate source_sha256 records the original bytes, so the engine can skip re-converting a source that hasn't changed.

Why the split matters for recovery. Lose a referenced file and its index rows die with it — fair, it was never Lantern's to keep. Lose a captured source, though, and the converted text survives in the snapshot store. Capture is preservation; reference is just indexing. Choosing the right class per corpus is choosing what you're promising to keep.

Same Index, Different Truth

Downstream, the two classes look identical: both produce chunks, both are searchable, both return citable results with the same provenance shape. They differ only in what counts as ground truth — the file, or the snapshot of the conversion. That single decision determines whether a vanished source takes its content to the grave or leaves a preserved copy behind.

Code

Captured reindex = convert, snapshot, index (skip unchanged sources)·python

def reindex_captured(corpus) -> dict:
    stats = {"skipped": 0, "converted": 0}
    for src in discover(corpus.roots, corpus.include_globs):
        source_sha = sha256(read_bytes(src))

        if source_sha == last_source_sha(corpus.id, src):
            stats["skipped"] += 1              # unchanged source — no costly convert
            continue

        text = corpus.adapter.convert(src)     # PDF/DOCX/HTML -> clean markdown/prose
        snap_sha = snapshot_store.append(text) # append-only, content-addressed
        # doc_sha256 == snap_sha: offsets refer to the CONVERTED text
        index_document(corpus.id, src, doc_sha=snap_sha,
                       source_sha=source_sha, text=text)
        stats["converted"] += 1
    return stats
# Referenced corpora skip the convert+snapshot steps — the file IS the text.

Exercise

Sort your own sources into two piles. Pile A: text files you'd be fine simply indexing — if one vanished, so be it. Pile B: PDFs, exported docs, web pages — things where the content is trapped in a format and the source might disappear. Pile A is referenced; Pile B is captured. For each Pile B item, note what converter you'd need to turn it into clean text.

Hint

The tell for Pile B is: 'if I lost this file, could I recreate the exact text?' If the answer is no because it's locked in a PDF or a website that might change, it needs capturing, not just referencing.

Two Kinds of Ground Truth

Referenced: The File Is the Document

Captured: The Conversion Is the Document

Same Index, Different Truth

Code

External links

Exercise

Progress

Comments 0