Captured Adapters

"When the writing is trapped inside a PDF or wrapped in HTML tags, you can't index it as-is. You have to free it first — and remember how you freed it."

The Hard Half of Ingest

Captured adapters handle the sources where the text isn't sitting in the open. A PDF, a Word doc, a slide deck — the writing is real but locked inside a binary format. A web page — the writing is buried in a thicket of HTML tags. A captured adapter runs a converter: a document-to-markdown step for the binary formats, a prose extractor for the tag-polluted ones. Out comes clean text; that clean text, not the original, becomes the document.

Convert, Snapshot, Index

The captured flow has one extra beat the referenced flow doesn't. Read the source and hash its bytes into a source_sha256. If that hash matches last time, skip — the expensive conversion doesn't run again. Otherwise, convert the source to clean text, snapshot that text into the append-only content-addressed store, then chunk and index the snapshot. The snapshot's hash becomes the document's doc_sha256, so every chunk offset refers to the converted text; the source_sha256 is kept alongside purely to detect when the original changes.

The Snapshot Is the Document

This is the key mental shift from referenced corpora. For a referenced file, the document is the file. For a captured source, the document is the conversion — the snapshot of clean text, not the PDF it came from. Offsets, chunk hashes, citations all refer to the snapshot. The original PDF could vanish tomorrow and every chunk would still resolve against its preserved snapshot. Capture doesn't just index a source; it preserves a specific, versioned reading of that source.

When you transform a source, preserve the transformation, not just the pointer. If ingesting a source means converting it, the converted output is the thing your index and citations actually depend on — so that output must be captured and preserved, not regenerated on demand from a source that might change or disappear. The snapshot of the conversion is the real ground truth for a captured corpus; treat it as precious, because nothing else can reproduce it.

Per-File Failures Never Abort the Run

Converting real-world documents is messy — some PDFs are malformed, some formats aren't supported, some conversions just fail. A captured reindex over hundreds of files must not die because file 200 was a broken scan. So per-file conversion failures are recorded in the run's error stats and the walk keeps going. You get an honest report — this many converted, this many skipped as unchanged, these specific files errored — instead of an all-or-nothing crash. Robust ingest degrades per-file, not per-run.

Code

Captured adapter: convert, snapshot, index — errors don't abort·python

class CapturedAdapter:
    """The source needs conversion. Convert -> snapshot -> index; skip unchanged."""

    def reindex(self, corpus) -> dict:
        stats = {"converted": 0, "skipped": 0, "errors": []}
        for src in discover(corpus.roots, corpus.include_globs):
            source_sha = sha256(read_bytes(src))
            if source_sha == last_source_sha(corpus.id, src):
                stats["skipped"] += 1                     # unchanged: no re-convert
                continue
            try:
                text = self.converter.convert(src)        # PDF/DOCX/HTML -> clean text
                snap = snapshot_store.append(text)        # doc_sha256 == snapshot hash
                index_document(corpus.id, src, doc_sha=snap, source_sha=source_sha,
                               text=text)
                stats["converted"] += 1
            except ConversionError as e:
                stats["errors"].append((src, str(e)))     # one bad file != dead run
        return stats

Exercise

Pick a source you'd need to convert before indexing — a PDF, a slide deck, a web page. Sketch the captured flow for it: what converter turns it into clean text, and what would you snapshot? Then answer the preservation question: if that original source were deleted a year from now, would your index still resolve? If not, you haven't captured the conversion — you've only referenced a source that can vanish.

Hint

The difference between referencing and capturing is whether you keep the converted output. If you'd have to re-run the converter (against a possibly-gone source) to answer a citation, you referenced when you should have captured. Snapshot the conversion so the citation survives the source.

The Hard Half of Ingest

Convert, Snapshot, Index

The Snapshot Is the Document

Per-File Failures Never Abort the Run

Code

External links

Exercise

Progress

Comments 0