Referenced Adapters

"The easy case: the file is already the writing. Read it, don't touch it, hand it on."

The Simple Half of Ingest

A referenced adapter handles the sources where no conversion is needed: markdown, plain text — files whose bytes, once decoded, are the document. There's no PDF to crack open, no HTML to strip. The adapter's whole job is to read the file, decode it to canonical text, and hand that text to the chunker. It is the shortest path through the ingest layer, and most of a personal corpus lives here.

Read, Never Rewrite

The one iron rule of any adapter — referenced or captured — is that it reads the source and never modifies it. A referenced adapter opens the file, decodes the bytes, and produces canonical text plus its hash. The file on disk is untouched: same bytes, same modification time, same everything. This is the ingestion-never-moves-a-file principle enforced at the lowest level. The adapter is a one-way valve; text flows out, nothing flows back into the source.

The Adapter Interface Is the Seam

Every corpus class, simple or complex, meets the rest of the engine at the same narrow interface: given a source, produce canonical text and a content hash. That uniform seam is what lets the chunker, the indexer, and everything downstream stay blissfully ignorant of where text came from. A markdown file and a converted PDF both arrive as canonical text with a hash; the machinery after the adapter can't tell them apart and doesn't need to. Defining that seam cleanly is what makes adding a new source type a small, local change.

Isolate 'how do I read this?' behind one uniform interface. The variety of the world — encodings, formats, conversions — should be absorbed at a single boundary, the adapter, and never leak past it. Everything downstream should receive one clean type: canonical text with provenance. When the messy diversity of inputs is quarantined at the edge, the core of the system gets to be simple and format-agnostic.

Even Simple Sources Earn an Adapter

You might think plain text is too trivial to need an adapter — just open the file, right? But even here there's a decode decision: which text encoding, how to handle line endings, what counts as the canonical form. Routing even the simplest source through an adapter means there's exactly one place where 'how is this read into canonical text?' is answered, for every source type. Consistency at the boundary is worth more than the few lines it costs; the alternative is decode logic scattered across the codebase, disagreeing with itself.

Code

A referenced adapter: read in place, emit canonical text + hash·python

class ReferencedAdapter:
    """The file's bytes ARE the document. Read, decode, hash — never modify."""

    def to_canonical(self, path: str) -> Document:
        raw = Path(path).read_bytes()             # read-only; source untouched
        text = raw.decode("utf-8")                # bytes -> canonical text, one decision
        return Document(
            relpath=path,
            text=text,                            # what the chunker will slice
            doc_sha256=sha256(raw),               # provenance
        )

# The chunker and indexer never learn this was a .md vs a .txt vs a converted PDF.
# They receive one shape — canonical text + hash — from every adapter. That
# uniformity is what keeps the core format-agnostic.

Exercise

Design the adapter interface for a system that ingests several source formats. Write the single function signature every adapter must implement — what it takes, what it returns. Then confirm: could you add a brand-new format by writing only one new adapter, touching nothing downstream? If not, your seam is leaking format-specific details past the boundary; find them and push them back behind the interface.

Hint

The interface should be something like 'source in, canonical-text-plus-hash out'. If any downstream code has to branch on 'is this a PDF or a markdown file?', the adapter boundary has failed — that branch belongs inside an adapter, not in the core.

The Simple Half of Ingest

Read, Never Rewrite

The Adapter Interface Is the Seam

Even Simple Sources Earn an Adapter

Code

External links

Exercise

Progress

Comments 0