Chunking, the Atom of Retrieval

"You don't want the book. You want the paragraph. Chunking is deciding where the paragraph ends."

Why Chunk at All?

A whole document is the wrong unit of retrieval. If someone searches for a specific idea, handing back an entire 10,000-word essay is nearly useless — the relevant passage is buried in it. And embedding models have a hard token ceiling; you can't vectorize an arbitrarily long document in one shot. So documents are cut into chunks: passages small enough to be a precise answer and to fit an embedding model's window, large enough to carry a coherent thought. The chunk, not the document, is what search actually returns.

The Chunk Is the Atom

Everything in the engine is built on the chunk as its smallest unit. A chunk is what gets indexed, what gets embedded, what gets ranked, what gets cited. It's the atom of retrieval — indivisible from the engine's point of view. And because it's the citable unit, it must carry its full provenance: which document it came from, exactly where in that document it sits, and the hash that lets anyone verify it. A chunk isn't just a slice of text; it's a slice of text that knows exactly where it lives.

Character Offsets Into Canonical Text

A chunk's location is recorded as character offsets — char_start and char_end — into the UTF-8-decoded document text. Not byte offsets (those break the moment a multi-byte character appears), not line numbers (ambiguous and fragile), but character positions into the exact same canonical text that was hashed into the document's id. This is what makes a chunk re-sliceable: hand someone the document and the two offsets, and they can cut the identical span and check it against the chunk hash. Offsets into canonical text are the difference between a citation you can verify and one you have to trust.

Define your atom, and make it carry its own address. Every retrieval system has a smallest unit it returns. Decide what yours is deliberately, then make that unit self-locating: it should know its source and its exact position, so any result can be traced back and re-verified from scratch. An atom that can't say where it came from can't be cited — and an uncitable result is just a rumor.

Whole-Document Mutation

One more rule flows from the chunk being the atom: when a document changes, you don't try to figure out which individual chunks shifted and patch them. You delete all of that document's chunks and reinsert the freshly computed set, in a single atomic transaction. Per-chunk patching sounds efficient but it's a corruption factory — one miscalculated boundary and the offsets no longer line up with the text. The whole document is the smallest safe unit of change, and atomic replacement keeps the index honest at every moment.

Code

A chunk knows its exact, verifiable location in the source·python

def chunk_document(text: str, profile) -> list[Chunk]:
    """Cut a document into chunks, each carrying its exact offsets into `text`."""
    chunks, cursor = [], 0
    for span in split_paragraphs(text, max_chars=profile.max_chars):
        start = text.index(span, cursor)      # character offset into canonical text
        end = start + len(span)
        chunks.append(Chunk(
            text=span,
            char_start=start,                  # re-sliceable: text[start:end] == span
            char_end=end,
            chunk_sha256=sha256(span),         # verifiable
        ))
        cursor = end
    return chunks

# On change: delete ALL of a document's chunks, reinsert the new set atomically.
# Never try to patch individual chunks — that's where offsets drift out of sync.

Exercise

Take a long document and cut it into chunks by hand at natural boundaries. For each chunk, write down its start and end character offset. Now verify: does text[start:end] give you back exactly that chunk, character for character? If your offsets are off by even one, you've just felt why the engine records offsets into canonical text and re-verifies against a hash — small drift silently breaks every citation.

Hint

The test of a good chunk record is that the document plus the offsets can reconstruct the exact chunk with no ambiguity. If you needed to also remember 'oh, and I stripped the trailing newline', your offsets aren't pointing at canonical text — and that gap is where verification fails.

Chunking, the Atom of Retrieval

Why Chunk at All?

The Chunk Is the Atom

Character Offsets Into Canonical Text

Whole-Document Mutation

Code

External links

Exercise

Progress

Comments 0