Why Chunk at All

Chunking exists for two reasons

Embedding models have a token limit. Anything longer than the limit gets silently truncated.
A whole document averages too many topics into one vector. A 10-page document about "billing" might cover trial signup, refund policy, dunning emails, and tax compliance. Embedding all of it as one point puts you in the average of those topics — far from any specific answer.

The retrieval-vs-context tradeoff

Smaller chunks → more precise retrieval but less context per hit. Larger chunks → richer context but noisier matches and a higher chance the chunk crosses topic boundaries. The sweet spot for most knowledge bases lands in the 200–800 token range with 10–20% overlap between adjacent chunks. That is a starting point, not a law — measure on your own corpus.

The four splitter families

Fixed-size — every N characters or tokens. Brain-dead but reliable.
Recursive character — try paragraphs, then sentences, then words. LangChain's default.
Structure-aware — Markdown headings, HTML tags, code AST. Preserves logical boundaries.
Semantic — embed sentences and start a new chunk when the running average drifts. Highest quality, slowest, most opaque.

Code

Fixed-size character split (the simplest baseline)·python

def fixed_chunks(text: str, size: int = 1200, overlap: int = 200):
    chunks = []
    start = 0
    while start < len(text):
        end = start + size
        chunks.append(text[start:end])
        start = end - overlap
    return chunks

chunks = fixed_chunks(open('handbook.md').read())
print(f'{len(chunks)} chunks, avg {sum(len(c) for c in chunks)//len(chunks)} chars')

Chunking exists for two reasons

The retrieval-vs-context tradeoff

The four splitter families

Code

External links

Exercise

Progress

Comments 0