Chunking Strategies That Don't Lose the Argument

The 500-token chunk is a default, not a law

A common starter chunk size is 500 tokens with 50-token overlap. It's a default that works on generic prose; it's a disaster on legal documents (where one paragraph spans 2,000 tokens), code (where boundaries are functions or classes), or chat logs (where a turn is the unit of meaning). Chunk size should follow the structure of the source.

Strategies that work

Structural chunking — split on headings, sections, function definitions, log entries. Aligns with how the document was authored.
Sliding window with overlap — generic prose where structure is light. Overlap so a sentence near the boundary stays attached to its neighbors.
Semantic chunking — embedding-based, splits where the embedding distance changes sharply. Useful for free-form text.
Contextual headers — prepend a one-sentence summary of the parent section to each chunk so retrieval doesn't lose the topic.

The argument-preservation test

For each chunking strategy, take a long argument from your corpus (a multi-paragraph claim, a step-by-step procedure). Ask: does the strategy keep the argument together? If the chunk break splits the conclusion from the premises, retrieval will surface either piece without the other and the model will work from half the case.

Code

Contextual chunk header (Anthropic-style)·python

def add_context_header(chunk: str, source_title: str, section: str) -> str:
    return (
        f"<source>{source_title}</source>\n"
        f"<section>{section}</section>\n"
        f"<content>\n{chunk}\n</content>"
    )

# After embedding the contextualized chunk, retrieval becomes
# robust to keyword drift across sections.

Chunking Strategies That Don't Lose the Argument

The 500-token chunk is a default, not a law

Strategies that work

The argument-preservation test

Code

External links

Exercise

Progress

Comments 0