Structure-Aware: Markdown, HTML, Code

~22 min · chunking, markdown, code

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Headings are free metadata

Markdown and HTML carry hierarchy that a generic splitter throws away. A structure-aware splitter walks the heading tree and keeps the parent-heading path attached to every chunk. Now your retrieval result is not just "...the deployment workflow..." — it is "Operations > Deploy > Production rollback — the deployment workflow...". The model gets context for free, and you get a citation breadcrumb.

Code splitting is a different problem

Plain text splitters break Python halfway through a function and Rust halfway through a generic. Use a language-aware splitter (LangChain's Language.PYTHON, tree-sitter, or AST-based) so a chunk is always a coherent function, class, or top-level statement. Search relevance jumps immediately.

Code

Markdown header splitter that preserves the breadcrumb·python

from langchain_text_splitters import MarkdownHeaderTextSplitter

splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ('#',   'h1'),
        ('##',  'h2'),
        ('###', 'h3'),
    ],
    strip_headers=False,
)

chunks = splitter.split_text(open('runbook.md').read())
for c in chunks[:3]:
    print(c.metadata, '→', c.page_content[:80].replace('\n', ' '))

Python-aware code splitter (LangChain)·python

from langchain_text_splitters import RecursiveCharacterTextSplitter, Language

py_splitter = RecursiveCharacterTextSplitter.from_language(
    language=Language.PYTHON, chunk_size=600, chunk_overlap=80,
)

chunks = py_splitter.split_text(open('app/services/embedding.py').read())
for c in chunks[:3]:
    print('---')
    print(c[:200])

External links

Exercise

Take a multi-page Markdown document (or your README + a runbook). Compare top-3 retrieval results between a plain recursive splitter and the Markdown header splitter for the same 5 queries. Note which gives more accurate, more attributable answers.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousTokens vs Characters: Stop Slicing Blind Next →Metadata That Earns Its Keep

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.