Semantic Splitting and When to Skip It

The idea

Semantic splitting embeds each sentence, computes the running cosine similarity between adjacent sentences, and starts a new chunk where similarity drops below a threshold. The result: chunks aligned to topic boundaries instead of arbitrary lengths.

Why it sometimes wins big

For long, narrative documents (blog posts, transcripts, novels) semantic splitting can outperform fixed-size by 10–20% on retrieval relevance because the chunks stay topical. The model never gets a Frankenstein chunk that ends mid-thought.

Why it often loses in production

Cost. You embed every sentence twice — once for splitting, once for storage.
Latency. Splitting an 80-page document can take minutes.
Opacity. When a chunk looks weird you cannot easily explain why.
Negligible win on already-structured docs. Markdown headings already mark topic boundaries — semantic splitting on top is wasted effort.

Use semantic splitting on long unstructured prose. Reach for the structure-aware splitter first when the document has any markup at all.

Code

Semantic splitter (LangChain experimental)·python

from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings

splitter = SemanticChunker(
    OpenAIEmbeddings(model='text-embedding-3-large'),
    breakpoint_threshold_type='percentile',
    breakpoint_threshold_amount=85,    # lower = more chunks
)

chunks = splitter.create_documents([open('long_essay.md').read()])
print(f'{len(chunks)} semantic chunks')

Semantic Splitting and When to Skip It

The idea

Why it sometimes wins big

Why it often loses in production

Code

External links

Exercise

Progress

Comments 0