Tokens vs Characters: Stop Slicing Blind

~18 min · chunking, tokens

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The problem with character splits

A 1200-character chunk in English is roughly 240 tokens. The same 1200 characters in Korean might be 700 tokens. The same 1200 characters of Python source might be 400. Your token budget is what the embedding model cares about — character counting silently misbehaves across languages and content types.

Token-aware splitting

Use the model's tokenizer to split. The two common patterns:

Wrap a tokenizer around a recursive splitter — split by paragraph/sentence, then check token count, then split further if needed.
Direct token-based slicing — encode the whole document, slice the token list, decode each slice back to text.

The second is faster but produces ugly mid-word boundaries. Most production systems use the first.

Code

LangChain's tokenizer-aware recursive splitter·python

from langchain_text_splitters import RecursiveCharacterTextSplitter
import tiktoken

enc = tiktoken.encoding_for_model('text-embedding-3-large')

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    encoding_name=enc.name,
    chunk_size=400,        # in TOKENS, not characters
    chunk_overlap=60,
    separators=['\n\n', '\n', '. ', ' ', ''],   # try paragraph first
)

chunks = splitter.split_text(open('handbook.md').read())
for c in chunks[:3]:
    print(len(enc.encode(c)), '→', c[:80].replace('\n', ' '))

Pure-token slicing (faster, uglier)·python

def token_chunks(text: str, size: int = 400, overlap: int = 60):
    ids = enc.encode(text)
    chunks = []
    start = 0
    while start < len(ids):
        slice_ids = ids[start:start + size]
        chunks.append(enc.decode(slice_ids))
        start += size - overlap
    return chunks

External links

Exercise

Take one English document and one Korean document of similar character length. Split each with a 1000-character splitter and a 400-token splitter. Print the token-count distribution. Note how character splitting starves the English chunks and overruns the Korean ones.