Tokens, Context Limits, and Truncation

Embedding models have hard token limits

An embedding call accepts text up to a maximum number of tokens, not characters. Tokens are subword pieces from the model's tokenizer — for English, roughly 0.75 words per token. Korean, Japanese, and code can be much denser.

Common limits in 2026:

OpenAI text-embedding-3-large: 8,192 tokens
Voyage voyage-3-large: 32,000 tokens
BGE-M3: 8,192 tokens
BGE-small-en-v1.5: 512 tokens (this trips people)
all-MiniLM-L6-v2: 256 tokens

What happens past the limit

Almost every API silently truncates. You will not get an error — the model embeds the front of your text and discards the rest. Your vector will look fine, your search will look like it works, and you will lose the back half of every long document.

How to defend against silent truncation

Use the model's tokenizer to count tokens before sending.
If tokens > limit, either chunk the input (next track) or use a long-context model.
Log the chunk length distribution after ingestion. If P99 hits exactly the limit, you are losing data.

Code

Count tokens before embedding (OpenAI)·python

import tiktoken

enc = tiktoken.encoding_for_model('text-embedding-3-large')

def count_tokens(text: str) -> int:
    return len(enc.encode(text))

long_text = open('white_paper.md').read()
print(count_tokens(long_text))   # if > 8192, you must chunk

Count tokens for a HuggingFace model·python

from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained('BAAI/bge-m3')
print(len(tok.encode(long_text)))   # uses the actual model tokenizer

Tokens, Context Limits, and Truncation

Embedding models have hard token limits

What happens past the limit

How to defend against silent truncation

Code

External links

Exercise

Progress

Comments 0