Token Economics

Tokens are the fundamental unit of API billing and context management. OpenAI uses tiktoken, a fast BPE (Byte Pair Encoding) tokenizer. Understanding token counts helps you manage costs and stay within context windows.

Encoding by Model Family

Encoding	Models
`o200k_base`	gpt-5.x, gpt-4o, gpt-4.1, o3, o4-mini
`cl100k_base`	gpt-4, gpt-3.5-turbo, text-embedding-3-*
`p50k_base`	Codex models, davinci/curie (legacy)

Prompt Caching

Identical prompt prefixes are automatically cached server-side. Cache hits are charged at approximately 10% of the input price. For example, gpt-5.4 cached input costs $0.25/1M tokens vs. $2.50/1M standard. Use prompt_cache_key and prompt_cache_retention: "24h" to optimize cache hit rates.

Why Korean and Japanese tokenize hotter

The default English tokenizer (cl100k_base / o200k_base) was trained on English-heavy corpora. Latin-script languages average ~4 chars per token. CJK languages average ~1.5 chars per token because each CJK character often consumes its own token slot. A 1000-character Korean prompt can be 600+ tokens — roughly 2x what an English speaker eyeballing the length would expect.

The practical consequence: a multi-lingual app can have wildly different per-user costs for the same product behavior. Either budget per-language or use server-side prompt caching aggressively to amortize the system-prompt portion.

Code

Counting tokens with tiktoken·python

import tiktoken

# Get encoding for a model
enc = tiktoken.encoding_for_model("gpt-4o")

# Or by encoding name directly
enc = tiktoken.get_encoding("o200k_base")

# Count tokens
text = "Hello, how are you today?"
tokens = enc.encode(text)
print(f"Token count: {len(tokens)}")  # → 6

Cost estimation helper·python

import tiktoken

def count_tokens_for_messages(messages: list[dict], model: str = "gpt-4o") -> int:
    """
    Approximate token count for a list of chat messages.
    Each message incurs a ~4-token overhead; each reply starts with 3 tokens.
    """
    enc = tiktoken.encoding_for_model(model)
    tokens_per_message = 3
    tokens_per_name = 1

    total = 0
    for msg in messages:
        total += tokens_per_message
        for key, value in msg.items():
            total += len(enc.encode(str(value)))
            if key == "name":
                total += tokens_per_name
    total += 3  # every reply is primed with <|start|>assistant<|message|>
    return total

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is the capital of France?"},
]
print(count_tokens_for_messages(messages))  # → ~26 tokens

Encoding by Model Family

Prompt Caching

Why Korean and Japanese tokenize hotter

Code

External links

Exercise

Progress

Comments 0