Context Windows: Long Is Not Free

Where the budget really lives

Current Claude models support large context windows (200K tokens on the standard Sonnet/Opus, with even longer windows on opt-in models). 'Long' is a capability, not an instruction. Cost scales with input tokens — sending 150K tokens costs 150K input tokens whether the model needed all of them or not.

Effective context vs theoretical

Theoretical context is the API limit. Effective context is the slice the model actually attends to well. Long-context retrieval (the 'needle in a haystack' benchmark) shows Claude is strong, but careful prompt structure (placing critical info near the question) helps even more. Do not rely on the model to dig through 100K tokens for one fact you could have placed last.

Token counting before sending

The messages.count_tokens() endpoint returns the input token count without running a completion — perfect for budget planning. Use it in CI to assert prompts stay under a budget you set, and in production to log surprise growth.

Principle: The context window is the road, not the destination. Travel the shortest route that still gets the model where it needs to go.

Code

Count tokens before paying for them·python

from anthropic import Anthropic
client = Anthropic()

count = client.messages.count_tokens(
    model="claude-sonnet-4-6",
    system=LONG_SYSTEM,
    messages=[{"role": "user", "content": LONG_USER}],
)
print("input tokens:", count.input_tokens)
assert count.input_tokens < 50_000, "prompt over budget"

Place the question near the answer·python

# Bad: question first, then 80K of context.
messages_bad = [{"role": "user", "content": "Summarize section 5.\n\n" + LONG_DOC}]

# Good: context, then explicit question at the end so it is closest to generation.
messages_good = [{"role": "user", "content": LONG_DOC + "\n\nNow summarize section 5."}]

Exercise

For one production prompt, count tokens, log the result, and add a CI assertion that fails if the count exceeds your chosen budget. Adjust the prompt to fit, not the budget.

Hint

If you cannot bring the prompt under budget, you might be doing the model's job inside the prompt — push more work to retrieval.

Context Windows: Long Is Not Free

Where the budget really lives

Effective context vs theoretical

Token counting before sending

Code

External links

Exercise

Progress

Comments 0