Context Window Implications

Context window is the maximum number of tokens the model can process in one request. Its implications cascade through application design, cost, latency, and architecture.

Context	Equivalent text	Useful for
4K	~3,000 words / 6 pages	Short conversations, simple tools
32K	~24,000 words / 48 pages	Longer documents, focused code review
128K	~96,000 words / 200 pages	Books, large codebases
1M	~750,000 words / 1,500 pages	Whole repositories, long-form research
10M	~7.5M words	Entire document corpora; LLaMA 4 Scout

The trade-offs

Cost. Per-token API pricing means longer prompts cost proportionally more. A 1M-token prompt at $1.25/1M is $1.25 just to read the input.
Latency. Prefill (processing the prompt) scales linearly with context length and is compute-bound. A 128K prefill on a 70B model takes seconds before any output appears.
KV-cache memory. Linear in context length. At 128K, can be 30+ GB.
Quality at depth. Models have a "lost in the middle" effect — information at depth 50K may not be retrieved as reliably as information at depth 1K, even when both are within the advertised context window.

Don't pay for context you don't use. RAG (retrieval-augmented generation) often beats stuffing every document into a long-context model — both on cost and on quality at depth.

Code

Cost estimator: long context vs RAG·python

# Hypothesis: 'just send the whole 200-page doc' vs 'retrieve top 5 chunks of 1000 tokens.'
def cost(input_tokens, output_tokens, in_per_1m, out_per_1m):
    return input_tokens / 1e6 * in_per_1m + output_tokens / 1e6 * out_per_1m

# 200-page doc ≈ 100K tokens
long_ctx  = cost(100_000, 500, 1.25, 10.00)    # Gemini 2.5 Pro
rag_5x1k  = cost(5_000, 500, 1.25, 10.00)
print(f"Long context: ${long_ctx:.4f},  RAG: ${rag_5x1k:.4f}")
# RAG is ~20× cheaper per query. Often the better default.

The trade-offs

Code

External links

Exercise

Progress

Comments 0