The 1M-Token Illusion

The number on the box is a feature flag, not a daily driver

By 2026, models routinely advertise 200K, 400K, 1M, even 2M token windows. Marketing materials wave those numbers like banners. Reality is messier: a window that fits is not a window you should fill. Three cliffs lie in wait.

Cliff 1 — Cost

For most major providers, per-token pricing changes once you cross a long-context threshold (usually around 200K input). Anthropic Sonnet doubles input price and lifts output meaningfully past 200K in 1M-beta mode. Gemini 2.5 Pro doubles input price past 200K. OpenAI's long-context tiers carry their own surcharge. Long context is a paid capability, not a free upgrade.

Cliff 2 — Latency

Time-to-first-token (TTFT) at full context is brutal. A real prefill of ~1M tokens commonly lands in the 30 second to 2 minute range before the model emits its first character. That is fine for batch analysis and unusable for an interactive code review. "Fast" and "long context" are usually two different choices.

Cliff 3 — Attention

Even when you can afford the bill and the wait, the model often does not actually attend to the entire input with equal fidelity. Benchmarks like RULER and NoCha show effective reasoning often collapsing well before the advertised limit. "It can read it" is not "it can think across it."

1M context is a nuclear reactor, not your daily driver. Powerful when you genuinely need it; ruinously wasteful when a 32K window would have done the same job better.

Code

Long-context cliffs at a glance·yaml

long_context_cliffs:
  cost:
    threshold: "~200K input on most providers"
    effect: "input price typically 2x; output also rises"
  latency:
    one_million_token_prefill: "30s to 2min TTFT is normal"
    effect: "interactive feel disappears"
  attention:
    benchmark: ["RULER", "NoCha", "LongBench v2"]
    effect: "effective reasoning often 30-50% of advertised window"

When long context is the right tool·python

def use_long_context(task):
    return (
        task.is_batch_or_offline
        and task.requires_holistic_reasoning
        and not task.tolerates_retrieval_pipeline
        and task.willing_to_pay_premium_tier
    )
# Otherwise, prefer RAG + small context + tool calls.

Exercise

Take one workflow you currently push into a 1M-token window. Sketch a RAG-plus-checkpoint version that delivers the same answer at 32K-64K. List what you would lose, and what you would gain in latency and cost.

Hint

If the only thing you would lose is the bragging right of saying '1M context,' that is your answer.

The number on the box is a feature flag, not a daily driver

Cliff 1 — Cost

Cliff 2 — Latency

Cliff 3 — Attention

Code

External links

Exercise

Progress

Comments 0