Advertised Window vs Effective Window

The number on the box is a ceiling

A model may accept a million tokens. That does not mean it reasons over every token with equal fidelity. The advertised window is an API limit; the effective window is the range where the model still uses information reliably for your task.

Why NIAH ('needle in a haystack') misleads

The classic NIAH test inserts one sentence ('the secret code is 42') into a long document and asks the model to retrieve it. Modern flagship models score above 95% on NIAH at 1M tokens. But that is a retrieval task, not reasoning. Pulling one sentence verbatim is much easier than synthesizing many scattered facts.

What the harder benchmarks show

RULER tests multi-needle, variable-tracking, and aggregation across long inputs — most flagship models drop below 50% accuracy by 128K. NoCha (Novel Challenge) tests narrative reasoning over book-length texts and scores often plateau or decline past 64K. LongBench v2 mixes reasoning across formats and effective context typically lands at 30-50% of the advertised window.

Evaluate your actual workload

If your task is legal cross-reference, codebase refactor, or long-session agent work, test that task directly. The relevant question is not 'can it find the needle?' but 'can it preserve intent across the whole job?'

A large context window is a ceiling, not a guarantee of uniform reasoning quality. Treat advertised numbers as marketing and effective numbers as engineering.

Code

Workload-specific eval scaffold·yaml

effective_context_test:
  task: "multi-hop code review"
  lengths: [32_000, 64_000, 128_000, 256_000]
  measure:
    - cites_required_files
    - preserves_constraints
    - avoids_stale_assumptions
    - finishes_within_output_budget
  baseline: "single-file review at 8K"
  fail_threshold: "effective accuracy < 60% of baseline"

Rule of thumb·text

Advertised window = ceiling for *what fits*
Effective window  = roughly 30-50% of advertised for *reasoning*
                  = closer to 80-95% of advertised for pure *retrieval*

Use evals on YOUR task, not provider marketing.

Exercise

Design one effective-context test for a workflow you care about. Make it harder than retrieving a single sentence — require at least two separated facts to be combined.

Hint

Pick a real failure pattern from your work. If you have not seen one, run a long session intentionally and watch what gets lost.

Advertised Window vs Effective Window

The number on the box is a ceiling

Why NIAH ('needle in a haystack') misleads

What the harder benchmarks show

Evaluate your actual workload

Code

External links

Exercise

Progress

Comments 0