Long-Context Limits — When the Model Forgets the Middle

Lost in the middle is a real effect

Even on 1M-token-window models, attention is not uniform. Information at the start and end is recalled with high fidelity; information buried in the middle is recalled with measurable degradation. The needle-in-a-haystack benchmarks have improved year over year, but they don't go away — and most production prompts are not as well-structured as a needle test.

Tactics that work

Order by importance — most relevant chunks first, not by document order.
Repeat the instruction at the end — a one-line reminder after a long evidence block re-anchors the task.
Trim aggressively — 200k tokens of marginally-relevant context is worse than 20k tokens of well-chosen context.
Hierarchical retrieval — first retrieve a summary level, then drill into the relevant section. Avoids dumping everything into the middle.

Test for it

Inject a known fact at three positions (start, middle, end) of a long context. Ask the model to recall it. If middle-position recall is materially worse, your prompt is feeling the lost-in-the-middle effect.

Code

Position-sensitive recall test·python

import random
import anthropic

def build_haystack(needle: str, position: str, total_tokens: int = 100_000) -> str:
    filler = "Lorem ipsum ... " * (total_tokens // 4)  # crude
    parts = filler.split(". ")
    if position == "start":
        out = needle + ". " + ". ".join(parts)
    elif position == "end":
        out = ". ".join(parts) + ". " + needle
    else:  # middle
        mid = len(parts) // 2
        out = ". ".join(parts[:mid] + [needle] + parts[mid:])
    return out

# Run with the same needle at three positions; compare recall.

Long-Context Limits — When the Model Forgets the Middle

Lost in the middle is a real effect

Tactics that work

Test for it

Code

External links

Exercise

Progress

Comments 0