Lost in the Middle

Position matters more than total length

The 2023 paper Lost in the Middle: How Language Models Use Long Contexts by Liu et al. showed something disturbing: when relevant information sits in the middle of a long context, model accuracy drops sharply. Performance is U-shaped — best at the start, second-best at the end, worst in the middle. The effect persists across model families and across years of capability improvements.

The lecture analogy

Imagine a 90-minute lecture. You remember the opening, you remember the wrap-up, but minute 47? Mostly fog. Models do exactly the same thing. Naive stuffing is dangerous: you can include the right fact and still place it where the model is least likely to use it.

Practical implications

Put critical instructions at the start or the end of your prompt. Sandwich your task between context, not buried in it. RAG retrieval order matters: put the most relevant chunks at positions 1, 2, and N-1, N — not the middle. Do not trust 'I gave it the data, it should know' when that data sits in the middle of a 100K-token blob.

If a fact decides the task, do not bury it in the middle of a giant prompt. The model is statistically least likely to use it there.

Code

Position-aware ordering·yaml

context_order:
  front:
    - live rules
    - source of truth
    - task objective
  middle:
    - supporting references
    - optional detail
  tail:
    - latest human decision
    - latest tool result
    - next action

Quick A/B test·python

def test_lim(model, fact, distractor_blob):
    # A: fact at start
    a = model.ask(f"FACT: {fact}\n\n{distractor_blob}\n\nWhat was the fact?")
    # B: fact in middle
    mid = len(distractor_blob) // 2
    b = model.ask(f"{distractor_blob[:mid]}\nFACT: {fact}\n{distractor_blob[mid:]}\n\nWhat was the fact?")
    return {"start": a, "middle": b}
# The gap will surprise you.

Position matters more than total length

The lecture analogy

Practical implications

Code

External links

Exercise

Progress

Comments 0