Context Compression — Summarize What You Don't Need

Cheaper to summarize than to send everything

If a document is 50k tokens but the answer only needs the conclusion, you can summarize the document offline once, store the summary, and send the summary at query time. The user gets the same answer with 5% of the tokens. The expensive part — reading 50k tokens — happens once.

Two compression patterns

Static compression — generate summaries at ingest time. Embed both the chunk and its summary; retrieve summaries first, drill into chunks only when needed.
Dynamic compression — at query time, rewrite long chunks down with a cheap model before passing to the expensive model. Useful for chat history compaction.

The fidelity tradeoff

Compressed contexts lose information. Summaries are interpretations, not the original text. For tasks that require exact quotes (legal, audit, citations) summaries are insufficient — keep originals available, compress only the surrounding context. For tasks where the gist is enough (intent classification, routing) summaries are net win.

Code

Static compression at ingest·python

def summarize(chunk: str) -> str:
    return client.messages.create(
        model="claude-haiku-4-5",  # cheap, fast
        max_tokens=128,
        messages=[{"role": "user",
                    "content": f"Summarize in 2 sentences:\n{chunk}"}],
    ).content[0].text

for doc in docs:
    for chunk in chunk_doc(doc):
        index.upsert(
            id=chunk.id,
            text=chunk.text,
            summary=summarize(chunk.text),  # used in low-cost retrieval pass
        )

Context Compression — Summarize What You Don't Need

Cheaper to summarize than to send everything

Two compression patterns

The fidelity tradeoff

Code

External links

Exercise

Progress

Comments 0