Summarization, Memory Compression, and Drift

When summaries beat raw history

For long-running chats, raw history grows unbounded and costs grow with it. Periodic summarization replaces older turns with a condensed assistant note, preserving the facts the model needs without the conversational fluff. Done well, this keeps quality high and tokens low.

What to preserve

Good summaries preserve: user-stated facts ('I work in pharma'), explicit preferences ('always reply in Spanish'), unresolved tasks, and conclusions reached. Bad summaries preserve: greetings, banter, and the model's own apologies. Hand-craft your summarization prompt to call out what to keep.

Drift is real

Each summarization pass loses fidelity. Run too many and the model's effective memory degrades into vague impressions. Mitigate by summarizing in tiers — last-N turns verbatim, middle tier summarized lightly, oldest tier heavily summarized — instead of one flat compression.

Principle: Summaries are lossy compression. Decide what is worth keeping before you call the compressor.

Code

Tiered summarization·python

VERBATIM = 8
SUMMARIZED = 24

def tiered(history: list[dict]) -> list[dict]:
    if len(history) <= VERBATIM:
        return history
    recent = history[-VERBATIM:]
    middle = history[-(VERBATIM + SUMMARIZED):-VERBATIM]
    older = history[:-(VERBATIM + SUMMARIZED)] if len(history) > VERBATIM + SUMMARIZED else []

    middle_summary = summarize(middle, depth="light") if middle else ""
    older_summary = summarize(older, depth="heavy") if older else ""

    notes = []
    if older_summary:
        notes.append({"role": "user", "content": f"<early_summary depth='heavy'>{older_summary}</early_summary>"})
    if middle_summary:
        notes.append({"role": "user", "content": f"<recent_summary depth='light'>{middle_summary}</recent_summary>"})
    return notes + recent

Exercise

Add tiered summarization to a multi-turn chat. Run the same 50-turn conversation with and without it; compare total input tokens, response quality on a probe question that depends on early context.

Hint

If the probe question fails after summarization, your summary prompt is dropping facts you need to preserve.

Summarization, Memory Compression, and Drift

When summaries beat raw history

What to preserve

Drift is real

Code

External links

Exercise

Progress

Comments 0