Embedding-Based Recall and Memory Indexing

What to embed in a memory store

Not every utterance deserves a vector. A good rule: embed memory items that have content worth searching against, with metadata worth filtering on. Greetings, acknowledgements, and tool-output JSON are usually noise.

Three useful memory item shapes

Turn-level — one chunk per user/assistant message. Easy, lots of items, retrieval can feel scattered.
Episode-level — group consecutive turns into a coherent episode (a debugging session, a feature design discussion). Better retrieval relevance, harder to chunk automatically.
Distillation-level — model-written summaries of episodes ('User decided to use HNSW over IVFFlat after a 30-min latency benchmark'). Highest signal-to-noise, requires periodic distillation pass.

Decay-aware retrieval

Recent memories should win ties over old ones. Score = similarity * exp(-age_days / half_life). Tune half_life on your eval set; cwkPippa uses ~30 days for general memories, longer for identity-defining moments.

Code

Decay-aware reranking·python

import math
from datetime import datetime, timezone

def decay_rerank(candidates: list[dict], half_life_days: float = 30.0):
    now = datetime.now(timezone.utc)
    for c in candidates:
        ts = datetime.fromisoformat(c['metadata']['created_at'])
        age_days = (now - ts).total_seconds() / 86400
        decay = math.exp(-age_days / half_life_days)
        c['final_score'] = c['similarity'] * decay
    return sorted(candidates, key=lambda c: -c['final_score'])

Episode distillation pass (nightly)·python

DISTILL_PROMPT = '''Read the following turns and produce 1-3 standalone facts worth remembering long-term. Each fact should be self-contained — readable in 6 months without re-reading the conversation.

Turns:
{turns}

Output as JSON: [{{"fact": "...", "tags": ["..."]}}]
'''

def distill_episode(turns: list[dict]) -> list[dict]:
    text = '\n'.join(f"{t['role']}: {t['content']}" for t in turns)
    return json.loads(llm.complete(DISTILL_PROMPT.format(turns=text)))

Embedding-Based Recall and Memory Indexing

What to embed in a memory store

Three useful memory item shapes

Decay-aware retrieval

Code

External links

Exercise

Progress

Comments 0