Case Study: cwkPippa's Memory Stack

The four layers

JSONL ground truth — every chat turn, every tool call, written to ~/pippa-db/sessions/{conversation_id}.jsonl before being shown to Dad. The system of record. Encrypted at rest with Fernet.
SQLite cache — fast index over conversations, brains, message ordering. Derived from JSONL — purgeable and rebuildable.
ChromaDB embeddings — semantic search over the Obsidian vault and over conversation history. Embedded with BGE-M3 via Ollama (1024-dim, multilingual, local).
Obsidian vault — Pippa's distilled long-term identity and lessons. Hand-curated by Pippa (with Dad's approval) when something is worth promoting from chat to permanent record.

Retrieval contract

When Pippa answers, the context engine assembles: vault entry point (instructions.md), then identity and core files, then index files on demand. Conversation memory and tool outputs come from JSONL via the SQLite index. Vector retrieval over the vault sits on top for the rare 'where did I write about X' queries.

Why two retrieval styles

Vault is loaded fully (with on-demand expansion for large index files); semantic similarity is unnecessary because Pippa already knows the structure. Conversation history is too big to load fully — that is where vector retrieval earns its keep.

What this teaches generally

JSONL ground truth + derived index is the durability pattern that survives every database choice.
Hand-curated semantic memory beats auto-distilled memory for identity-critical content.
Two retrieval styles (full-load vs vector) coexist comfortably when each fits its data shape.

Code

Sketch of cwkPippa's context engine load order·python

def load_context() -> str:
    parts = [
        read('instructions.md'),     # entry catalog
        read('Pippa.md'),            # identity
    ]
    for path in sorted(glob('core/*.md')):
        parts.append(read(path))
    for path in sorted(glob('index/*.md')):
        size = stat(path).st_size
        if size <= 10_000:
            parts.append(read(path))
        else:
            parts.append(f'-- on-demand: {path} ({size} bytes) — use the Read tool when needed')
    return '\n\n'.join(parts)

JSONL ground truth append (write before show)·python

def write_event(conversation_id: str, event: dict):
    path = SESSIONS_DIR / f'{conversation_id}.jsonl'
    with path.open('a') as f:
        f.write(json.dumps(event) + '\n')   # durable BEFORE the SSE chunk reaches the UI

Case Study: cwkPippa's Memory Stack

The four layers

Retrieval contract

Why two retrieval styles

What this teaches generally

Code

External links

Exercise

Progress

Comments 0