RAG vs Stuffing: Pick by Lookup Frequency

Two ways to bring your data

You can either stuff the relevant text into the prompt or retrieve it on demand. Stuffing is simpler — paste the doc and ask. Retrieval is leaner — embed the corpus, fetch the top-k chunks, paste only those. Both are valid; the right answer depends on how big and how variable your knowledge base is.

The decision rule

If the knowledge base fits comfortably in context AND the same content feeds most queries, stuff with prompt caching — the cache makes repeated reads cheap. If the knowledge base is much larger than context OR queries vary widely, retrieve. cwkPippa stuffs the persona + vault index (stable, ~30KB cached) and retrieves specific files on demand via the Read tool.

What good RAG looks like

Quality embedding model (the SDK does not ship one — pair with Ollama bge-m3, OpenAI text-embedding-3, or similar), reasonable chunk size (512-1024 tokens with overlap), a re-ranker if you have many candidates, and provenance preserved through to the answer so you can cite.

Principle: RAG is overkill for small stable corpora. Stuffing is overkill for big variable ones. Match the pattern to the data.

Code

Stuffing with prompt caching·python

STABLE_KB = open("product_kb.md").read()  # 30KB of stable docs

resp = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": f"Use the knowledge base to answer.\n\n<kb>\n{STABLE_KB}\n</kb>",
            "cache_control": {"type": "ephemeral"},
        }
    ],
    messages=[{"role": "user", "content": user_question}],
)

RAG: retrieve top-k chunks then ask·python

# Pseudo: retrieve top-3 chunks via your vector store
hits = vector_store.query(user_question, k=3)
context = "\n\n".join(f"[doc {h.id}] {h.text}" for h in hits)

resp = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system="Answer using ONLY the provided context. If the answer is not in context, say so.",
    messages=[{
        "role": "user",
        "content": f"<context>\n{context}\n</context>\n\nQuestion: {user_question}",
    }],
)

Exercise

For one knowledge-driven feature, prototype both stuffing-with-cache and RAG. Measure latency, cost, and answer quality across 30 representative queries. Decide based on data, not gut.

Hint

Cost comparison: stuffing is paid once per cache window; RAG pays per query. Crossover is usually somewhere around 10-50 queries per cache TTL.

RAG vs Stuffing: Pick by Lookup Frequency

Two ways to bring your data

The decision rule

What good RAG looks like

Code

External links

Exercise

Progress

Comments 0