Retrieval-Augmented Generation Is Not Magic Glue

RAG is a search problem first

RAG is often described as "the model gets to read your data," which makes it sound like magic. It's a search problem with the LLM bolted on top: retrieve relevant chunks, pack them into the prompt, ask the model to answer using only those chunks. If retrieval brings garbage, the model produces well-written garbage with citations. The bottleneck is almost always retrieval quality, not the LLM.

The five pieces

1. Chunking — how you split documents.
2. Embedding — how chunks become vectors.
3. Index — where vectors live (FAISS, Chroma, Postgres pgvector, dedicated DBs).
4. Retrieval — query → top-K chunks. Often hybrid (dense + BM25).
5. Synthesis — the LLM call that turns chunks into an answer.

Where most RAG systems fail

• Chunks are too big (LLM gets answers, but the chunk is also full of unrelated content).
• Chunks are too small (retrieval gets keyword matches that miss the surrounding argument).
• Top-K is set blindly (5 by default, sometimes 50 helps, sometimes 1 is right).
• No reranking after the initial retrieval.
• Synthesis prompt doesn't insist on citations from the provided chunks only.

Diagnose before tuning

If outputs are wrong, check: did the right chunk get retrieved? If yes, fix the synthesis prompt. If no, fix retrieval. Tuning the LLM call when retrieval is broken is wasted work.