When RAG Is the Wrong Answer

The cases where RAG hurts

RAG is the right default for most LLM-over-private-data apps. It is the wrong tool when:

The corpus fits in the context window. If your knowledge base is 50 pages, just stuff it. Building retrieval is wasted effort.
The question requires structured reasoning over the whole corpus. 'How many of our docs mention compliance?' is a SQL query, not a vector search.
The user wants a transformation, not a lookup. 'Translate this' or 'summarize this' does not need retrieval — you already have the input.
Latency is the killer constraint. Each retrieval call adds 50–500 ms. For autocomplete-style UX, this is too much.
The data updates faster than you can re-embed. Live trading data, sensor streams — push these to the model directly, do not try to embed them.

Long context vs RAG

With 200K-1M token context windows, the line moves. For corpora under ~150K tokens, stuffing the entire knowledge base into context can outperform RAG (better synthesis, no retrieval misses) at higher cost per call. Measure on your own data — there is no universal answer.

Code

Decision sketch·text

Corpus ≤ 50 docs?               → stuff into context, skip retrieval
Question is structured?         → SQL or filter, not vector search
Question is a transformation?   → no retrieval needed
Latency budget < 100 ms?        → retrieval is too slow
Data updates faster than embedding pipeline? → direct injection
Else                            → RAG

When RAG Is the Wrong Answer

The cases where RAG hurts

Long context vs RAG

Code

External links

Exercise

Progress

Comments 0