Feedback Loops and Self-Correcting Retrieval

~22 min · rag, advanced

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The single-pass RAG ceiling

One retrieve → one answer is fine for crisp questions. It breaks for vague queries or compound questions where one search cannot return everything you need. Two patterns push past this ceiling.

Self-RAG: model judges its own answer

After generating, ask the model to score the answer's groundedness against the retrieved chunks. If the score is low, retry with a rewritten query or expanded k. The cost is one extra LLM call; the win is a built-in quality gate.

Agentic retrieval

Give the LLM a 'search' tool and let it decide when to call it. The model can plan multi-hop queries: first retrieve general context, then narrow with a follow-up search informed by what it found. This is how Anthropic's tool-use, OpenAI's function calling, and modern agent frameworks all expose retrieval.

Code

Self-RAG critique loop·python

def rag_with_self_critique(question: str, max_retries: int = 2):
    for attempt in range(max_retries + 1):
        result = rag(question)
        critique_prompt = (
            f'Question: {question}\n\n'
            f'Answer: {result["answer"]}\n\n'
            f'Context used: {[c["text"][:200] for c in result["citations"]]}\n\n'
            f'Score the answer 1-5 on how well the context supports it. '
            f'Output: just the number, then a one-line reason.'
        )
        score_line = llm.complete(critique_prompt)
        score = int(score_line.strip()[0])
        if score >= 4 or attempt == max_retries:
            return result
        question = expand_query(question, n=1)[0]   # rewrite and try again

Search as a tool (agentic)·python

tools = [
    {
        'name': 'search_knowledge_base',
        'description': 'Search the company knowledge base for the given query. Returns top-5 chunks with sources.',
        'input_schema': {
            'type': 'object',
            'properties': {'query': {'type': 'string'}},
            'required': ['query'],
        },
    }
]

response = anthropic_client.messages.create(
    model='claude-opus-4-7',
    max_tokens=2000,
    tools=tools,
    messages=[{'role': 'user', 'content': 'What is our refund policy for annual plans?'}],
)

# Loop: read tool_use blocks, call retrieve(), feed back as tool_result, until model responds with text.

External links

Exercise

Wire your RAG behind a search tool exposed to the LLM. On 10 multi-hop questions ('When was the most recent policy that updates the legacy 2023 retention rule?') compare single-pass RAG vs agentic retrieval. Note answer quality and total tool-call count.