Evaluating and Tuning the Stack

~22 min · hybrid, evaluation

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The eval set is the most valuable artifact you will build

Every retrieval improvement is a guess until you measure it. Build a labeled eval set of 50–200 query-document pairs reflecting your real users. For each query, list the document ids that are 'definitely relevant'. That tiny set powers all future tuning.

The four metrics worth knowing

Hit rate @ k — fraction of queries where any relevant doc is in top-k. Easy to read.
MRR (Mean Reciprocal Rank) — 1 / position of the first relevant. Captures 'how high did the right answer land'.
nDCG @ k — accounts for graded relevance and position. Industry standard.
Recall @ k — fraction of relevant docs you actually retrieve. Critical for RAG when the LLM needs all of them.

Run eval on every change

Treat retrieval like code: every tweak (chunk size, model, RRF k, reranker) gets a numbered eval run. Plot the metrics over time. The day a 'small change' tanks recall, the chart catches it before users do.

Code

Build a tiny eval harness·python

import json
from statistics import mean

EVAL = json.load(open('eval.json'))   # [{'query': str, 'relevant_ids': [str]}]

def hit_rate(retriever, k=5):
    hits = []
    for ex in EVAL:
        results = retriever(ex['query'], k=k)
        ids = [r['id'] for r in results]
        hits.append(any(rid in ids for rid in ex['relevant_ids']))
    return mean(hits)

def mrr(retriever, k=10):
    scores = []
    for ex in EVAL:
        results = retriever(ex['query'], k=k)
        ids = [r['id'] for r in results]
        rank = next((i for i, rid in enumerate(ids) if rid in ex['relevant_ids']), None)
        scores.append(0.0 if rank is None else 1.0 / (rank + 1))
    return mean(scores)

print('hit@5:', hit_rate(vector_only,    5))
print('hit@5:', hit_rate(vector_plus_bm25, 5))
print('mrr:  ', mrr(vector_plus_bm25_then_rerank))

External links

Exercise

Build a 50-row eval set from your real corpus (query, list of relevant_ids). Compute hit@5 and MRR for three pipelines: vector only, hybrid + RRF, hybrid + RRF + cross-encoder. Save the numbers. Re-run them after every chunking or model change.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousCross-Encoder Reranking Next →Query-Side Tricks: HyDE and Multi-Query

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.