Reranking — When the Top-K Lies

Vector similarity is not relevance

Embedding-based retrieval ranks by cosine similarity, which is correlated with relevance but not the same thing. The top-K from a vector search routinely contains a chunk that's similar in vibe but answers a slightly different question. Rerankers fix this by scoring each candidate against the actual query with a more expensive model.

The two-stage pattern

Retrieve — cheap, broad. Get top-50 from vector + BM25 hybrid.
Rerank — expensive, narrow. Use a cross-encoder or LLM to score each of the 50 against the query.
Pack — feed the top-N reranked chunks to the synthesis model.

When reranking earns its cost

Recall matters more than precision (legal, medical, support — you want the right chunk in the answer set).
Queries are short and ambiguous (single keywords, partial phrases).
The corpus has near-duplicates (rerankers help disambiguate).

When to skip

Latency budget is tight (you can't add 200ms).
Retrieval is already 95% accurate at top-3.
Cost dominates and accuracy is acceptable.

Code

Reranking with Cohere or LLM judge·python

import cohere

cohere_client = cohere.Client()
results = cohere_client.rerank(
    query=user_query,
    documents=top_50_chunks,  # from vector search
    model="rerank-english-v3.0",
    top_n=5,
)

final_chunks = [r.document for r in results.results]

# Or LLM-as-judge alternative:
# Send the 50 chunks to an LLM with a scoring prompt and pick top 5.

Reranking — When the Top-K Lies

Vector similarity is not relevance

The two-stage pattern

When reranking earns its cost

When to skip

Code

External links

Exercise

Progress

Comments 0