Rerank as a Deepening Pass

"Cheap retrieval casts a wide net. The reranker is the expensive eye that looks closely at only the fish you already caught."

Two Ways a Model Can Judge Relevance

There are two shapes of relevance model, and the difference is everything. A bi-encoder (what vector search uses) embeds the query and each document separately into vectors, then compares them. Because documents can be embedded ahead of time, this is fast and scales to millions — but the query and document never actually meet inside the model. A cross-encoder (a reranker) feeds the query and a document into the model together and scores the pair jointly. The model attends to both at once, catching subtle relevance a separated comparison misses.

Better and Slower Are the Same Property

The cross-encoder is more accurate precisely because it looks at query and document together — and that's exactly why it's expensive. You can't precompute anything: every query-document pair is a fresh model call. Rerank a thousand candidates and that's a thousand forward passes, right now, with no caching possible. The very thing that makes it accurate (joint attention) is the thing that makes it too costly to run over an entire corpus. So you don't.

The Deepening Pattern: Retrieve Wide, Rerank Narrow

The resolution is a two-stage pipeline. First, use the cheap methods — BM25 and vectors, fused — to pull a wider-than-usual candidate pool: instead of the top 8, fetch the top 50. These cheap retrievers are great at recall (getting the right answers somewhere in the pool) even if their ordering is rough. Then run the expensive cross-encoder on just those 50, and take its top 8. You've spent 50 model calls, not a million, and you get precision-ranked results. Retrieve wide and cheap; rerank narrow and expensive.

Put expensive precision after cheap recall, never instead of it. The pattern generalizes far beyond search: use a cheap, high-recall filter to shrink the candidate set, then spend your expensive, high-precision method only on the survivors. Running the costly step over everything is wasteful; running it over a well-chosen shortlist is where its accuracy actually pays off. Two stages beat one heroic stage.

Opt-In, and Honest When It Can't Run

Because reranking costs real model calls, Lantern leaves it off by default — a per-query flag you set when the extra precision is worth it. And since it calls the model server, it obeys the same honesty rule as vector search: when the reranker is unreachable, the request degrades and says so, rather than silently returning the un-reranked order dressed up as reranked. Deepening is a capability you opt into with your eyes open, not a hidden step that quietly fails.

Code

Retrieve 50 cheaply, rerank those 50, return the top 8·python

def search_with_optional_rerank(query: str, k: int = 8, rerank: bool = False):
    if not rerank:
        # Cheap path: hybrid retrieval returns the top-k directly.
        return hybrid(query, k=k)

    # Deepening path: retrieve WIDE cheaply, then rerank NARROW expensively.
    pool = hybrid(query, k=50)                      # high recall, rough order, cheap
    if not reranker.reachable():
        # Honest degradation — do NOT pretend the pool was reranked.
        return Envelope(results=pool[:k], degraded=["reranker_unavailable"])

    scored = reranker.score(query, [c.text for c in pool])  # 50 model calls, not 1e6
    reranked = [c for c, _ in sorted(zip(pool, scored), key=lambda p: -p[1])]
    return Envelope(results=reranked[:k], degraded=[])

Exercise

Think of an expensive check you run in your own work — a slow test, a manual review, a costly API call. Are you running it over everything, or over a cheaply-filtered shortlist first? Design the two-stage version: what cheap, high-recall filter could shrink the candidate set before the expensive step? Estimate how many expensive operations you'd save.

Hint

The recipe is always the same: a cheap filter that rarely throws away a real answer (high recall) narrows the field, then the expensive, precise step runs only on the survivors. If you're running the costly step over the full set, you're paying for precision on candidates a cheap filter would have already rejected.

Rerank as a Deepening Pass

Two Ways a Model Can Judge Relevance

Better and Slower Are the Same Property

The Deepening Pattern: Retrieve Wide, Rerank Narrow

Opt-In, and Honest When It Can't Run

Code

External links

Exercise

Progress

Comments 0