Fusion Without False Certainty

Score scales disagree

FTS bm25 and vector cosine have different shapes and directions. One may be better when lower while another increases with similarity. Normalizing and adding them produces query-dependent numbers with no stable meaning.

RRF uses rank instead of raw score. It rewards items near the top of several lists without calibrating incompatible scales. The method is simple, robust, and easy to extend with another lane.

A fusion score is not probability

RRF at 0.03 is not three-percent truth, and 0.08 is not eighty-percent merge confidence. It is only an ordering signal from rank positions. A percent or confidence label claims more meaning than the math provides.

Use Hybrid rank or combined order, and show lane ranks plus match reasons in detail. Hiding a score may be fine; hiding provenance is not.

A missing lane is a state

When embeddings fail, lexical results may continue. Label them lexical-only and show Meaning unavailable instead of pretending fusion remains complete. Otherwise users read relevance change as content change.

Vector-only service may likewise continue during FTS rebuild. Degradation policy should preserve lane independence and distinguish zero results from zero available lanes.

Evaluate with a query set

One pleasing result cannot validate fusion. Build a fixed set spanning exact names, Korean paraphrases, acronyms, and long conceptual questions, then compare lexical, vector, and fused top-k.

Use relevance judgments from people who read source. Clicks contain position bias, and opening a curation candidate may mean it looks wrong rather than relevant. Behavioral proxies are not ground truth.

Rank fusion orders evidence; it does not invent confidence. Do not mix incompatible scales. Expose rank, availability, and match reason.

Code

A small Reciprocal Rank Fusion implementation·python

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for rank, item in enumerate(ranking, start=1):
            scores[item] = scores.get(item, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

text = ["a", "b", "c"]
meaning = ["b", "d", "a"]
assert rrf([text, meaning])[0] in {"a", "b"}

Score scales disagree

A fusion score is not probability

A missing lane is a state

Evaluate with a query set

Code

External links

Exercise

Progress

Comments 0