Where Keyword Search Breaks

The keyword gap

Classic search engines and SQL LIKE queries match character sequences. They never look at meaning. Search for "refund policy for yearly plans" and you will miss every document that uses "annual subscription cancellation" — same intent, zero overlapping words.

Stemming, lemmatization, and synonym dictionaries patch the easiest cases (running → run, cheap ↔ inexpensive) but they collapse the moment you cross a real semantic gap:

"how to fix a memory leak" misses "debugging garbage-collection issues"
"the bank refused our loan" matches articles about river banks just as eagerly
"PyTorch slow on M3" misses "accelerate Apple Silicon training with MLX"

What we want instead

We want a representation where meaning is geometry. Two passages that mean the same thing should land near each other in some space, even if they share no words. That representation is what an embedding gives us, and the rest of this quest is about learning to ingest, store, search, and trust them in production.

Sanity-check the gap on your own data

Before you adopt vector search, prove the gap exists in your corpus. Sample 30 real user queries and 30 documents from your knowledge base, then count how many query-document pairs are obvious matches semantically but have zero token overlap. If the count is high, vector search is justified. If it is near zero, BM25 and a synonym list will probably do the job for a fraction of the cost.

Code

Show the gap with a one-liner·python

from collections import Counter

query = 'refund policy for yearly plans'
doc   = 'how to cancel an annual subscription'

def token_overlap(a: str, b: str) -> int:
    a_tokens = Counter(a.lower().split())
    b_tokens = Counter(b.lower().split())
    return sum((a_tokens & b_tokens).values())

print(token_overlap(query, doc))   # 0 — keyword search would miss this entirely

Audit your own corpus·python

import csv
from pathlib import Path

pairs = csv.DictReader(Path('audit_pairs.csv').open())
zero_overlap = 0
for row in pairs:
    if token_overlap(row['query'], row['doc']) == 0 and row['relevant'] == 'yes':
        zero_overlap += 1
print(f'{zero_overlap} relevant pairs that keyword search cannot match')

Where Keyword Search Breaks

The keyword gap

What we want instead

Sanity-check the gap on your own data

Code

External links

Exercise

Progress

Comments 0