One Question, Two Search Lanes

Lexical retrieval catches names

FTS5 excels at exact words, path fragments, proper names, and quoted phrases. When recalling a failure code or product name, exact matching is better than a vector returning a semantically broad sentence first.

Its reasoning is inspectable: matched tokens and fields can be shown, and rebuild from text is deterministic. Its blind spot appears when equivalent meaning uses entirely different language.

Semantic retrieval joins different wording

A vector lane can connect 'cost paid every time' with a note titled 'session boot token weight.' That makes it useful for duplicate candidates and Related panels.

The model, chunking, and pooling remain hidden inside the score. High similarity does not prove two memories are equivalent, and proper names or short code tokens may perform worse than lexical search.

Expose lanes before fusion

A hybrid-only list erases which lane found what. Operators cannot tell whether FTS is broken or embeddings are stale, and users may mistake a semantic guess for an exact hit.

Offer All, Text, and Meaning views, and keep lexical plus vector rank on fused cards. Deduplicate the item without deleting its evidence.

Defend authorization before and after fusion

Constrain candidate fetch by actor scope, then check policy again before returning the fused list. Indexing every stewarded vault must never place a super fragment into an unauthorized context.

You may report that policy reduced results, but protected titles and snippets remain hidden. Do not evaluate retrieval relevance and authorization correctness as one metric.

Lexical and semantic search are not rival models; they are observations with different blind spots. Fuse order without erasing lane provenance or permission.

Code

An exact-term lane with SQLite FTS5·sql

CREATE VIRTUAL TABLE notes USING fts5(path, body);
INSERT INTO notes(path, body) VALUES
  ('memory/weight.md', 'session boot token weight'),
  ('memory/restore.md', 'forward write recovery');
SELECT path, bm25(notes) AS rank
FROM notes
WHERE notes MATCH '"session boot"'
ORDER BY rank;

Lexical retrieval catches names

Semantic retrieval joins different wording

Expose lanes before fusion

Defend authorization before and after fusion

Code

External links

Exercise

Progress

Comments 0