Keep Search Failure Inside Search

Do not open a transaction across a network wait

Opening a SQLite write transaction before requesting embeddings keeps the lock alive while an external provider stalls. Reindexing and metadata updates wait, and one timeout spreads into a database outage.

Reverse the order. Read mirror bytes and hash outside the transaction, compute the embedding, then upsert in a short transaction. Recheck source hash before commit to reject stale work.

Do not turn one item failure into batch failure

A long CJK note may exceed a token limit, or malformed text may break one tokenizer call. Rolling back the whole batch lets one row prevent hundreds of valid items from progressing.

Per-item fallback records a typed failure and commits the rest. Deterministic chunking may handle long notes, or only the vector lane can remain unavailable while FTS continues. Expose failure but narrow its scope.

The index must support purge and reingest

Patching old vectors through a schema or model migration can mix rows from different versions. Build a versioned replacement and atomically switch the pointer when ready.

During rebuild, live truth and ledger remain healthy and available lanes continue. A derived-index rebuild that blocks source writes has taken authority hostage.

Report health per lane

One green or red search badge hides partial failure. Separate FTS readiness, vector degradation, queue lag, and last indexed ledger revision.

After recovery, verify replayed failed items, stale-hash rejection, and a distinctive result. A live process and an index current with source are different proofs.

Derived-search failure must not pull source or another lane down with it. Keep external computation outside locks and reduce blast radius with per-item failure and per-lane health.

Code

Compute externally, then commit briefly·python

import sqlite3

content_hash = "abc123"
embedding = [0.1, 0.2, 0.3]  # computed before BEGIN

db = sqlite3.connect(":memory:")
db.execute("CREATE TABLE vectors(path TEXT PRIMARY KEY, hash TEXT, value TEXT)")
with db:
    db.execute(
        "INSERT INTO vectors VALUES (?, ?, ?)",
        ("memory/a.md", content_hash, repr(embedding)),
    )
assert db.execute("SELECT hash FROM vectors").fetchone()[0] == content_hash

Do not open a transaction across a network wait

Do not turn one item failure into batch failure

The index must support purge and reingest

Report health per lane

Code

External links

Exercise

Progress

Comments 0