🏮

Lantern Quest

Updated: 2026-07-16

Build a corpus engine, not a RAG-in-a-box

The origin story of Lantern — Dad's own corpus engine, the retrieval sibling of the image engine Ember and the music engine Bonfire. Why you build a deterministic engine that finds your own words again instead of renting a RAG service, and what every architectural choice teaches about owning the way you remember what you wrote.

8 tracks · 33 lessons · ~6h · difficulty: intermediate-to-advanced

Level 0Unlit Wick

0 XP0/33 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Lantern is not a RAG chat app, not a vector database with a bow on it, not a second brain. It is an independent engine that owns corpus registration, ingestion, retrieval, and preservation over a decade of scattered writing — with a light web console as its first client. This quest walks that engine end to end: the name as thesis (a lantern is carried fire — it finds things in the dark and never burns the house down), the core bet (the gathering problem is solved by a registry, not by moving files), the anti-hype spine (deterministic before judgment — no model in the retrieval path, so search works with the brain down), the durability obligation (citable or nothing — content-addressed chunk ids that survive re-index, machine moves, and database rebuilds), the two retrieval regimes (hybrid search versus keystroke completion — different physics, one engine), the ingest boundary (referenced versus captured corpora, cleaning at conversion not in place), the evidence layer (pinned passages, claims, deterministic revalidation), and the closing discipline of owning your own corpus. It is the corpus engine telling its own origin story — conceptual open-sourcing, not a repo you clone.

Tracks

01🏮The Lantern
0/4 lessons
What Lantern is, why its name is the thesis, and why it exists
Before any retrieval mechanics, the orientation. Lantern is a corpus engine — not a RAG chat app, not a vector database with a bow on it, not a second brain. This track decodes the name (a lantern is carried fire), places Lantern in the sibling family of engines, tells the two-Pippa origin story, and frames the one problem the whole engine exists to solve: a decade of writing scattered across a dozen homes.
Lesson list (4)
Quiz · 4 questions→
02🗂️Registry, Not Move
0/4 lessons
The core bet — gather by declaration, and never touch a file
The founding trap was moving files to gather them. This track builds the alternative in full: a declarative manifest that names where writing already lives, an idempotent walk that reads and never rewrites, the two corpus classes (referenced and captured), and the one non-disposable store — the snapshot layer that preserves captured text forever.
Lesson list (4)
Quiz · 4 questions→
03⚙️Deterministic First
0/4 lessons
No model in the path — keep the finding boring, do the judging above
The anti-hype spine of the whole engine. Everyone bolts a language model onto search and calls the model's opinions 'retrieval'. Lantern refuses: no model call anywhere in the retrieval path, so search works with the brain unreachable. This track builds that discipline — why the model goes above the API, what the determinism guarantee buys you, why Lantern is not a RAG chat, and why a degraded search must announce itself instead of quietly lying.
Lesson list (4)
Quiz · 4 questions→
04🔖Citable or Nothing
0/4 lessons
Content-addressed chunk ids, so a citation never dies on the next re-index
The one forward-looking obligation the engine carries from day one. A citation that breaks when you rebuild the index is worthless. This track builds the durability layer: why fragile citations are the default failure, how a content-addressed chunk id survives re-index and machine moves, why chunking profiles must be immutable, and how a fully disposable index can still hold citations that never break.
Lesson list (4)
Quiz · 4 questions→
05🔍Two Regimes, One Engine
0/5 lessons
Hybrid search and keystroke completion are different physics
The retrieval heart of the engine, in full. Semantic search and phrase completion look similar and are nothing alike — different latency budgets, different mechanics, different failure modes. This track builds both: why the API refuses to merge them, how full-text BM25 ranking works, how dense vectors and Reciprocal Rank Fusion combine, why reranking is a deepening pass and not the default, and how completion hits keystroke speed with no network at all.
Lesson list (5)
Quiz · 5 questions→
06📥The Ingest Boundary
0/4 lessons
How text gets in — chunk it, adapt it, clean it at the seam
Where raw sources become searchable chunks. This track builds the ingest layer: the chunk as the atom of retrieval and why a whole document is the smallest unit of mutation, the simple referenced adapters where the file is the text, the captured adapters that convert PDFs and HTML-polluted content into clean prose, and the hard-won rule that cleaning happens at the conversion seam — never by editing the originals in place.
Lesson list (4)
Quiz · 4 questions→
07🧾The Evidence Layer
0/4 lessons
Pinned passages, stanced claims, and deterministic revalidation
The product ceiling, built on the stable chunk ids from earlier. Retrieval finds text; evidence records what you concluded from it. This track builds that layer: why a pinned passage copies the exact text instead of trusting a pointer, how claims and stanced citations keep judgment above the API, how contradictions become typed links between claims, and how revalidation stays honest using nothing but string equality.
Lesson list (4)
Quiz · 4 questions→
08🕯️Own Your Corpus
0/4 lessons
Three surfaces, engine before clients, and why you build instead of rent
The closing discipline. Three retrieval surfaces coexist in this family and none is allowed to absorb another — resisting the 'unify all retrieval' instinct is an architectural act. This track closes the quest: the boundary table that keeps the surfaces separate, the engine-first build order that made clients easy, the honest case for building a corpus engine instead of renting one, and what it actually means to own the way you find your own words again.
Lesson list (4)
Quiz · 4 questions→

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.

Tracks

01🏮The Lantern

02🗂️Registry, Not Move

03⚙️Deterministic First

04🔖Citable or Nothing

05🔍Two Regimes, One Engine

06📥The Ingest Boundary

07🧾The Evidence Layer

08🕯️Own Your Corpus