Curation Is the Quality Lever

"The engine wasn't wrong. It faithfully completed exactly what the corpus contained — and the corpus contained HTML tag soup."

The Junk Completions

Here's a war story with a clean lesson. A corpus was built from web content, and its phrase completions started coming back as nonsense: fragments like "data emotion playful img class …" — bits of HTML attributes, not prose. The completion feature looked broken. The instinct was to go patch the completion endpoint to filter out the junk. That instinct was wrong, and understanding why is the whole point of this lesson.

The Engine Was Behaving Correctly

Completion mines the corpus for how phrases actually continue. If the indexed text is riddled with raw HTML tags and attributes, then the honest continuations of many phrases are attribute fragments. The engine wasn't malfunctioning — it was doing its job perfectly on polluted input. Garbage in, garbage out, at the level of a whole engine. The bug wasn't in the retrieval; it was in what got fed to it.

Fix the Corpus, Not the Engine

So the fix belonged at the conversion seam, not the query path. The web content was re-captured through an extractor adapter that strips HTML tags into clean prose: pull a title from the frontmatter, remove the markup, mine prose out of structured data while blocking identifier-shaped keys, and enforce a minimum length so fragments don't survive. Now the snapshot holds prose, and every downstream path — search, completion, everything — mines clean text. One fix at the boundary healed every symptom at once, because they all shared the same polluted source.

Quality lives at the conversion boundary — clean there, never in the query path. When results are junky, the tempting fix is a filter at the point of use. Resist it: that filter treats one symptom while the pollution keeps flowing everywhere else. Fix the input at the adapter, into the snapshot, and every consumer of that corpus gets clean data for free. Curation is the highest-leverage place to improve a retrieval system — a clean corpus makes a mediocre engine look brilliant, and a dirty corpus makes a brilliant one look broken.

Clean Into the Snapshot, Never the Original

One critical constraint holds the whole thing together: the cleaning happens into the converted snapshot, and the chunk text stays an exact slice of that converted text. You never reach back and edit the original source files to clean them — that would violate the never-touch-the-source rule. The extractor reads the messy original, produces clean prose, and snapshots that. The original stays exactly as it was; the snapshot is the cleaned reading; the offsets and hashes all refer to the snapshot. Curation and 'originals stay put' coexist perfectly, because the cleaning lands in the captured layer, not the source.

Code

Clean at the adapter: extract prose into the snapshot, leave the source alone·python

def extract_prose(raw_html_ish: str) -> str:
    """A site-extract-style adapter: turn tag-polluted content into clean prose."""
    title = frontmatter_title(raw_html_ish)          # pull a heading from metadata
    body = strip_html_tags(raw_html_ish)             # remove <div class=...> soup
    prose_parts = []
    for key, value in mine_structured_text(raw_html_ish):
        if is_identifier_key(key):
            continue                                  # block 'class', 'data-emotion', etc.
        if len(value) >= 20:                          # prose floor: no fragment survives
            prose_parts.append(value)
    text = join_prose([title, body, *prose_parts])
    return text
    # This text is what gets snapshotted and chunked. The messy original on disk
    # is never edited — cleaning lands in the captured snapshot, not the source.

Exercise

Think of a time a system gave you bad output. Ask the diagnostic question: was the logic wrong, or was the input dirty? For a case where the input was dirty, decide where the fix belongs — at the point of use (a filter on the output) or at the input boundary (cleaning the data as it enters). List what would break if you filtered at the output instead of cleaning at the boundary.

Hint

Filtering at the output fixes exactly one consumer and has to be re-implemented for every other consumer of the same dirty data. Cleaning at the input boundary fixes every consumer at once, forever. If the same pollution shows up in multiple places, that's proof the fix belongs upstream, at the boundary.