Skip to content
C.W.K.
Stream
Lesson 01 of 05 · published

Keep the Table of Things That Did Not Work

~12 min · measurement, proxies, evidence, process

Level 0Wet Clay
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

A Day of Plausible Ideas, All Wrong

Before any of the measurement in this track existed, a full day went into building a mechanical detector for the defect the first track described. Every proxy tried was reasonable. Every one failed, and the failures are recorded in a table nobody is allowed to delete.

Thin content. Flag items whose sections fall below a length floor. Result: it flagged sixty-nine of the seventy-seven items that existed in the fragment-first format the scan could read and thousands of sections. It paints the corpus and ranks nothing.

A punctuation marker. Look for a specific construction that felt like a signature of the defect. Result: sixty-six of seventy-seven items, over fifteen hundred hits, and the hits were dominated by entirely legitimate uses.

A lexicon-free non-word detector. Clever: find words used in two grammatical patterns that only co-occur in malformed constructions, then filter by rarity. Result: thirty candidates, one true positive. Three percent, and the false positives were ordinary words.

An existing screening scan, reused as a ranker. Result: no correlation at all. Items scoring high on the scan averaged fewer real findings than items that had already been cleaned.

Why the Table Has to Be Kept

Not as an obituary. Every one of those ideas is what a competent person reaches for on day one, which means that without the record the next person spends the same day rebuilding one of them and arriving at the same number.

So each row has to carry what it measured, what the result was, and — this is the part people leave out — why it failed, in a sentence. "No correlation" is a result. "Dominated by legitimate uses, because the construction is also how the language forms ordinary compounds" is a reason, and a reason transfers to the next idea in a way a number does not.

The Line to Read the Table With

One important caveat, and it is easy to get backwards. This table is not evidence that mechanical detection is impossible. It is a record of proxies tried with nothing to validate against. Every one of them was a guess about what the defect looks like, evaluated by whether its output felt right.

The moment ground truth existed — a set of items scored by a reader — the same exercise produced something that landed. Which is the actual lesson: the proxies did not fail because the domain resists measurement. They failed because they were built before anything could tell them they were wrong.

Get ground truth before building a detector, not after. A detector built without it can only be evaluated by whether its output looks plausible, and plausible output is exactly what a wrong detector produces — which is why an entire day can end with four confident, useless instruments.

Code

The dead-ends table — kept, and read before proposing anything new·text
PROXY                          RESULT              WHY IT FAILED
-----------------------------  ------------------  ------------------
thin content under a length    69 of 77 items,     paints the corpus;
floor                          3231 sections       ranks nothing

a punctuation construction     66 of 77 items,     the construction is
that felt like a signature     1570 hits           also how ordinary
                                                   compounds are formed

lexicon-free nonword detector  30 candidates,      the pattern it keys
(word in two patterns that     1 true positive     on is shared by many
co-occur only when broken),    -> 3%               perfectly real words
rarity-filtered

an existing screening scan     no correlation      it fingerprints the
re-used as a ranker            at all              authoring path, not
                                                   the prose

WHAT SURVIVED AS MECHANICAL (worth a watchword line, NOT a worklist):
  a specific connective that occurs 3 times in the whole corpus,
  all 3 broken. presence-alone shape - natural writing never
  reaches for it.
  formal-register leaks: 22 hits across 14 items.

READ THIS TABLE AS: "proxies built with nothing to validate against"
NOT AS:             "mechanical detection is impossible here"

External links

Exercise

Pick a detector or lint rule your team relies on and find out what it was validated against. If the answer is "it seemed to catch real things," build a small ground-truth set — twenty items judged by hand — and measure the rule against it. Write the result down whichever way it comes out, because a rule that turns out to work is now a rule you can defend, and one that does not has been costing attention silently.
Hint
Twenty items is usually enough to distinguish a working rule from a useless one, because the failure modes in this space are dramatic rather than marginal. If the answer looks like a close call at twenty, that is itself informative: the rule is at best weak, and weak rules that fire often are net negative.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.