A Seeded Reader Is Not a Scanner

The Invariant Looks Violated, and Is Not

"No quality scores, anywhere" and "a reading run that scores every item out of twenty" appear to be in direct conflict. Resolving it requires reading the original finding precisely rather than by its slogan.

What the invariant closed was that quality is not scanner-rankable. The evidence behind it was three surface metrics that turned out to fingerprint the authoring path rather than the prose — they told you which tool produced something, not whether it was any good. A seeded reader is not that instrument and does not have that failure mode.

The distinction that does the work: a scanner counts features of the text; a reader judges whether the text lands. The invariant forbids the first from ranking. It never said nothing may.

The Split That Keeps It Honest

Three responsibilities, deliberately separated, and each separation prevents a specific failure.

The engine draws the sample — seeded, reproducible, mechanical. A reader who picks their own passages lets bias in through the front door, and a score you cannot re-draw is a rumor rather than a measurement.

An uncontaminated session scores it — the only instrument that sees the defect at all, and only while it cannot reach the source.

The engine aggregates and ranks — as lifecycle fact, never verdict. The number goes on a worklist, not on a card as a grade.

The engine never judges; the reader never chooses its own sample. Either violation collapses the instrument back into the thing the invariant forbids.

What the Recorded Seed Is Actually For

Narrower than it looks, and one intuitive use is measurably wrong. It is not for re-reading the same passages after a repair: that was tried, and because a repair moves the indices the draw rides on, the same seed produced a completely different set. What the seed genuinely buys is re-running an unchanged corpus for a second reader. What it does not buy is reproducing a past run for audit: the corpus moves, the drawn positions move with it, and the language promises only its raw generator stream across versions — not every algorithm built on top of one. Auditing a past run means having persisted what that run actually drew.

Measuring a repair is a fresh draw's job — and that is the better question anyway. "Is this fixed" is what you want to know; "did it fix those ten" is the easier question and biased toward yes.

When an invariant appears to forbid something useful, re-read what it actually closed before either violating it or abandoning the idea. Most invariants are narrower than their slogans, and the narrow version usually permits the useful thing while still forbidding the thing that caused the rule.

Code

The draw — mechanical, seeded, and declaring its own population·python

import random

def draw(item, size=10, seed=None):
    """Pool = prose that carries clause order. That definition is
    part of the contract, because it changes what a low score MEANS.
    """
    pool = []
    for fragment in item.fragments(locale="target"):
        pool += fragment.body_paragraphs()      # in
        pool += fragment.callout_bodies()       # in
        pool += fragment.exercise_prompts()     # in
        pool += fragment.exercise_hints()       # in
        pool += fragment.section_descriptions() # in
    pool += item.track_descriptions()           # in
    pool += item.shell_descriptions()           # in
    pool = [p for p in pool if len(p) >= 40]    # a floor: shorter
                                                # than this carries
                                                # no clause order
        # EXCLUDED, deliberately:
        #   code bodies  - source-language text is CORRECT there,
        #                  and including them would score every
        #                  technical item badly for the wrong reason
        #   short labels - no clause order to judge

    seed = seed if seed is not None else random.randrange(2**31)
    rng = random.Random(seed)
    n = min(size, len(pool))
    return {
        "seed": seed,                  # recorded: a score you cannot
        "drawn": rng.sample(pool, n),  # re-draw is a rumor
        "pool_size": len(pool),
        "note": (f"scored on all {n} pooled passages"
                 if n < size else f"{n} of {len(pool)}"),
    }


# The `note` is not decoration. A total of 12 is a PERFECT score
# at 6 passages and a middling one at 10, and the reader must
# never have to guess which they are looking at.

Exercise

Find a review or audit in your process where the reviewer chooses what to look at. Replace the choosing with a seeded draw, record the seed, and write down the pool definition — what is eligible and what is excluded, with a reason for each exclusion. Then run it and compare the findings to the last self-chosen review. The difference is the bias you were previously buying.

Hint

Writing the pool definition is where the real work is, and it is usually the first time anyone has stated what the review is supposed to cover. Expect to discover that two people had different answers, which is more useful than anything the run itself produces.

The Invariant Looks Violated, and Is Not

The Split That Keeps It Honest

What the Recorded Seed Is Actually For

Code

External links

Exercise

Progress

Comments 0