The Statistics That Lied: Count What Landed

The Half of the Bug That Hid the Other Half

Truncation degraded the corpus. That is a bug, and a fixable one. What made it survive was the second half: the statistics reported the material that had been dropped, so every instrument agreed the run had been healthy.

The mechanism is almost too simple to notice in review. Each counter was incremented while the section was being gathered. The truncation happened later, in a different function, operating on a joined string that no longer knew what it was made of. Nothing was wrong with either piece of code in isolation.

Intent Versus Reality

A counter incremented at collection time answers "how much did we intend to include?" A counter derived after assembly answers "how much is actually in the thing we sent?" Both are legitimate questions and they usually agree, which is precisely why the distinction is invisible until the day they diverge — and the day they diverge is the day you most need the second one.

The rule that falls out is easy to apply and worth applying broadly: count on the artifact, not on the way to it. If a number describes what a payload contains, derive it from the payload after it is final. If it describes an attempt, name it as an attempt.

Disclose the Gap Rather Than Hiding It

Once counting moves after assembly, a second opportunity appears: the difference between what was gathered and what landed is itself a useful number. Reporting how many items were dropped turns a silent degradation into a visible signal, and it gives you a threshold that actually means something — a handful of dropped headlines is a healthy day at capacity, hundreds means the budget no longer fits the reader.

When the Output Is the Only Honest Instrument

It is worth sitting with how this was found. Not an alert, not a test, not a dashboard — the generated brief mentioned in passing that a section it expected was absent. The model was reading the corpus, so it was the only observer looking at the artifact rather than at the process that produced it.

That is a general and slightly uncomfortable lesson about instrumenting a pipeline. Metrics collected at each stage describe the stages. If nothing measures the final artifact, then a defect introduced between stages is invisible to all of them, no matter how many you add.

A metric that describes an artifact must be computed from that artifact. Counting on the way there measures intent, and intent and reality agree right up until the moment something goes wrong — at which point every dashboard will confidently tell you nothing did.

Code

Counting intent versus counting the artifact, and disclosing the gap·python

# BEFORE: counted while gathering -- these describe INTENT.
def build_corpus(con):
    stats = {"extracts": 0, "samples": 0, "articles": 0}

    for sample in latest_trend_samples(con):
        sample_lines.append(render(sample))
        stats["samples"] += 1          # <- counted here...

    corpus = header + shelves + extracts + samples
    return corpus[:CORPUS_MAX_CHARS], stats   # <- ...dropped here.
    # The run reports 4 samples. The corpus contains none. Nothing in
    # the pipeline disagrees, because every stage measured its own
    # stage and the defect happened BETWEEN two of them.


# AFTER: counted from the finished artifact -- these describe REALITY.
def build_corpus(con):
    corpus, included = _assemble_within_budget(con)   # returns what LANDED

    stats = {
        "samples":  len(included.samples),
        "extracts": len(included.extracts),
        "articles": len(included.headlines),
        # The gap is itself the useful number: a handful means a healthy
        # day at capacity, hundreds means the cap no longer fits.
        "headlines_dropped": included.dropped,
        "corpus_chars": len(corpus),
    }
    return corpus, stats

Exercise

Pick a metric your team trusts that describes the contents of something produced — items in a report, records in an export, files in a bundle. Find where it is computed. If it is anywhere other than on the finished artifact, construct the scenario where the two would differ, and check whether anything else in the system would notice.

Hint

Look for the counter being incremented inside a loop that builds a list, rather than being computed as a length at the end. The loop version is nearly always measuring intent, and it survives because the two agree in every test — tests rarely exercise the capacity limit that makes them diverge.

The Half of the Bug That Hid the Other Half

Intent Versus Reality

Disclose the Gap Rather Than Hiding It

When the Output Is the Only Honest Instrument

Code

External links

Exercise

Progress

Comments 0