Sanity-Check Against a Fact You Already Have

"Sanity-check any ranking against a fact you already know."

Two kinds of check, and only one of them works here

Validation asks whether the data is well-formed: right types, right ranges, no nulls where nulls are forbidden, counts that are plausible. It is internal — everything it needs is inside the response.

Verification asks whether the data is right, and it is necessarily external. It requires a fact from outside the system to compare against.

Almost every codebase has a lot of the first and almost none of the second, because validation is cheap and generic and can be generated from a schema, while verification requires knowing something about the domain and writing it down. And the truncation bug is precisely a defect that only verification can see: 1,083 alphabetically-ordered tickers is a perfectly valid response, and the roster it produced is wrong in a way no schema can express.

The cheapest verification: a known member

You do not need a comprehensive oracle. For a ranking, one known member is usually enough, and the criterion for choosing it is stability: something that must be in the set for reasons that will not change month to month.

For a list of the largest US companies, a handful of megacaps qualify — not because their exact rank is predictable, but because their presence is. Any list of the top ten American companies by market value that omits several of the most valuable companies in the world is broken, and it is broken regardless of what happened in the market that week.

One known member beats a hundred type checks. For any operation that produces a set, name a member that must be present and assert it. It costs one line, it catches the entire family of silent-truncation failures, and it is the only check that can catch the case where every part of the response is valid and the whole is wrong.

Choose the fact so it fails for the right reason

The craft is in picking a fact that is stable enough not to fire spuriously, but specific enough to catch a real defect.

Too specific — asserting an exact rank, or an exact market value — and the check fails every time the world moves, which trains everyone to ignore it. An assertion people have learned to skip is worse than no assertion, because it occupies the slot where a real one would go.

Too loose — asserting only that the list is non-empty, or that it has ten entries — and it passes happily on the truncated data, which is exactly what happened.

The middle is membership without ordering: these particular names must appear somewhere in the result. Stable across market moves, and violated immediately by any truncation, filter error, or universe misconfiguration.

The domain-knowledge cost is the point. This kind of check cannot be auto-generated, because it encodes something a human knows about the subject. That is not a weakness — it is the only mechanism by which domain knowledge gets into your test suite at all. Every time you fix a bug that no schema could have caught, you have learned a fact worth asserting.

Code

Validation, verification, and the gap between them·python

# VALIDATION -- internal. All of this passed on the truncated data.
assert response.status_code == 200
assert isinstance(payload["results"], list)
assert all("ticker" in row for row in payload["results"])
assert 500 < len(symbols) < 20_000     # 1,083 sits happily inside


# VERIFICATION -- external. This is the one that fails.
#
# Membership, not ordering: these must APPEAR, at any rank.
# Stable across market moves; violated instantly by truncation,
# a broken filter, or a misconfigured universe.
ANCHORS = {"MSFT", "NVDA", "GOOGL"}  # illustrative megacaps, all
                                     # past C -- so this set fires
                                     # on the A-C truncation itself

missing = ANCHORS - set(symbols)
if missing:
    raise RuntimeError(
        f"universe is missing known megacaps: {sorted(missing)} "
        f"(got {len(symbols)} symbols, "
        f"first={symbols[0]}, last={symbols[-1]})")

# The error message names the two facts that diagnose it instantly:
# how many came back, and where the list stops alphabetically.

Exercise

For one dataset your systems produce, write down a fact you know about it that is not derivable from the data itself — a member that must be present, a total that must be within a range, a relationship between two fields that must hold. Turn it into an assertion. Then ask why it was not already there; the answer is almost always that nobody had yet been burned in a way that made the fact feel worth writing down.

Hint

Good anchors are boring and famous: the biggest customer must be in the customer list, the primary region must appear in the region breakdown, yesterday's known total must be within a few percent of today's. Boring is the point — an anchor that could plausibly change is an anchor that will page you at 3am for no reason.

Two kinds of check, and only one of them works here

The cheapest verification: a known member

Choose the fact so it fails for the right reason

Code

External links

Exercise

Progress

Comments 0