The Ruler Is Part of the Measurement

p12 over three years and p12 over a century are different claims, and a surface that prints them identically is asserting a commensurability it does not have.

Back to the reference needles

The first lesson of this quest described the physical assay: rub the sample, read the streak, compare against needles of known purity. Everything rested on that last step, and one detail was left hanging deliberately — the answer is exactly as good as the reference set.

An assayer with three needles can tell you roughly what you have. An assayer with twenty can tell you much more precisely. Both hand you a confident-sounding answer. If neither mentions how many needles they own, the two answers look identical on paper, and you have no way to tell which one to lean on.

A percentile is the software version of the needle set, and it has exactly the same problem: the number carries no visible signal about how much history stands behind it. "99.7th percentile" reads the same whether it was computed over 145 years or over eleven weeks.

The frame, planted a second time

This is where the quest's central sentence earns its keep. A measurement that cannot state its own limits is not a measurement — it is a claim wearing a measurement's clothes.

The first track applied it to a date that did not say whose calendar it used. This track applies it to the most respectable-looking number on the entire dashboard. A percentile feels like the rigorous option. It is the thing you reach for precisely because it is more honest than a threshold. And it can be dishonest in a subtler way, because the dishonesty is not in the number at all — it is in what the number quietly implies about its own foundations.

Every statistic has a denominator, and half of them do not show it. Percent, percentile, rank, average, rate — each one is a compression, and each compression discards the thing that would let a reader calibrate it. When you present a compressed number, present the shape of what was compressed alongside it, or you have handed over a conclusion while keeping the evidence.

Two thresholds in the code, doing different jobs

The implementation defends this on two fronts. There is a floor: below a minimum number of points, the percentile function returns nothing at all rather than a number computed from too little. And there is a disclosure: every row that does carry a percentile also carries the date its history starts and how many years that spans.

The floor handles the case where the answer would be meaningless. The disclosure handles the much larger and more dangerous case where the answer is meaningful but less meaningful than it appears — which is most of them, most of the time, in almost every system that reports a percentile.

Why a floor is not enough on its own. A minimum point count draws one line and calls everything above it fine. But there is no single count that makes a series comparable to a different series with ten times the history. The floor prevents nonsense; only the disclosure prevents a false equivalence between two numbers that both cleared it.

Code

The same percentile, four very different claims·text

GAUGE            PERCENTILE   WINDOW      WHAT IT ACTUALLY SAYS
--------------   ----------   ---------   ---------------------------
CAPE (US)             99.7    145.6 yr    higher than all but ~0.3%
                                          of readings since the 1880s

Buffett (US)          99.0     74.3 yr    higher than all but ~1% of
                                          the post-war period this
                                          series covers -- NOT "the
                                          highest", which p99 is not

HY credit spread      10.3      3.0 yr    below ~90% of readings in
                                          the window a data licence
                                          permits -- and that window
                                          is three years, not ninety

composite             None      0.0 yr    3 points so far, so it has
                                          earned no percentile at all

# All four could render as a number between 0 and 100 in the same
# font. Only one of them is a statement about a century.
#
# Note the wording of rows 2 and 3. p99 is not "the highest" and
# p10 is not "the lowest" -- roughly 1% and 10% of history sit
# beyond them. A table that teaches you to state what you know has
# no business rounding its own percentiles up to superlatives.

Exercise

Find a percentile, rank, or 'above average' claim in a product you use — a fitness app, an analytics tool, a benchmark report. Try to determine the reference population and the window from the interface alone. Note how long it takes and whether you succeed. Then ask what would change in your reading of the number if the window turned out to be ten times shorter than you assumed.

Hint

Comparative claims almost never disclose their reference set, because disclosing it usually makes the claim less impressive. 'Top 10% of runners' is a very different statement depending on whether the population is all humans or the people who bought this watch.

Back to the reference needles

The frame, planted a second time

Two thresholds in the code, doing different jobs

Code

External links

Exercise

Progress

Comments 0