Skip to content
C.W.K.
Stream
Lesson 03 of 05 · published

A Source That Stopped Updating

~13 min · data-sources, extension, assumptions, validation

Level 0Raw Ore
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
The dataset that made a 145-year window possible stopped being updated. The gauge did not stop.

The problem

The long US valuation series exists because an academic assembled it and published it — price, earnings and inflation inputs reaching back to the 1880s. That single artifact is why one gauge on this dashboard can speak about a century and a half.

And it froze. The published sheet stops at a date several years in the past, on retirement. The inputs it drew on are still published, by public sources, every month. So the choice is: end the series at the freeze, or extend it — and extending a scholarly dataset with your own arithmetic is exactly the kind of thing that turns into a fabricated number if you do it casually.

Three things that make the extension honest

Recompute, then validate against the source's own answer. Rather than continuing from the sheet's final published value, the pipeline parses the raw inputs and recomputes the metric itself — then gates ingestion on the recomputation staying within a small drift threshold of the sheet's own published column, over the overlapping period.

Be exact about what that step is for, because the obvious reading is wrong and the real purpose is the better lesson. It is not a method-reproduction proof, because the recomputed value is never used: the pipeline stores and extends from the sheet's own published column, and the recomputation exists only to produce a drift figure and raise on it. Nothing downstream depends on it.

What it actually is, in the code's own words, is a canary against a format change — the sheet is a spreadsheet whose columns can move, and a silent reshuffle would feed the most important gauge on the dashboard garbage that still parses. So the check is not "prove your reimplementation is right." It is "compute the same quantity a second, independent way, and refuse to proceed if the two disagree." That is a cheaper and more general instrument than it first appears: you do not need the second computation's output for it to be worth writing, only its disagreement.

Extend from live public series where you can, and name the one place you cannot. The price and inflation inputs for months past the freeze are real observations from currently-published series. The ten-year real-earnings denominator is not: no public series carries it, so it is carried forward on a stated trend — a fixed annual real growth rate, calibrated against a known recent print. So the extension is two-thirds observation and one-third projection, and the honest move is not to hide the third. It is stated in the source string of every row it produces, which is where the next point picks up.

Put the assumption in the source string. The extension needs one estimate for a slow-moving component, calibrated against a known recent print. That assumption is written into the source string of every extended row, so a row read years from now announces that it was extended and on what basis.

Reimplementing someone's method requires proving you reproduced it. Any time you rebuild a published metric from raw inputs, you have made an interpretation. Overlap the original and check agreement before you trust a single extended point — and gate on it, so a future change to your implementation that silently diverges fails rather than ships.

What the source string buys you

Look at what a reader can determine from the stored row alone: the method is derived rather than reported, the extension is in use, the assumption's magnitude, and the date it was calibrated against. That is enough to decide how much weight to put on the value, without reading any documentation.

Compare it to the alternative, which is what most systems do: a source field saying the vendor's name, and a footnote in a wiki that the person reading the number in two years will never see.

Estimated rows and measured rows share a series, and that is fine. The series is continuous — measured through the freeze, extended after it — and every row states which it is. The mistake would be either purism (truncate the series and lose the recent end of a 145-year window) or laundering (extend silently so the whole series reads as measured). Marking each row is what lets both halves coexist without either misleading.

Code

The extension, and what every row it writes announces·text
SERIES: US cyclically adjusted P/E

  1881 .. freeze     from the published dataset
                     -> recomputed from raw P / E / CPI inputs
                     -> INGEST GATED on staying within a small
                        drift threshold vs the sheet's own column
                     -> if the recomputation disagrees, the
                        pipeline stops rather than ships

  freeze .. today    from currently-published index + inflation
                     series, with ONE stated assumption for the
                     slow-moving component, calibrated against a
                     known recent print

AND THE ROW SAYS SO. Live source string on a served row:

  "derived:shiller-ext P_real/E10trend(4%yr real, cal 2026-02)"

# derived    -> not reported by a vendor; computed here
# shiller-ext-> the extension is in use for this row
# 4%yr real  -> the assumption, with its magnitude
# cal 2026-02-> what it was calibrated against
#
# Four facts, in one string, travelling with the value forever.

External links

Exercise

Find a metric your systems derive from an external source and check whether a stored row can tell you three things: that it was derived rather than reported, which method version produced it, and what assumptions that method makes. If the source field only names the vendor, add the method and its assumptions — and note how many rows already in your store can no longer be characterized at all.
Hint
The failure this prevents is not being wrong today; it is being uninterpretable in two years, when the method has changed twice and nobody can tell which rows came from which version. A source string that names the method is a cheap version stamp on every value you ever compute.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.