Skip to content
C.W.K.
Stream
Lesson 03 of 05 · published

Put the Caveat in the Payload

~12 min · api-design, payload, disclosure, read-model

Level 0Raw Ore
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
A caveat in the documentation reaches one reader. A caveat in the payload reaches every consumer, forever.

Four places you could put the disclosure

Everyone agrees the window should be disclosed. Where it goes is the actual decision, and the options are not equivalent.

In the docs. Free, and reaches only the people who read the docs at the moment they need the caveat. Which is, in practice, the person who wrote them.

In the frontend component. Better — the reader sees it. But it lives in one component. The command line does not have it. A delegation brief quoting the number does not have it. The next surface someone builds does not have it, and will not know it was supposed to.

Computed on demand by each consumer. Now every consumer needs the full history to work out the span, which means every consumer re-implements the same query and one of them gets it subtly wrong.

On the row. The read model computes it once, and every consumer of that row — every surface, every export, every brief, every future integration nobody has thought of — receives it whether they asked or not.

Disclosure travels with data, not with documentation. If a number cannot be interpreted without a qualifier, the qualifier is part of the number and belongs in the same payload. Any other placement is a bet that every future consumer will independently discover the caveat — and that bet loses on the first integration nobody told you about.

What the row actually carries

Each read-model row holds the value and its data date, the percentile, the count of points behind it, the date the history starts, and the span in years. None of those is optional in the sense that matters: drop any one and the rest overstate what is known.

It also carries the series' own minimum, median and maximum, which do quiet work — a percentile tells you the rank, the distribution tells you the shape. The live CAPE row is the case in point: value 43.19 at the 99.7th percentile, historical median 16.61, historical maximum 44.20. The rank says near the top. The median says about two and a half times the typical reading. The maximum says and not far below the highest ever recorded. Rank alone expresses none of that.

Computed on read, not on write

One structural point that makes this affordable. The percentile and the window are computed on read from the append-only series rather than stored alongside each snapshot.

That is the correct direction, and not only for storage reasons. A percentile is not a property of the observation — it is a property of the observation relative to everything else known so far, and that changes every time a new row arrives. Storing it at write time would freeze a relationship that is still moving, and you would get a table full of percentiles that were true on the day they were computed. Deriving on read means the answer is always relative to the current history, which is the only version of the question anyone is actually asking.

The cost is real and it was measured. Recomputing over the whole series on every request is not free, and the answer was not to precompute — it was to cache on the engine's write counter. The engine is the single writer, so counting writes is cheap and never misses one: the cached read model rebuilds when a write has been attempted, and otherwise not at all. A manual refresh is still instantly visible, and a quiet dashboard stops rebuilding tens of thousands of rows on every page load.

Code

One gauge row as served — the number, and everything needed to read it·json
{
  "gauge": "cape",
  "market": "us",
  "series": "SHILLER",
  "value": 43.1884,
  "data_date": "2026-08-06",
  "fetched_at": "2026-08-07T22:11:13.840312+00:00",

  "percentile": 99.7,
  "points": 1748,
  "first_date": "1881-01-01",
  "window_years": 145.6,
  "hist_min": 4.7842,
  "hist_median": 16.6063,
  "hist_max": 44.1979,

  "source": "derived:shiller-ext P_real/E10trend(4%yr real, cal 2026-02)"
}

// Seven context fields for one value. The CLI gets them. The
// dashboard gets them. A delegation brief quoting this row gets
// them. Nobody has to know to ask.
//
// Note the source string: it names the EXTENSION method and its
// assumption, because months past the sheet's freeze are estimates.
// A number that was estimated says so where it is stored.

External links

Exercise

Find an API response in your systems that returns a number requiring a caveat — a rate over an unstated period, an average over an unstated population, a status computed with an unstated threshold. Add the qualifying field to the payload, then grep your consumers for how many were displaying that number without it. The count is usually higher than expected, because each consumer was written by someone who had the caveat in their head at the time.
Hint
A good test: could a new consumer, reading only your response schema, render this number correctly with no other knowledge? If not, the schema is under-specified, and every correct consumer today is correct by accident of who wrote it.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.