Warming Up Is a State, Not a Smaller Answer

"A partial result is a WRONG result, not a small one."

The second instance of the same shape

The truncation bug had a sibling on the same day, and it did not come from a client library at all. The concentration pass needs share counts for every company in its universe. On a cold start the cache is empty, and the counts arrive per company. The first real run had counts for 399 of an intended 500 — and it computed a concentration figure from those 399 and published it.

The result was plausible. It was also wrong in a way that mattered, and it went wrong in a specific direction: the megacaps' counts had not arrived yet, so they were missing from the denominator and from the top ten, and a machinery manufacturer took a slot it never belonged in.

Same lesson, different mechanism. And that is what elevates it from a bug to a class: two independent code paths, one day, produced the same failure, and both looked entirely plausible.

The fix is a third state

The instinct is to fix each cause — follow the cursor properly, warm the cache faster. Both were done. But the durable fix is structural: the pass now has a state it can be in, and "incomplete" is that state rather than a value.

When the universe is not full, the pass stores a progress record — how many it has, how many it needs, how many share fetches failed, the full coverage breakdown — and it publishes no concentration figure at all. Not a provisional one, not one with an asterisk. None.

That is the design decision worth taking away. A number with a caveat still gets read as a number; readers do not weight caveats the way authors imagine they do. The only reliable way to prevent a partial result from being consumed as a complete one is for there to be no number to consume.

Say which state you are in, and let one of the states have no answer. Most systems have exactly two states: an answer, or an error. The missing third is not yet — legitimately operating, nothing wrong, no answer available. Systems without it are forced to express "not yet" as either a wrong answer or a false alarm, and they usually pick the wrong answer.

Coverage as a first-class output

The progress record is not a boolean. It carries the full breakdown: the size of the common-stock universe, how many of those had prices, how many candidates were considered, how many companies remained after deduplication, how many had share counts, how many were fetched this pass, how many are still missing, and how many fetches failed.

Eight numbers to explain the absence of one. That ratio is right. When a system declines to answer, the most useful thing it can produce is a precise account of what it is waiting for — because the alternative is somebody logging into a database at midnight to work out the same thing by hand.

The sizing decision that follows. Once "partial" is a state that publishes nothing, a cold start that dribbles in over several days means several days of no answer. So the per-pass fetch limit was sized to cover the whole candidate pool in one pass, and the fetches were made concurrent to make that affordable. The honest state made the performance requirement visible — before, the slow warm-up was hidden behind a wrong number that looked fine.

The second instance of the same shape

The fix is a third state

Coverage as a first-class output

Code

External links

Exercise

Progress

Comments 0