One Loop, Three Markets, One Window

The obvious design here is three schedulers. The right one is one, and the reason is a property of the data rather than of the code.

The problem, stated the tempting way

Three markets in three timezones. New York closes, then Seoul opens, then Tokyo, and none of them agree on what today is. The instinctive design writes itself: one scheduled job per market, each firing after its own close, each with its own holiday calendar.

Now count what you have signed up for. Three schedules to keep in sync as daylight saving shifts one of them twice a year. Three holiday calendars, two of which you will get wrong at least once. Three failure surfaces. Three places to look when a number did not update. And a new class of question that did not exist before: what happens when two of them overlap?

Why one window is enough

The unlock is not clever scheduling. It is a property established elsewhere in the design: each snapshot row carries its own data-date, asked of the provider. Once that is true, the run time stops being load-bearing.

One pass at one quiet hour asks every source for its latest reading. A market that has already closed contributes today's close. A market that is mid-session, or has not opened, contributes its previous close — correctly labelled with that previous date, because the label comes from the market rather than from the clock the loop happens to be running on. The append-only store deduplicates naturally: tomorrow's pass sees the same row for a market that has not moved, and a market that has moved gets a new dated row.

So the loop does not need to know when any market closes. It only needs to run at an hour when asking is useful, which is a much weaker requirement — and one that a single number in a config file satisfies.

Make the data carry the fact, and the schedule stops mattering. A great deal of scheduling complexity exists to make sure a job runs at the moment when the answer will be right. If each answer states which moment it belongs to, the job can run whenever, and an entire category of timing bugs becomes unreachable.

In-process, and never a second one

The loop is a single asyncio task inside the serving process — not a cron entry, not a separate service. That is family doctrine, and it buys three concrete things: the loop shares the engine's single writer so there is no cross-process write coordination; it can read live application state; and it cannot drift out of sync with the code it schedules, because it ships in the same process.

The maintenance slot rides the same pass: after the refresh, the database backup rotation runs, then the log size cap. Ordering matters and is deliberate — backups run after the refresh so the day's rows are inside the day's backup.

Failures must not kill the loop. Each pass writes its own summary into stored state, and neither the refresh nor the maintenance step raises out of the loop. A crashed loop is the worst outcome available: it takes down tomorrow's data too, and it does it silently. A survived-but-failed pass shows up as a number that stopped advancing, which is visible in the one place a user is already looking.

The problem, stated the tempting way

Why one window is enough

In-process, and never a second one

Code

External links

Exercise

Progress

Comments 0