A Source Is a Row, Not a Code Path

The Decision That Shapes Everything Downstream

An aggregator has to answer one question before any others: how does the outside world get in? There are two families of answer. In the first, coverage lives in code — a module per publisher, a parser per site, a deploy every time you want a new one. In the second, coverage lives in data: a table of source rows, each row naming a kind and a configuration, and a small number of handlers that know how to execute each kind.

The second is not merely tidier. It changes who is allowed to extend the system. When adding a section to your morning reading is a row insert, the person doing it is the person reading; when it is a code change, the person doing it is whoever can deploy. For a tool with an audience of two, that difference is the whole product.

What a Source Row Actually Holds

The interesting columns are not the obvious ones. A row carries its kind — a syndicated outlet feed, a search-engine news query, a provider API — and a configuration blob whose meaning depends on that kind. Then it carries the state the fetch loop needs to be polite: the validators the server last handed back, when it was last fetched, how many times in a row it has failed, whether it is currently enabled, and if not, why.

That last group is the part people leave out of the first version and regret. Politeness is not a behavior you remember to perform; it is state you keep, per source, so the next round can behave correctly without recomputing anything.

No Shelf Fetches While It Renders

The companion rule is narrower than the slogan people usually reach for, and the narrowness is the interesting part. Shelf rendering never reaches the network: a tab is served entirely from the store, and ingest happens on a schedule or when someone presses refresh. What the rule does not say is that no handler ever fetches — opening an article does, on its first open, because extraction is deliberately lazy. That is one bounded request caused by one deliberate tap, which is a different thing from a list view quietly making forty.

This is what keeps the reader honest under load. A shelf that fetches while rendering has a latency that depends on somebody else's server, fails in ways that look like your bug, and — because renders are frequent and unpredictable — turns your polite client into an accidental hammer. Reading from the store and refreshing on a schedule decouples the two completely, and it means the answer to "why is this slow?" is always somewhere in your own process.

The Bit That Is Not Free

Declared sources buy provenance and editability. They do not buy identity. A source row tells you where an article entered from; it says nothing about whether the story behind it already entered from somewhere else. Those are separate problems and they need separate mechanisms, and conflating them is how a reader ends up showing the same event seven times while its architecture document claims it deduplicates.

Configuration is the interface to the outside world; code is the interface to the kinds of outside world. If adding coverage requires a deploy, the boundary is in the wrong place — you have written a parser per publisher instead of a handler per protocol.

Code

The source registry: kind plus config, and the state politeness needs·sql

-- A source is (kind, config) plus the state politeness requires.
-- Note what is NOT here: nothing about which shelf shows it. That
-- binding lives in its own table, because one feed can feed many
-- shelves and a shelf is not a property of the feed.

CREATE TABLE sources (
  id                   INTEGER PRIMARY KEY,
  kind                 TEXT NOT NULL,          -- outlet-rss | news-query | provider-api
  name                 TEXT NOT NULL UNIQUE,   -- a LABEL, not the identity
  config               TEXT NOT NULL DEFAULT '{}',  -- meaning depends on kind

  -- conditional-GET state: handed to us by the server last time
  etag                 TEXT,
  last_modified        TEXT,

  -- circuit-breaker state: why the next round may skip this source
  enabled              INTEGER NOT NULL DEFAULT 1,
  consecutive_failures INTEGER NOT NULL DEFAULT 0,
  disabled_reason      TEXT,

  last_fetch_at        TEXT,
  created_at           TEXT NOT NULL
);

-- Many-to-many on purpose: a feed is not owned by the shelf that
-- happens to show it first.
CREATE TABLE tab_sources (
  tab_id    INTEGER NOT NULL REFERENCES tabs(id)    ON DELETE CASCADE,
  source_id INTEGER NOT NULL REFERENCES sources(id) ON DELETE CASCADE,
  PRIMARY KEY (tab_id, source_id)
);

Exercise

Take any system you have built that talks to third-party services, and count the deploys it would take to add one new provider. If the answer is more than zero, find the line where a provider-specific detail crossed from configuration into code, and decide whether that detail is genuinely protocol-shaped or just happened to be written there first. Then write down what the configuration blob for that provider kind would have to contain to make the answer zero.

Hint

The detail that resists is usually authentication or pagination, and it usually resists because one provider does it differently enough that someone special-cased it. That is a real difference in kind — which means it deserves a new kind with its own handler, not a branch inside the existing one.

The Decision That Shapes Everything Downstream

What a Source Row Actually Holds

No Shelf Fetches While It Renders

The Bit That Is Not Free

Code

External links

Exercise

Progress

Comments 0