The Backlog Is a Red Herring

Two True Facts and a False Connection

Fact one: most cards on the reader's screen had no picture. Fact two: thousands of articles in the store had never been through the enrichment passes that find pictures. Both were true, both were measured, and the connection between them was invented by everybody who looked at it, including me.

The plausible story writes itself — there is a backlog, the backlog is why things are unenriched, so raise the budget or run the passes more often. The measurement that killed it took one query: how many articles are actually on a screen? Union the visible head of every enabled shelf and deduplicate. The answer was a couple of hundred, against a store of several thousand. Around two percent.

The Other Ninety-Eight Percent Is Never Looked At

This is the part that makes the backlog not merely unimportant but actively misleading. Articles that fall off the visible head are not queued for later attention; retention composts them in a fortnight and nobody ever scrolls that far. So the backlog is not work waiting to be done — it is work that will never be needed.

Enriching it is not slow, or expensive, or premature. It is pointless, in the precise sense that the output has no consumer. And an enrichment budget aimed at it lands almost entirely on rows that will be deleted unread, no matter how generous you make the budget.

Then What Was Actually Wrong?

The same measuring pass answered that too, because once you are counting the visible head you may as well count what is in it. Seventy percent of the cards on screen carried no picture — and the overwhelming majority of those were a kind of row the picture-finding pass skips deliberately, without fetching, because that kind of row provably has no picture to find.

So the real situation was the exact inverse of the assumed one. The problem was not a budget too small to reach the backlog; it was that no budget of any size could ever have helped the specific cards that were failing. That is not a tuning problem. It is a different problem entirely, and you cannot see it from the backlog.

Compute the Head After the Projection, Not Before

One subtlety worth stating, because it is easy to get wrong when you build this: the visible head must be computed after everything that changes what reaches the screen — after duplicates are collapsed, after muting, after any promotion. Compute it from the raw query instead and you will aim your budget at rows that the projection is about to remove, which reintroduces the original problem in a form that looks like it was solved.

Size a solution against what is consumed, not against what exists. A backlog is a compelling number because it is large and easy to count. The number that matters is almost always small, harder to compute, and on the other side of your rendering pipeline.

Code

Computing the visible head, after the projection and interleaved by rank·python

HEAD_CARDS = 12          # what fits above the fold, per shelf


def head_article_ids(con, per_shelf: int = HEAD_CARDS) -> list[int]:
    """The articles about to be looked at -- every enabled shelf's first
    screen, interleaved so no one shelf eats the enrichment budget.

    Measured: this union is ~184 articles, 2% of the store, while the
    riders were sweeping the whole firehose newest-first. The other 98%
    ages out unseen, so enriching it was work spent where nobody was
    looking. Computed AFTER dedupe, muting and the image floor, because
    those decide what actually reaches the screen.
    """
    shelves = []
    for tab in list_tabs(con):
        if not tab["enabled"] or tab["kind"] not in ("topic", "foryou", "queue"):
            continue
        rows = tab_articles(con, tab, limit=per_shelf)   # full projection
        if rows:
            shelves.append([int(r["id"]) for r in rows])

    # Interleave by RANK, not by shelf: take every shelf's first card,
    # then every shelf's second. A busy shelf cannot spend the budget
    # before a quiet one gets its top card enriched.
    ordered, seen = [], set()
    for rank in range(per_shelf):
        for shelf in shelves:
            if rank < len(shelf) and shelf[rank] not in seen:
                seen.add(shelf[rank])
                ordered.append(shelf[rank])
    return ordered

Exercise

Find a backlog metric your team watches — an unprocessed queue, a table of pending rows, a count of stale records. Work out what fraction of it is ever read by anything before it expires or is superseded. Then check whether any recent work was scoped against the backlog number rather than against the consumed fraction.

Hint

The fastest way to compute the consumed fraction is usually to instrument the read side for a day rather than to reason about it. Reasoning tends to produce the answer 'most of it, eventually', which is exactly the assumption that makes a backlog look like a problem in the first place.

Two True Facts and a False Connection

The Other Ninety-Eight Percent Is Never Looked At

Then What Was Actually Wrong?

Compute the Head After the Projection, Not Before

Code

External links

Exercise

Progress

Comments 0