There Is No Free Picture in a Shell

The Cheap Enrichment That Cannot Work

Finding a picture for an article is usually easy and deterministic: fetch the page, read the social-preview tags out of its head, done. No model, no guessing, one small request. So when most cards lacked pictures, the obvious move was to run that pass over them.

It cannot work on the rows that need it, and the reason is worth understanding precisely rather than as a rule of thumb. A large share of the delivered articles are not the publisher's page at all — they are an aggregator's redirect wrapper, a page whose only purpose is to bounce a browser onward. Its preview tags describe the aggregator, not the article. Probing one returns a picture, technically. It is the aggregator's own logo, byte-identical on every such page.

Measure the Negative Result Properly

This is a nice example of a measurement whose entire value is a negative. Probing a handful of these wrappers and comparing the returned image URLs takes minutes, and it converts a plausible plan into a closed question: the wrappers are not merely unhelpful, they are uniformly unhelpful, so no sampling strategy or retry policy rescues them.

The cost side deserves the same rigor. Each of those pages is large — the useful signal sits near the end, so no bounded read helps — and every one buys nothing. Establishing that a class of work has a known cost and a provably empty payoff is worth as much as finding a fix, because it removes an option that would otherwise keep getting proposed.

Skip It, and Record That You Skipped It

The right handling is to stamp such rows as checked without fetching them. That sounds like a small optimization and it is really a correctness measure: without the stamp, every round rediscovers the same rows as un-enriched and spends its budget on them forever. A negative cache is what turns "we know this will not help" from a comment into behavior.

The same applies to genuine failures. A page that was probed and had no preview tag should also be stamped, so the budget moves on. The distinction between a row that has never been tried and one that has been tried and yielded nothing is exactly what a bounded budget needs in order to make progress.

Where the Pictures Actually Come From

Two lanes end up doing the real work, and neither is the one everybody proposed. Feeds that come directly from a publisher usually carry an image in the feed entry itself — free, no extra request. And for a wrapper, the only way to a real picture is to resolve it to the underlying article first, which is a different and more expensive operation, worth spending only on rows a person is about to look at. That is the next two lessons.

A measurement that closes an option is as valuable as one that opens a fix. "This will not work, here is the number" prevents the same suggestion from returning every planning session — and negative results are the ones nobody writes down, which is why they get re-proposed.

Code

Skipping provably-empty work, and stamping so the budget can move on·python

def is_shell(url: str) -> bool:
    """An aggregator redirect wrapper -- a real link, but the
    aggregator's identity, not the publisher's."""
    return urlsplit(url or "").netloc.endswith(AGGREGATOR_HOST)


def run_og_pass(con, head_ids: list[int]) -> dict:
    """Read preview tags off article pages to fill missing images.

    Measured before building: a wrapper page's preview image is the
    aggregator's own logo, byte-identical on every wrapper probed. The
    fetch is large (the signature sits at the END of the page, so no
    bounded read helps) and buys nothing. So wrappers are stamped
    WITHOUT fetching -- they reach a picture only via resolution.
    """
    checked = fetched = found = 0
    for row in _imageless_candidates(con, OG_PER_ROUND, head_ids):
        if is_shell(row["url"]):
            # Negative cache, not an optimization: without the stamp
            # every round rediscovers these rows and spends its whole
            # budget re-deciding not to fetch them.
            _stamp_og_checked(con, row["id"])
            checked += 1
            continue
        image = _read_preview_tags(row["url"])
        fetched += 1
        _stamp_og_checked(con, row["id"], image_url=image)   # stamp on MISS too
        found += 1 if image else 0
    return {"skipped_shells": checked, "fetched": fetched, "found": found}

Exercise

Find an enrichment or backfill job you run on a schedule and check whether it records failed attempts. If it does not, work out what fraction of each run is spent re-attempting the same permanently-failing rows. Then decide what the stamp should contain — a boolean is usually wrong, because you will eventually want to retry after a policy change and will have no way to tell old failures from new ones.

Hint

A timestamp plus a short reason is almost always the right shape. It lets you re-run only what failed before a given date, which is exactly what you need after fixing the thing that caused the failures — and a boolean forces you to either retry everything or nothing.

The Cheap Enrichment That Cannot Work

Measure the Negative Result Properly

Skip It, and Record That You Skipped It

Where the Pictures Actually Come From

Code

External links

Exercise

Progress

Comments 0