The Paywall That Was Never a Paywall

A Permanent Verdict About the Wrong Thing

Two articles carried the marker meaning permanently unextractable. The recorded reason said the page came back empty, and the obvious interpretation was a paywall or a dead link — both genuinely permanent, both correctly handled by never retrying.

Both extracted perfectly the moment their redirect wrapper was resolved to the publisher's real URL. Thousands of characters each. There was never a paywall. The extraction service was choking on the wrapper's client-side redirect, and the wrapper is not the article — it is a piece of plumbing that happened to be the URL on file.

The Diagnosis Named the Wrong Layer

This is what makes the bug worth a lesson rather than a fix. The failure was real, the handling was correct for what it believed, and the belief was about the wrong object. "This article cannot be extracted" was recorded when the true statement was "this wrapper cannot be extracted" — and those differ in exactly the way that matters, because one is a permanent property of the content and the other is a removable property of the path to it.

The cost compounded. Because the verdict was permanent, the system stopped trying. A misdiagnosis at one layer became a permanent loss at another, and the reader simply never got those articles.

Resolve the Path Before Judging the Destination

The fix reorders the operation: resolve the wrapper to the real URL first, then extract, then judge. A permanent verdict is only permitted once you are certain you were talking to the thing you meant to judge.

Resolution is expensive here — the signature needed to unwrap sits at the end of a large page, so no partial read helps — which is why it happens lazily, on the handful of articles a person actually opens or queues, and never across the backlog. That is the same aim-do-not-size lesson from the measurement track, arriving from a completely different direction.

Ask What the Error Message Is a Statement About

The generalizable habit is small and worth building: when an error crosses a layer, check what object its message refers to. "Empty page content" is a statement about a fetch, and it had been stored as a statement about an article. Passing an error upward without re-scoping it is how a lower layer's transient, path-specific problem becomes an upper layer's permanent fact about a domain object.

A permanent verdict must name what it is a verdict about. "Extraction failed" is not a property of an article until you have proved you were extracting the article — and if there is a redirect, a wrapper, or a proxy in between, you have not.

Code

Resolve the path, then extract, then judge — in that order·python

def resolve_shell_for_reading(con, article: dict) -> dict:
    """Turn a redirect wrapper into its publisher URL BEFORE extracting.

    Measured: both articles the clean lane had stamped as PERMANENTLY
    failed ('returned empty page content') extracted fine the moment
    the wrapper was resolved -- 4.5k and 13.4k chars. The extractor
    chokes on the wrapper's JS redirect, not on the publisher. So the
    stamp was a permanent verdict about the WRONG OBJECT, and it cost
    real articles.

    Lazy, here, because unwrapping costs ~550KB (the signature sits at
    the END of the page, so no bounded read helps) -- affordable for
    the handful someone opens, never for the backlog.
    Failure is silent: the wrapper URL is still a working link.
    """
    url = str(article.get("url") or "")
    if not canonical.is_shell(url):
        return article
    try:
        real = _unwrap(url)
    except Exception:
        return article          # soft: we simply extract the wrapper
    return {**article, "url": real}


async def get_reader(con, article_id: int):
    article = store.article(con, article_id)
    article = resolve_shell_for_reading(con, article)   # <- path first
    return await extract_and_judge(con, article)        # <- then verdict

Exercise

Find a place where your system records a permanent negative state about an entity — unreachable, invalid, unsupported. Trace the error that produces it back to where it was raised, and check what object that layer was actually talking about. Then check whether anything between the two could have caused the failure by itself.

Hint

Redirects, proxies, CDNs, wrappers and gateways are the usual culprits, because each is a hop that can fail independently of the destination. If any of them sits in the path, a failure attributed to the destination is at best unproven — and if the verdict is permanent, unproven is the same as wrong.

A Permanent Verdict About the Wrong Thing

The Diagnosis Named the Wrong Layer

Resolve the Path Before Judging the Destination

Ask What the Error Message Is a Statement About

Code

External links

Exercise

Progress

Comments 0