Politeness Is State You Keep, Not Manners You Remember

What Politeness Actually Costs the Other Side

Every fetch you make is work somebody else pays for. A feed endpoint may be generated per request, and the publisher serving it is not being compensated by your reading. So the question is not "am I allowed to fetch this?" but "what is the cheapest way to learn whether there is anything new?"

HTTP answered that question decades ago and most aggregators still ignore the answer. When a server hands you a feed it usually also hands you validators: an entity tag, a last-modified timestamp, or both. Store them on the source row. Send them back next time as If-None-Match and If-Modified-Since. A server that has nothing new answers 304 Not Modified with no body at all — it does not render the feed, does not serialize it, does not transfer it.

Record 304 as Its Own Outcome

The subtle part is bookkeeping. A 304 is neither an error nor a successful ingest, and collapsing it into either one corrupts something. Count it as an error and your circuit breaker will eventually disable your best-behaved sources — the ones that update rarely and answer 304 most often. Count it as a successful fetch of zero articles and your logs will suggest a source has gone dry when it is merely quiet.

Give it a third status. The fetch log then tells you three genuinely different things: this source gave us articles, this source told us there was nothing new, and this source failed.

Serial Is a Politeness Feature

It is tempting to fan a fetch round out concurrently, and for your own latency it would help. But a round that hits forty sources in parallel is a small burst against whoever happens to host several of them, and the fastest way to get a private tool blocked is to look like a crawler.

A plain sequential loop over the enabled sources, sharing one client with a configured timeout and an identifying user agent, gives you per-host concurrency of one for free — not because you implemented a semaphore, but because there is only ever one request in flight. A fetch round taking twenty-odd seconds costs nothing, because nobody is waiting on it: the reader is served from the store.

Identify Yourself

Send a user agent that names the product and says what it is. This is not decoration. It is what lets an administrator looking at their logs distinguish a small private reader from an anonymous scraper, and decide accordingly. A tool that hides what it is has made a decision about how it wants to be treated.

The cheapest request is the one that transfers nothing. Conditional GET is not an optimization you add when traffic grows; it is the difference between a client that costs a publisher a header exchange and one that costs them a full render, every round, forever.

Code

Conditional GET with three outcomes, and a deliberately sequential round·python

def run_source(con, source, client):
    """One source, one round. Conditional GET, three outcomes."""
    headers = {}
    if source.get("etag"):
        headers["If-None-Match"] = source["etag"]
    if source.get("last_modified"):
        headers["If-Modified-Since"] = source["last_modified"]

    try:
        resp = client.get(source_url(source), headers=headers)
    except Exception as exc:
        # A transport failure is a failure: it counts toward the breaker.
        record_fetch(con, source["id"], status="error",
                     detail=f"{type(exc).__name__}: {exc}")
        return {"status": "error"}

    if resp.status_code == 304:
        # Its OWN status. Not an error (would trip the breaker on the
        # best-behaved sources), not an empty success (would read as
        # 'this feed has gone dry').
        record_fetch(con, source["id"], status="not_modified", http_status=304)
        return {"status": "not_modified", "new": 0}

    if resp.status_code != 200:
        record_fetch(con, source["id"], status="error",
                     http_status=resp.status_code, detail=resp.text[:200])
        return {"status": "error", "http_status": resp.status_code}

    entries = parse_entries(resp.content, source["kind"])
    new = sum(1 for e in entries if upsert_article(con, source["id"], e))
    con.commit()
    # Store the validators the server just gave us, for next round.
    record_fetch(con, source["id"], status="ok", http_status=200,
                 new_articles=new,
                 etag=resp.headers.get("ETag"),
                 last_modified=resp.headers.get("Last-Modified"))
    return {"status": "ok", "entries": len(entries), "new": new}


# The round itself: one client, one request in flight, identified.
with httpx.Client(follow_redirects=True,
                  timeout=FETCH_TIMEOUT_SECONDS,
                  headers={"User-Agent": USER_AGENT}) as client:
    for source in enabled_and_bound_sources:
        results.append(run_source(con, source, client))

Exercise

Find something in your own systems that polls a third party on a timer. Check whether it sends conditional headers, and whether its logs can distinguish 'nothing new' from 'nothing returned'. If it cannot, work out what its failure-counting logic currently does with a quiet source — and whether the sources it would disable first are the ones you would least want to lose.

Hint

The tell is a single boolean or a try/except around the whole poll, producing exactly two outcomes. Two outcomes cannot express three states, so one of the three is being silently folded into another — and the one that gets folded is almost always 'nothing changed', because it is the one nobody thought about.

What Politeness Actually Costs the Other Side

Record 304 as Its Own Outcome

Serial Is a Politeness Feature

Identify Yourself

Code

External links

Exercise

Progress

Comments 0