Skip to content
C.W.K.
Stream
Lesson 01 of 05 · published

The Cursor That Was Thrown Away

~13 min · pagination, http, war-story, bug

Level 0Raw Ore
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"A truncated page reads exactly like a complete one."

The mechanism, in one sentence

The provider paginates with a next-page URL: a fully-formed link carrying the cursor and every filter from the original request, already encoded in its query string. You follow it, you get the next page, you repeat until there is no next link.

The bug is that this client library, given both a URL and a params dictionary, replaces the URL's query string with those parameters rather than merging into it. So a call that passes the API key as params does not add a key to the next-page URL — it deletes the cursor and the filters, and asks for the unfiltered first page again.

Every request returned 200. Every page parsed. The loop terminated normally. And the universe of active US common stock came back as 1,083 tickers, in alphabetical order, ending somewhere in the C's.

What the wrong answer looked like

Here is where this stops being a routine pagination bug. That truncated list was the denominator for a concentration measure: how much of the US market is its ten largest companies. Ranked within an alphabetically-truncated universe, the top ten came out as the largest companies whose tickers begin with A, B, or C.

So the product published a roster of "the ten largest companies in America" that omitted the largest semiconductor company on earth — along with every other megacap whose ticker starts past C. Not a rounding error. Not a slightly stale figure. A specific, confident, comprehensively wrong answer, rendered in the same typography as a correct one.

Be careful about what this bug does and does not explain, because the roster had a second defect at the same time and they are easy to conflate. Truncation explains the omissions cleanly. What it explains about the odd inclusions is genuinely contested, and the honest thing is to show you the argument rather than a verdict. The commit that fixed the truncation asserts the roster followed from it directly, and prints what shipped: four A–C megacaps, then a run of smaller A–C names, then the machinery maker at the tail. So truncation clearly shaped the list. But several A–C companies larger than that machinery maker are also absent from it, and truncation cannot explain their absence — that needs the second bug, a cold share-count cache that had not yet fetched the largest companies, fixed two and a half minutes before the truncation was, and covered by the third lesson of this track.

Two independent failures, one plausible-looking roster, and no clean attribution of each symptom to one cause. That is the useful shape to leave it in. A track whose thesis is say which of the two you have would be a poor place to round a contested reading up to a settled one.

Why no check could have caught it
Walk through the validations you would write. Status code: 200, every page. Schema: valid, every field present. Parse: clean. Row count: 1,083, which is a perfectly plausible number of listings and nowhere near a suspicious round figure. Uniqueness: fine. Null checks: fine. Every property that can be checked from inside the response was correct. The only thing wrong was what was missing, and absence has no schema.

The general shape of silent truncation

This class is worth recognizing because the same shape recurs across very different systems.

A paginated fetch that loses its cursor. A database query with a default row limit nobody remembers setting. A directory listing that hits a page size. A search index that returns the first thousand hits. A message consumer that stops at a batch boundary. A regular expression that matches only the first occurrence per line.

In every case: the result is well-formed, the operation reports success, and the deficiency is only visible from outside — either by knowing what should have been there, or by having an independent count to compare against.

Assert on the shape of the whole, not the validity of the parts. Validity checks answer "is what I received well-formed?" They cannot answer "did I receive all of it?" For any operation that iterates over an unbounded set, you need at least one assertion about the total — an expected order of magnitude, a provider-reported count, a known member that must be present — because well-formedness will never notice a missing tail.

Code

The fix, and the comment that explains why it exists·python
def polygon_common_stock_symbols(limit_pages: int = 30) -> list[str]:
    """Active US COMMON STOCK, paginated.

    Following `next_url` requires appending the key to the URL, NOT
    passing it as `params`: httpx REPLACES a URL's existing query
    string with `params`, which silently threw away the cursor and
    every filter. The symptom was a universe that stopped at 1,083
    tickers, alphabetically A-C -- so the concentration roster's
    "top ten companies in America" listed Caterpillar and
    omitted NVIDIA and Microsoft (live 2026-08-06). A truncated page
    reads exactly like a complete one until you know what should be
    in it."""
    key = polygon_key()
    if not key:
        raise RuntimeError("Polygon key is not provisioned")
    out: list[str] = []
    url = (f"{POLYGON_REST}/v3/reference/tickers?market=stocks"
           f"&type=CS&active=true&limit=1000&apiKey={key}")
    with httpx.Client(timeout=60) as client:
        for _ in range(limit_pages):
            response = client.get(url)          # <- no params=
            response.raise_for_status()
            payload = response.json()
            out.extend(row["ticker"] for row in payload["results"])
            nxt = payload.get("next_url")
            if not nxt:
                break
            # append to the cursor URL; never rebuild its query
            url = nxt + ("&" if "?" in nxt else "?") + f"apiKey={key}"
    return out

External links

Exercise

Find every paginated fetch in a codebase you work on and check two things: does it assert anything about the total it expected, and what happens if the loop's page limit is reached? Most such loops exit silently at their cap, returning a partial set that every downstream consumer treats as complete. Add a total assertion to one of them and see whether it passes today.
Hint
The page-limit guard is the one people add for safety and then forget is load-bearing. If hitting it returns normally rather than raising, the safety valve has become a silent truncation switch — and it fires on exactly the day the dataset grows past what anyone tested with.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.