One Door Out: The Shelf That Skipped the Shaping Helper

A Badge That Was Missing From Exactly One Shelf

Cleaned articles carry a small badge — a mark meaning this one has been pre-processed and will open instantly. It appeared everywhere except the personalized shelf. Not intermittently; never. And on that same shelf, the payload was enormous: a five-row window shipping over nine thousand characters of article body the client had no use for.

One cause, two symptoms. Every other shelf built its rows through a shared shaping helper — the function that decides which columns leave the database, derives the booleans the client needs, and drops the heavy text fields. The personalized shelf did not. It selected the article rows directly and returned them.

Two Failure Directions From One Omission

This is what makes the bug worth a lesson rather than a footnote. Skipping the shaping helper failed in both directions at once, and the two look nothing alike.

Things that should have been added were missing: the badge's boolean is derived from a timestamp column, and nothing derived it. Things that should have been removed were present: the full extracted body is stored on the row and every other path strips it. A reviewer looking for the missing badge would find the derivation and add it — and never notice the payload, because a payload being too big produces no error anywhere. It is slower, and slower is not a symptom anyone reports.

Name the Door

The fix is not "remember to call the helper." It is to make the helper the only way rows can leave, and to name it so that its absence is legible in review. A function called _rows reads like an implementation detail and invites a caller to write their own; the same function called card_rows — the door article rows leave through — makes a hand-rolled query next to it look like what it is.

The general shape: when a payload has invariants, those invariants belong to a single serialization boundary, not to each endpoint's good intentions. Endpoints choose which rows; the door decides what a row looks like.

Test the Property, Not the Endpoint

This survived a large test suite because every test asserted the behavior of one shelf. Each shelf had tests; the personalized one had tests too, and they passed, because they checked that it returned the right articles in the right order — which it did. Nobody had written the test that says every endpoint returning article rows returns them in the card shape. That test is one loop over the routes, and it would have caught this on the day it was introduced.

If a payload shape is an invariant, give it exactly one door and test it as a property. Per-endpoint tests verify the endpoints someone thought to write tests for; a property test over all of them verifies the ones nobody thought about, which is where this class of bug always lives.

Code

One named door, and the property test that covers routes nobody thought of·python

def card_rows(con, sql: str, params: tuple = ()) -> list[dict]:
    """The single door article rows leave the database through.

    Named, not private: a hand-rolled SELECT sitting next to a call to
    `card_rows` looks wrong, where one sitting next to `_rows` looks
    like someone had a reason.
    """
    out = []
    for r in con.execute(sql, params):
        row = dict(r)
        # DERIVE what the client needs...
        row["cleaned"] = bool(row.get("cleaned_at"))
        row["clean_failed"] = bool(row.get("clean_failed_at"))
        row["queued"] = bool(row.get("queued_at"))
        # ...and DROP what it must never receive. Bodies are reader-only;
        # shipping them is invisible (no error, just slower).
        for heavy in ("clean_text", "extracted_text"):
            row.pop(heavy, None)
        out.append(row)
    return out


# The property test that would have caught it on day one. Per-endpoint
# tests all passed: the shelf DID return the right articles in the right
# order. Nobody had asserted the shape across every route.
@pytest.mark.parametrize("path", ARTICLE_RETURNING_ROUTES)
def test_every_article_route_returns_card_shape(client, path):
    for row in client.get(path).json()["articles"]:
        assert "cleaned" in row, f"{path} skipped the shaping door"
        assert "clean_text" not in row, f"{path} leaked an article body"

Exercise

Pick an entity your API returns from more than one endpoint. Write a single test that loops over every route returning it and asserts two things: one field that must always be present, and one field that must never be. Run it. If it passes everywhere, you have cheap insurance; if it fails, you have found today's version of this bug.

Hint

The must-never-be-present assertion is the one that finds things, because nothing else in your stack is looking for it. Good candidates are internal identifiers, soft-delete flags, full text bodies, and anything the ORM includes by default because someone wrote select-star once and it worked.

A Badge That Was Missing From Exactly One Shelf

Two Failure Directions From One Omission

Name the Door

Test the Property, Not the Endpoint

Code

External links

Exercise

Progress

Comments 0