Pages of a Projection Are Not Disjoint — and the Paragraph That Lied for Nine Rounds · Tidings Quest

An Offset Counts Rows You Did Not Show

Pagination by offset assumes the rows the database skips are the rows the reader already saw. For a plain table that holds. For a shelf built as a projection it cannot, and the reason is structural rather than a bug: the query over-fetches, then collapses repeated stories, drops muted rows, and promotes others before trimming. Every one of those steps removes rows the offset already counted.

So the offset and the rendered position drift by exactly however much each page dropped. Measured across three live shelves, the second page began before the first page had ended by one, seven and four rows respectively. Not a race, not a stale cache: an arithmetic consequence of the design.

The Fix Nobody Wants and the One That Is Right

The textbook answer is a cursor: remember where you stopped by identity rather than by count. That is correct for a stable ordered set, and it is the wrong trade here, because the projection is rebuilt on every render. A cursor would have to encode the state of deduplication, muting and promotion at the moment the previous page was computed — which is to say, materialize the projection somewhere. Enormous complexity to make an exact promise that a rebuilt shelf cannot honor anyway.

The honest alternative is to stop pretending pages are disjoint and merge on the client. Merge incoming rows by identity, and also by the same punctuation-blind key the deduplication uses, so a different copy of an already-shown story does not appear as new. Cheap, and correct for the situation that actually exists.

Infer "Has More" — Never Count

The related trap is the total. A projection rebuilt per render does not know how many rows it will produce without producing all of them, and a count taken from the underlying table is a count of rows that will not all survive. So a shelf that reports a total is reporting a number that is wrong in a direction it cannot even predict.

Infer instead: a page shorter than requested means the end. It gives up the progress indicator and never lies, which is the correct trade for a surface whose content is a judgment rather than a table.

The Paragraph That Claimed a Capability for Nine Rounds

One last failure, the smallest in the codebase and the most uncomfortable. The architecture document stated the app worked offline with a queue for actions taken while disconnected. It said so for nine design rounds. Neither existed. Nobody had lied — an early plan had included both, they were never built, and the paragraph was never revisited.

The instructive part is what happened when it was found: the paragraph was corrected, not the code. Building an offline mode to make the documentation true would have been implementing an unexamined feature to avoid admitting a stale sentence. An offline story is a feature to decide on, not a bug to fix. The document now says the app is installable and not offline-capable, which is what is true.

When documentation and reality disagree, find out which one you actually want before changing either. The reflex to make the document true by building the thing is how systems acquire features nobody chose — and the document is by far the cheaper of the two to change.

Code

Client-side merge by id and story key, with has-more inferred rather than counted·typescript

// Shelf pages are NOT disjoint and cannot be: the shelf over-fetches,
// collapses repeats, and drops muted rows before trimming, so the SQL
// offset and the rendered position drift by however much each page
// removed. Measured across three live shelves: overlapping rows at
// each page boundary: 1, 7 and 4 rows on the three measured.
//
// A server-side cursor would have to encode the state of dedupe,
// muting and promotion at the moment the previous page was computed --
// i.e. materialize a projection that is rebuilt every render. Wrong
// trade. Merge on the client instead.
function mergePage(existing: Article[], incoming: Article[]): Article[] {
  const seenIds = new Set(existing.map((a) => a.id));
  // Also merge by the store's own punctuation-blind title key, so a
  // DIFFERENT copy of an already-shown story is not treated as new.
  const seenStories = new Set(existing.map((a) => titleKey(a.title)));

  const fresh = incoming.filter(
    (a) => !seenIds.has(a.id) && !seenStories.has(titleKey(a.title)),
  );
  return [...existing, ...fresh];
}

// 'Has more' is INFERRED from a short page, never from a count: a
// projection rebuilt on every render does not know its own total, and
// a count of the underlying table counts rows that will not survive.
const hasMore = incoming.length >= requestedLimit;

Exercise

Find a paginated list in your system where the query filters rows after they are selected. Request page one and page two, and count how many identities appear in both. Then read your project's architecture document and find one capability it claims — check whether that capability exists today.

Hint

For the second half, favor paragraphs describing behavior under failure: offline support, retries, degradation, recovery. Those are written early from intent, exercised rarely, and almost never revisited — which makes them the most likely place a document is describing a system that was planned rather than the one that was built.

Pages of a Projection Are Not Disjoint — and the Paragraph That Lied for Nine Rounds

An Offset Counts Rows You Did Not Show

The Fix Nobody Wants and the One That Is Right

Infer "Has More" — Never Count

The Paragraph That Claimed a Capability for Nine Rounds

Code

External links

Exercise

Progress

Comments 0