The Same Story, Seven Times: Identity Versus Sameness

Correct Identity, Duplicated Shelf

Canonical-URL identity is right. Two rows with the same cleaned URL are the same article, the uniqueness constraint enforces it, and nothing gets stored twice. And a shelf can still show one event seven times.

The measurement: roughly eight percent of a day's rows were repeats of a story already present, with a worst case of one story arriving seven times through seven different wrapper URLs. Every one of those URLs was genuinely distinct. Every row was a real delivery through a real source. Identity was not violated — it simply does not answer the question a reader is asking, which is not "is this the same URL?" but "is this the same news?"

Two Different Relations, One Confusing Word

It helps to name them separately. Identity is exact, cheap, and a property of the record: same canonical URL, same row. Sameness is fuzzy, judgmental, and a property of the world: two records describing one event. Identity belongs at ingest, where it prevents storing the same delivery twice. Sameness belongs at render, where it decides what a person should see.

Collapsing sameness at ingest is the tempting shortcut and it destroys things you cannot get back. Provenance disappears — you can no longer tell which sources carried a story, which is real information about how the day propagated. The decision becomes irreversible. And a fuzzy rule applied destructively will eventually merge two genuinely different articles with similar titles, with nothing left to recover from.

Keep the Most Useful Copy, in the First Copy's Position

When collapsing at render you must decide which copy survives, and the answer is not "the first one." The copies differ in quality: one may already be pre-processed, another may carry a picture, another may point at the publisher directly rather than at a wrapper. Rank them and keep the best.

But keep it where the first one was. Position carries meaning on a time-ordered shelf, and promoting a copy to its own later position would reorder the shelf as a side effect of deduplication — one mechanism quietly doing two jobs, which is how a shelf becomes inexplicable.

Grouping Key: Punctuation-Blind, Not Clever

The key that decides sameness should be the dullest thing that works — a normalized title with punctuation and case flattened. The temptation is similarity scoring, and it is a poor trade here: it introduces a threshold to tune, it can merge unrelated pieces, and it costs real time on every render. A blunt key under-merges, which is the safe direction: a duplicate that slips through is a mildly untidy shelf, while a wrong merge hides an article the reader will never know existed.

Store by identity; display by sameness. Anything exact enough to enforce with a constraint belongs in the store, and anything requiring judgment belongs in a projection you can rebuild — because the judgment will be wrong sometimes, and you want that to be a display bug rather than a data loss.

Code

Render-time collapse: best copy, first copy's position, blunt key·python

def dedupe_stories(rows: list[dict]) -> list[dict]:
    """Collapse the same story delivered by several sources.

    Measured: 8% of a day's rows were duplicates -- one story arrived
    seven times under seven different wrapper URLs, so canonical-URL
    identity (correct at ingest) cannot see it.

    RENDER-time only: every row stays in the store, and the copy that
    survives is the most useful one, holding the position of the FIRST
    copy so the shelf's ordering is untouched.
    """
    best: dict[str, dict] = {}
    order: list[str] = []
    for row in rows:
        key = title_key(str(row.get("title") or ""))   # punctuation-blind
        if not key:
            key = f"__id{row.get('id')}"               # never merge on empty
        if key not in best:
            best[key] = row
            order.append(key)                          # position of copy #1
        elif _story_rank(row) > _story_rank(best[key]):
            best[key] = row                            # ...but the best copy
    return [best[key] for key in order]


def _story_rank(row: dict) -> tuple:
    """Which copy is most useful to a reader, in priority order."""
    return (
        bool(row.get("cleaned_at")),        # already pre-processed
        bool(row.get("image_url")),         # has a picture
        not is_shell(row.get("url", "")),   # points at the publisher
        bool(row.get("summary")),           # has a dek
    )

Exercise

Find somewhere your system deduplicates and classify it: is the rule exact or fuzzy, and is it applied destructively at write time or reversibly at read time? If it is fuzzy and destructive, construct two records the rule would wrongly merge and work out what would be lost. Then decide whether the rule can move to read time.

Hint

Fuzzy-and-destructive usually got that way for performance, and the performance argument is usually stale — it was made when the collapse happened over the whole table rather than over one page of results. Measure it at the size you actually render before accepting that the write-time version is necessary.

Correct Identity, Duplicated Shelf

Two Different Relations, One Confusing Word

Keep the Most Useful Copy, in the First Copy's Position

Grouping Key: Punctuation-Blind, Not Clever

Code

External links

Exercise

Progress

Comments 0