Saved Is Forever, Unsaved Is Compost

A Reader Accumulates Faster Than You Think

Forty sources on a half-hourly cycle is thousands of rows a day. Almost none of it will ever be read: it arrives, sits for a few hours in the recent window, and is displaced by more of the same. Keeping all of it forever is not principled, it is just inertia — it inflates every query, every backup, and every index for material nobody will look at again.

So a reader needs a retention policy. And the moment you write one, you have made a promise about its exception, because the whole point of a save button is that saving means something.

Two Classes, Stated Out Loud

The policy that survives contact with real use is blunt: unsaved articles older than the window are compost; saved articles are permanent; and all events are permanent regardless of what happened to the article they refer to. Two weeks is a reasonable window for a news reader — long enough that nothing you were mid-way through disappears, short enough that the store stays a working set.

Sharing later joined saving as a permanence claim, and the reasoning is worth keeping because it is a product argument rather than a technical one: publishing an article is a stronger commitment than filing it, and a shelf of your own posts that silently empties after a fortnight is a worse lie than one that never existed.

One Deleter, Named

Here is the structural part. It is not enough for the retention function to be careful about which rows it spares. There must be exactly one code path in the entire system that deletes an article, and every other path must be structurally incapable of it. That means no cascade may reach the article table, no cleanup job may take a shortcut, and no admin helper may exist "just for testing."

The reason is that a permanence guarantee is a claim about all possible executions, not about the one function that thinks it owns the subject. A careful deleter plus one careless cascade is a system with no guarantee at all — and the next lesson is exactly that story.

Retention Covers Everything Derived

The last piece is easy to forget: articles are not the only thing that accumulates. The fetch log, cached samples, rendered audio, extracted text — every derived artifact needs a window too, tied to the lifetime of what it explains. A fetch log kept forever to explain articles that were composted a year ago is pure sediment.

A retention policy is a promise, and promises need exactly one enforcer. If two code paths can delete the protected thing, you do not have a policy — you have a convention, and conventions are broken by whoever writes the next feature without reading this one.

Code

One deleter for articles, and a compost pass for everything derived·python

def prune(con) -> int:
    """The ONLY thing in this system that deletes an article.

    Everything else that wants an article gone must go through here.
    No cascade reaches this table (see the next lesson for why that
    sentence had to be earned), and events survive in the JSONL log
    regardless of what happens to the row they point at.
    """
    cutoff = (datetime.now(timezone.utc)
              - timedelta(days=RETENTION_DAYS)).isoformat(timespec="seconds")
    cur = con.execute(
        """DELETE FROM articles WHERE fetched_at < ?
           AND id NOT IN (
               SELECT article_id FROM article_state
               WHERE saved_at IS NOT NULL OR shared_at IS NOT NULL
           )""",
        (cutoff,),
    )
    con.commit()
    return cur.rowcount


def compost(con) -> dict:
    """Retention covers everything DERIVED, not only articles. Each
    derived artifact ages out against the lifetime of what it explains."""
    return {
        "articles":      prune(con),
        "fetch_log":     prune_fetch_log(con),      # diagnostic, not truth
        "trend_samples": prune_trend_samples(con),  # per-topic depth
        "tts_cache":     prune_tts_cache(con),      # keyed to live articles
    }

Exercise

In a system you own, find the data that grows without bound and write the retention rule you would want in one sentence, including its exception. Then count how many code paths can currently delete the protected class — remembering to check cascades, cleanup jobs, and test fixtures that run against real environments. If the count is not one, you have a convention rather than a policy.

Hint

Cascades are the ones people miss, because they are declared once in a schema and then never appear in any code you would grep for a delete. List the foreign keys pointing at your protected table and read their ON DELETE clauses before you trust your count.

A Reader Accumulates Faster Than You Think

Two Classes, Stated Out Loud

One Deleter, Named

Retention Covers Everything Derived

Code

External links

Exercise

Progress

Comments 0