Append-Only Ground Truth: Record the Act, Not the State

Two Ways to Remember That an Article Was Liked

The obvious way is a column. Add liked to the article row, set it true, done. It is one write, it is easy to query, and it is what most applications do. The other way is to append a record — at this instant, this article, the act of liking — to a file that is never edited, and to derive the boolean when anyone asks.

The column stores a conclusion. The log stores what happened. That difference sounds academic until the first time you need something the conclusion cannot answer: when was it liked, was it liked before or after it was read, was it ever unliked, how many things were liked last Tuesday, and what did the reader's taste look like a month ago. A column has thrown all of that away by design; the log never had to decide in advance which questions mattered.

The Vocabulary Is Small and Closed

An event vocabulary should be short enough to hold in your head — opened, liked, disliked, saved, shared, queued, and their reversals. Keeping it closed is what makes projections tractable: every consumer knows the complete set of things that can appear, and adding a verb is a deliberate act with a visible blast radius.

Note that reversals are events too. Un-saving is not the deletion of the save; it is a new fact that happened later. That is the whole discipline in one line — you never edit the past, you append the correction — and it is what lets a projection be a pure function of the log rather than of the order in which you happened to apply patches.

Why JSONL, and Why Beside the Database

The log wants properties a relational table is bad at: appends that cannot corrupt earlier lines, a format readable by anything, and durability that does not depend on the schema of a working set that gets rebuilt. One JSON object per line in a plain file gives all three. It survives a database rebuild, it can be replayed into a new schema, and when something looks wrong you can read it with your eyes.

The relational store still earns its place — it holds the article rows, the shelves, the caches, and the projections that need indexes. The split is deliberate: the durable spine is a file, the queryable working set is a database, and if the two ever disagree, the file wins.

The Test That Tells You It Is Real

There is one question that separates a genuine event log from a table with a timestamp column: could you delete the entire database and rebuild every shelf, score, and statistic from the log alone? If the answer is yes, projections are safe to change, because a wrong one is a bug you fix and recompute. If the answer is no, then somewhere a projection has become the only copy of a fact, and that fact is now as fragile as the code that wrote it.

Store the act; derive the state. A conclusion is one answer to one question you thought of in advance. The act is the raw material for every question you have not thought of yet — including the ones your future ranking function will need and cannot ask retroactively.

Code

The append-only spine, and a projection as a fold over it·python

# One JSON object per line, appended, never rewritten.
# {"ts": "...", "type": "like", "article_id": 4812}
# {"ts": "...", "type": "save", "article_id": 4812}
# {"ts": "...", "type": "unsave", "article_id": 4812}

EVENT_KINDS = (
    "read", "like", "unlike", "dislike", "undislike",
    "save", "unsave", "share", "queue", "unqueue",
)


def append_event(kind: str, article_id: int, **extra) -> None:
    """The only write path into ground truth. A reversal is a NEW event,
    never an edit of the one it reverses -- that is what keeps a
    projection a pure function of the log."""
    if kind not in EVENT_KINDS:
        raise ValueError(f"unknown event kind: {kind}")
    row = {"ts": utcnow(), "type": kind, "article_id": article_id, **extra}
    with open(EVENT_LOG_PATH, "a", encoding="utf-8") as fh:
        fh.write(json.dumps(row, ensure_ascii=False) + "\n")


def project_state(log):
    """Every shelf, badge and score is one of these -- a fold over the
    log. Drop the database and this still returns the right answer."""
    state = {}
    for e in log:                      # chronological; later wins
        st = state.setdefault(e["article_id"], {})
        if e["type"] == "save":       st["saved_at"] = e["ts"]
        elif e["type"] == "unsave":   st["saved_at"] = None
        elif e["type"] == "like":     st["liked_at"], st["disliked_at"] = e["ts"], None
        elif e["type"] == "dislike":  st["disliked_at"], st["liked_at"] = e["ts"], None
    return state

Exercise

Pick a boolean or status column in a system you own — anything set by a user action. Write down three questions you cannot answer about it today because only the conclusion was stored. Then check whether anything in the system already needs one of those answers and is approximating it from a timestamp or an audit table bolted on afterwards.

Hint

The three questions are almost always: when did it change, what was it before, and how many changed in some window. If you find an audit table that was added later to answer exactly those, you have found a log that grew accidentally — and it is probably incomplete, because it was written to satisfy one report rather than to be ground truth.

Two Ways to Remember That an Article Was Liked

The Vocabulary Is Small and Closed

Why JSONL, and Why Beside the Database

The Test That Tells You It Is Real

Code

External links

Exercise

Progress

Comments 0