One Door: Corpus.record

Every write, the same four steps

Search Spark's engine for the line that appends to the log and you find exactly one, inside Corpus.record. Moments, photos, Pippa's results, turns, kept phrases, trips, purposes and thread bindings all reach the disk through that method. The first code block is the whole of it. Under one lock it does four things, in this order:

Append the op to the month file, veiled and fsynced (the previous lesson).
Apply it to the SQLite mirror inside a transaction of its own.
Remember the log's new fingerprint in the mirror, unless some lines are unreadable, in which case the mirror stays marked stale.
Publish it to whoever is watching the moment it belongs to, so a phone or a browser sees it the instant it exists.

Then it returns the op. That is the entire contract, and every domain method in corpus.py ends by calling it.

Why the log goes first

The order is the safety argument. Suppose the process dies between steps 1 and 3. The op is on disk, while the mirror may or may not have it and still remembers the old fingerprint. At the next boot the engine compares the fingerprint it computes now with the one the mirror remembers, sees a difference, and rebuilds the mirror from the log. The log always wins.

The fingerprint is cheap on purpose: a SHA-256 over each month file's name, size and modification time in nanoseconds. Computing it reads no content, so record can afford it on every write. A test plays the crash directly. It closes the corpus, writes one more moment.create straight into the month file the way an interrupted write would leave it, and reopens. The stray moment is there and the count is two.

Reverse the order and you get the failure the design exists to prevent: a mirror that shows something the log never recorded, which the next rebuild would silently erase.

One lock, kept off the loop

The lock is a threading.RLock. Re-entrant, because the domain helpers call each other while holding it: creating a moment checks for a replayed request, reads the mirror, records the op and reads the result back, each step taking the same lock again. One lock for the whole corpus is fine here because Spark has one user and one engine process; the mirror's own docstring calls it a single-user app.

The engine serves requests from an asyncio event loop, though, and an fsync plus a lock wait would stall every other request on that loop. So async code never calls the corpus directly. It goes through asyncio.to_thread, and the turn runner wraps that in a small helper whose comment says the whole reason: corpus writes fsync and take a threading lock, so keep them off the loop.

Seventeen ops

Everything Spark can ever say about its world fits in a closed vocabulary. The third code block is the mirror's switchboard, one handler per op:

Family	Ops
Moments	`moment.create`, `moment.edit`, `moment.delete`, `moment.restore`
What a moment holds	`media.add`, `result.add`
The work that produced it	`turn.start`, `turn.finish`
Learning and links	`mark.add`, `mark.remove`, `link.add`, `link.remove`
Rows Dad edits	`trip.create`, `trip.edit`, `purpose.create`, `purpose.edit`
The thread a material lives in	`thread.bind`

The values inside ops are closed too, and checked before anything is written: five moment kinds (photo, conversation, text, live, recall), seven media roles, ten turn kinds, three ways a turn can end (completed, failed, interrupted). A bad request raises CorpusError, the route answers 400, and the log never sees it. A test parametrizes five bad moments, an unknown kind, an impossible date, a blank capture time, an id that is not a ULID, a trip that does not exist, and asserts each is refused with nothing written.

A deletion is an op

The second code block shows what an append-only rule does to a delete. delete_moment appends moment.delete, and the mirror sets a deleted flag. restore_moment appends moment.restore, and the flag goes back. Nothing leaves the log. An edit works the same way: moment.edit carries a fresh revision_id and only the fields that may change (situation, place name, the two languages, trip, text), and the mirror patches only those. The value it replaced is still in the log, in the op before. A test edits, deletes and restores one moment, then reads the raw log: four lines, moment.create, moment.edit, moment.delete, moment.restore, and the restored moment still carries the edit.

Marks and links follow the same shape, with ids built from what they point at: a kept phrase is keep: plus its normalized text, a link is the moment plus the item's key. Removing one appends a remove op that sets active = 0; adding the same id again reactivates the row. Spark's house sign says it in one line: a correction, a deletion and a restoration are new ops.

Principle: Put validation in the helpers and writing in one place. A helper may refuse; only record may write. Then "what can change the corpus?" has a one-word answer you can check with a search.

Code

cwkSpark, backend/corpus.py: the one write path (verbatim, lines 236-257)·python

    def record(self, op: dict[str, Any]) -> dict[str, Any]:
        with self._lock:
            mirror = self._mirror()
            self.log.append(op)
            mirror.apply(op)
            if not self._stats.get("opaque"):
                mirror.set_meta("log_fingerprint", self.log.fingerprint())
            name = str(op.get("op"))
            moment_id = op.get("moment_id") or (
                op.get("id") if name.startswith("moment.") else None
            )
            if moment_id:
                body = dict(op)
                body.pop("op", None)
                # A turn goes out as the mirror holds it, status included: a
                # turn the engine starts on its own (a cold review) reaches a
                # client only through this event, and a client that keeps
                # listening while a turn runs must be able to tell.
                if name in ("turn.start", "turn.finish"):
                    body = mirror.turn(str(op.get("id"))) or body
                self.events.publish(str(moment_id), {"type": name, "record": body})
            return op

cwkSpark, backend/corpus.py: deleting and restoring are both appends (verbatim, lines 369-382)·python

    def delete_moment(self, moment_id: str) -> bool:
        with self._lock:
            if self.moment(moment_id) is None:
                return False
            self.record({"op": "moment.delete", "id": moment_id, "deleted_at": utcnow()})
            return True

    def restore_moment(self, moment_id: str) -> dict[str, Any] | None:
        with self._lock:
            current = self.moment(moment_id, include_deleted=True)
            if current is None or not current.get("deleted"):
                return None
            self.record({"op": "moment.restore", "id": moment_id, "restored_at": utcnow()})
            return self.moment(moment_id)

cwkSpark, backend/mirror.py: the seventeen ops and their folds (verbatim, lines 785-805)·python

_HANDLERS = {
    "moment.create": Mirror._moment_create,
    "moment.edit": Mirror._moment_edit,
    "moment.delete": Mirror._moment_delete,
    "moment.restore": Mirror._moment_restore,
    "media.add": Mirror._media_add,
    "result.add": Mirror._result_add,
    "mark.add": Mirror._mark_add,
    "mark.remove": Mirror._mark_remove,
    "link.add": Mirror._link_add,
    "link.remove": Mirror._link_remove,
    "trip.create": lambda self, op, seq: self._named_create("trips", op, seq),
    "trip.edit": lambda self, op, seq: self._named_edit("trips", TRIP_EDITABLE, op, seq),
    "purpose.create": lambda self, op, seq: self._named_create("purposes", op, seq),
    "purpose.edit": lambda self, op, seq: self._named_edit(
        "purposes", PURPOSE_EDITABLE, op, seq
    ),
    "thread.bind": Mirror._thread_bind,
    "turn.start": Mirror._turn_start,
    "turn.finish": Mirror._turn_finish,
}

Exercise

Take an app of yours that writes to its database from several places. List every write site. Then route all of them through one function that appends an op to a log, applies it to the database in a transaction, and stores a cheap fingerprint of the log. Write a test that appends an op straight to the log file (the state a crash between append and apply would leave), reopens the app, and asserts the database caught up. Finally, turn one delete into an op and show the deleted value is still in the log.

Hint

A fingerprint from file names, sizes and modification times is enough to notice that the log moved; it never has to read the content. Validate in the domain functions and raise before calling the write function, so a refused request leaves no line behind.

Every write, the same four steps

Why the log goes first

One lock, kept off the loop

Seventeen ops

A deletion is an op

Code

External links

Exercise

Progress

Comments 0