Skip to content
C.W.K.
Lesson 01 of 06 · published

The Log Is the Truth

~15 min · event-log, jsonl, durability, encryption

Level 0First Word
0 XP0/47 lessons0/13 achievements
0/100 XP to next level100 XP to go0% complete

A rule that needs a file format

On the day Spark was designed, Dad settled two translation modes, practical and high quality, and then one rule over both of them:

실용이든 고품질이든, 맥락은 쌓여야 해. 한번 쓰고 버리는 게 아니라 모든게 코퍼스 자료야.

"Practical or high quality, the context has to build up. Nothing is used once and thrown away; everything is corpus material."

A sentence like that is easy to agree with and easy to break. A cache gets cleared, a table gets migrated, a row gets updated in place, and the earlier reading is gone. So Spark keeps the rule in the shape of its storage. The engine has exactly one ground truth: an append-only log of operations. Every page you see, a moment, a trip, a search result, is computed from that log and can be computed from it again.

Three things on disk

Path under the data rootWhat it holdsStanding
corpus/<YYYY-MM>.jsonlEvery op, one JSON object per line, one file per UTC month of receiptGround truth
media/<moment_id>/<media_id>/<filename>Photos, recordings and documents exactly as uploaded, never re-encodedKept bytes
mirror.sqlite3The SQLite view every read goes throughDerived, never truth

The data root is not written in the code either. It comes from the family registry through the host's profile, and an environment variable is the explicit override that tests use. One process writes the log: the engine. Nothing else appends to it, and nothing ever rewrites it.

Why the month of receipt

A file per month keeps each file small and makes order obvious: 2027-03.jsonl sorts before 2027-04.jsonl by name alone. The month is the one in which the engine received the op, in UTC, never the month the photo was taken. A capture that sat on the phone for a week still lands at the end of the current file, so the log only ever grows at its tail.

The capture's own time is a separate fact, carried inside the record: captured_at, timezone and local_date, supplied by the client that saw it happen. The engine adds received_at from its own clock, and the helper that produces it says so in its docstring: server receipt time, never capture truth. Keeping the two apart is what later lets Spark tell a late capture from a fresh one.

The append

The first code block is the whole write, and the read that mirrors it. Three details carry the weight:

  • The line is veiled first. encrypt_for_write turns the JSON into a Fernet token when the key is present and passes it through plain when it is not.
  • A torn tail gets a newline before the next op. If an earlier append was cut short before its newline, by a power loss or a killed process, the next op would glue itself onto that fragment. Both would read back as one undecodable line, and replay would throw away a good record along with the broken one. Checking the last byte costs one seek.
  • fsync before returning. When append returns, the bytes have reached the disk. The caller, the one write path you meet in the next lesson, touches the mirror only after that.

A test pins the torn case. It writes half an op by hand, appends a real one after it, deletes the mirror and reopens. The real moment is there, and health reports exactly one corrupt line. The test's fragment is plain JSON. With the veil on, a torn tail is a cut Fernet token and counts as opaque, with the consequences shown in the last section.

Reading it back

replay walks the month files in name order and each file in line order. It never sorts by id, on purpose. Spark's ids are ULIDs, which begin with a millisecond timestamp, so they look sortable. Two ids minted in the same millisecond differ only in random low bits, though, and sorting them would scramble two ops whose order was known. The order of lines in the files is the order things happened to the engine, and it is the only order replay trusts.

A line replay cannot read is skipped and counted, never fatal. Its first character decides the bucket. A line that starts with { and still fails to parse is corrupt: a torn or mangled plain line. Any other failure is opaque. Usually that is a veiled line under a key this process does not hold right now. A torn veiled line lands there too, since a cut token still starts with g.

The veil, and what it is for

Each line is encrypted on its own with Fernet, through the family kit's Veil, the same pattern cwkPippa uses for its conversation logs. Line by line, because an append-only file has to stay appendable; encrypting the whole file would mean rewriting it for every op. Reading needs no flag. A Fernet token starts with g in base64 and a plain JSON line starts with {, so one file may hold both, and a plain line written before the key existed still reads afterwards.

The passphrase lives in the macOS Keychain on the service host. The threat model is modest by design: a preview thumbnail, a glance over a shoulder, a backup copy landing somewhere new. It does not stop someone who holds both the disk and the Keychain. The policy is graceful too. A Mac without the key writes plain text, logs one warning, and the health route says "veil": false, so a missing key is loudly visible and never fatal. Media files and the mirror stay plain working surfaces. Only the JSONL ground truth is veiled.

A blind mirror says it is blind

The dangerous case runs the other way: the key existed when the lines were written and cannot be read now, because the keychain is locked or the files are a copy on another Mac. Replay skips those lines, so the mirror it builds is missing moments while looking perfectly healthy. The second code block is the engine refusing to trust such a mirror. It leaves the log fingerprint unset, so the next boot or the next explicit rebuild tries again, and health reports degraded: true with the number of opaque lines.

One more guard rides on the same signal. At startup the engine seeds Dad's six translation purposes if the log has never had them. While any line is opaque it seeds nothing, because a mirror built blind may only look empty, and seeding then would write rows that already sit in the lines it cannot read.

Principle: A view built from partial evidence has to say it is partial. Spark turns "some of the log was unreadable" into a number on the health route, and into a refusal to seed rows that a partial view would only think are missing.

Code

cwkSpark, backend/store.py: the append and the replay (verbatim, lines 71-104)·python
    def append(self, record: dict[str, Any]) -> None:
        month = datetime.now(timezone.utc).strftime("%Y-%m")
        path = self.directory / f"{month}.jsonl"
        path.parent.mkdir(parents=True, exist_ok=True)
        line = veil.encrypt_for_write(json.dumps(record, ensure_ascii=False))
        # A torn last line (an append cut short before its newline) would
        # otherwise swallow this op: both would read back as one undecodable
        # line and replay would drop the intact record with the torn one.
        prefix = "" if _ends_with_newline(path) else "\n"
        with path.open("a", encoding="utf-8") as fh:
            fh.write(prefix + line + "\n")
            fh.flush()
            os.fsync(fh.fileno())

    def replay(self, stats: dict[str, int] | None = None) -> Iterator[dict[str, Any]]:
        """Every op in capture order. Lines this process cannot read are
        skipped and counted in ``stats``: ``opaque`` for veiled lines under
        a key it does not hold right now, ``corrupt`` for plain lines that
        are not JSON."""
        for path in self.files():
            with path.open(encoding="utf-8") as fh:
                for raw in fh:
                    raw = raw.strip()
                    if not raw:
                        continue
                    try:
                        op = json.loads(veil.maybe_decrypt_line(raw))
                    except json.JSONDecodeError:
                        if stats is not None:
                            key = "corrupt" if raw.startswith("{") else "opaque"
                            stats[key] = stats.get(key, 0) + 1
                        continue
                    if isinstance(op, dict) and op.get("op"):
                        yield op
cwkSpark, backend/corpus.py: a rebuild that met unreadable lines stays stale, and health says degraded (verbatim, lines 197-227)·python
    def _rebuild_locked(self, fingerprint: str) -> None:
        assert self.mirror is not None
        stats: dict[str, int] = {}
        count = self.mirror.rebuild(self.log.replay(stats))
        self._stats = stats
        # A mirror built while veiled lines were unreadable is missing
        # moments. Leave its fingerprint unset so the next boot, or the
        # next explicit rebuild, tries again instead of trusting it.
        if stats.get("opaque"):
            log.warning(
                "mirror rebuilt from %d ops with %d unreadable veiled lines; "
                "it stays marked stale until they read",
                count,
                stats["opaque"],
            )
            self.mirror.set_meta("log_fingerprint", None)
        else:
            self.mirror.set_meta("log_fingerprint", fingerprint)
        log.info("mirror rebuilt from %d ops", count)

    def health(self) -> dict[str, Any]:
        with self._lock:
            mirror = self._mirror()
            counts = mirror.counts()
            opaque = self._stats.get("opaque", 0)
            return {
                **counts,
                "opaque_lines": opaque,
                "corrupt_lines": self._stats.get("corrupt", 0),
                "degraded": bool(opaque),
            }

External links

Exercise

Build a small append-only log for an app of your own: one JSON object per line, one file per month of receipt, fsync on every append. Then attack it. Append half a record by hand with no newline, append a real record, and replay. Make the reader count what it skipped in two buckets (torn versus unreadable) and check that the real record survived. Finally, add a health field that turns true whenever the reader skipped a line it should have been able to read, and make one piece of startup work refuse to run while that field is true.
Hint
Before each write, read the file's last byte; if it is not a newline, write one first. Bucket failures by the first character: a line that starts with { and fails to parse is torn, anything else belongs to a key or format this process does not hold. If you encrypt lines, a torn token lands in the second bucket, so decide what your health field should say about it.

Progress

Progress is local-only — sign in to sync across devices.
← 🌐 A Translator That RemembersQuiz · 4 questionsNext →One Door: Corpus.record
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.