Files Before Rows

The promise in the docstring

The store opens with a sentence that works as a contract: files precede their DB rows, so an interrupted write may leave an unreferenced original, never a missing original behind a committed asset. Read it as a choice between two failures. One is a file with no row: some wasted disk, invisible to every screen, harmless. The other is a row with no file: a library card that opens nothing, a set whose frame cannot be published, a reference profile that still lists the image while every read of its bytes answers 503 with The original image is unavailable. Avatars picks the harmless failure, and it does that by putting the work in order.

A name that is also a checksum

save_asset first asks Pillow whether the bytes are an image at all. It calls verify(), then opens the bytes a second time to read width, height and format, since Pillow's documentation says to reopen an image after verifying it. The extension comes from the format Pillow detected, never from the uploaded filename. PNG, JPEG, WebP, GIF and HEIC are accepted; anything else is refused with Use PNG, JPEG, WebP, GIF or HEIC.

Then the bytes are hashed, and the SHA-256 becomes the filename: <sha256>.png, <sha256>.heic. Clips follow the same rule as <sha256>.mp4. The name is the content. Two uploads of the same bytes land on the same file, and any file can be checked against its own name at any time.

Written once, checked on reuse

The docstring of write_original is one line: content-addressed and written once; an existing file must still match. It has two branches, and neither ever opens an existing original for writing.

The file exists. Read it, hash it, compare with the name. A match means the bytes are already safe, and nothing is written. A mismatch means the disk no longer holds what the name promises, and the store refuses with Stored original failed its checksum. as a 503. It does not repair by overwriting. I like that: a quiet overwrite would hide the fact that the disk lost something.
The file is new. Write to a temporary sibling, .<uuid>.tmp, opened with mode "xb", which fails if the path exists instead of truncating it. Write, flush Python's buffer, call os.fsync so the operating system pushes the bytes out of its cache, then os.replace the temporary file onto the final name. On macOS and Linux that rename is atomic: a reader sees no file or the whole file, never half of one.

The order, gap by gap

Only after write_original returns does save_asset call add, which inserts the asset row. Walk a crash through each gap:

The engine dies	What is left	What happens next
while the temporary file is written	a hidden `.tmp` file	nothing; no row points at it
after the rename, before the insert commits	a complete original with no row	nothing visible; uploading the same bytes again finds the file, checks its hash and reuses it
after the insert commits	the original and its row	the library shows it

No line in that table ends with a row pointing at nothing. That is the whole design, and it costs two system calls.

Principle: When two stores must agree, write the one that is referenced before the one that refers. A crash then leaves garbage, never a dangling pointer.

The original is never touched again

Everything that looks like a change to an image is a new file somewhere else.

The upload route reads at most 100 MB, and its comment is precise about what that is: a transport size bound, not a resizing operation. Native originals remain exactly the uploaded bytes.
A HEIC from a phone stays HEIC on disk. Browsers and generators get a lossless PNG derivative from browser_image, whose docstring ends with the promise that the archived bytes never change. A test asserts the stored file is identical to the upload before and after that derivative is made.
Thumbnails are WebP files in thumbnails/, named by the original's hash and rendered on first request.
A clip's poster frame sits beside them, and the clip's docstring calls it a regenerable derivative.

One more rule rides on content addressing. When an upload's bytes match an asset the store already knows, the new asset inherits every subject of the old one and lists it as a reference. Uploading the same file under another name cannot shed whose face it carries. How that ancestry is walked, and why it protects a soul, belongs to Protection Follows the Pixels, later in this quest.

Why so strict? Ember keeps the first copy of every generated image, and Avatars keeps a second, durable one so that cleaning up Ember's history can never take a chosen avatar with it. A second copy is worth exactly as much as its bytes are true. Writing them once and checking them on every reuse is how the copy stays worth keeping.

Code

cwkAvatars, backend/store.py: save_asset, verify, hash, write the file, then add the row (verbatim, lines 153-172)·python

    def save_asset(self, data: bytes, *, label: str, subjects: list[str], origin: str, references: list[str] | None = None, **metadata) -> dict:
        try:
            with Image.open(BytesIO(data)) as im:
                im.verify()
            with Image.open(BytesIO(data)) as im:
                width, height, format = im.width, im.height, im.format
        except (OSError, UnidentifiedImageError, Image.DecompressionBombError) as exc:
            raise StudioError("The original is not a supported image.") from exc
        ext = {"PNG": "png", "JPEG": "jpg", "WEBP": "webp", "GIF": "gif", "HEIF": "heic"}.get(format)
        if not ext:
            raise StudioError("Use PNG, JPEG, WebP, GIF or HEIC.")
        digest = hashlib.sha256(data).hexdigest()
        # Reimports retain every known subject; an upload cannot launder a
        # protected avatar by stripping its filename or embedded metadata.
        known = [a for a in self.list("asset") if a["sha256"] == digest]
        subjects = sorted(set(subjects).union(*(set(a["subjects"]) for a in known)))
        refs = list(dict.fromkeys([*(references or []), *(a["id"] for a in known)]))
        filename = f"{digest}.{ext}"
        self.write_original(filename, data, digest)
        return self.add("asset", label=label[:200], subjects=subjects, origin=origin, references=refs, filename=filename, sha256=digest, width=width, height=height, mime=Image.MIME[format], bytes=len(data), archived=False, **metadata)

cwkAvatars, backend/store.py: write_original, written once and checked on reuse (verbatim, lines 174-187)·python

    def write_original(self, filename: str, data: bytes, digest: str) -> Path:
        """Content-addressed and written once; an existing file must still match."""
        path = self.root / "originals" / filename
        if path.exists():
            if hashlib.sha256(path.read_bytes()).hexdigest() != digest:
                raise StudioError("Stored original failed its checksum.", 503)
        else:
            temporary = path.with_name(f".{identity()}.tmp")
            with temporary.open("xb") as handle:
                handle.write(data)
                handle.flush()
                os.fsync(handle.fileno())
            os.replace(temporary, path)
        return path

Exercise

Write save_original(root, data) for your own app. Name the file by its SHA-256. If the target exists, hash it and refuse on a mismatch. Otherwise write a temporary file opened exclusively, fsync it, rename it into place and fsync the directory. Then insert the database row. Write a test that raises an exception between the rename and the insert, and assert that the file exists, that no row exists, and that a second call with the same bytes completes cleanly.

Hint

To fsync a directory on macOS or Linux, open it with os.open(path, os.O_RDONLY) and call os.fsync on that descriptor. For the crash test, monkeypatch your insert to raise: you are proving the order of the writes, not simulating a power cut.

The promise in the docstring

A name that is also a checksum

Written once, checked on reuse

The order, gap by gap

The original is never touched again

Code

External links

Exercise

Progress

Comments 0