Skip to content
C.W.K.
Stream
Lesson 02 of 05 · published

An Atomic Package

~15 min · file-formats, persistence, atomic-writes, autosave, swift

Level 0Blank Canvas
0 XP0/37 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

A Folder That Is One Document

An Ariandel painting on disk is a package: a folder named <id>.ariandel in the app's Documents/Artworks, which the Files app shows as a single document. Inside (first code block):

  • Artwork.json, the manifest: canvas size, background, the layer stack and every layer's properties.
  • Layers/<layer-id>.tiles, one file per layer, every tile in the engine's native format, which is libmypaint's premultiplied fix15. A mask's paint is a layer of its own, stored the same way under the mask's id.
  • Selections/<id>.selection, one per selection saved on purpose.
  • Preview.png (2048 px on the long edge) and Thumbnail.png (512 px), flattened at save time for the gallery and Quick Look.

Two rules make the format trustworthy. The tiles are stored losslessly in the format the engine paints in, so a reopened artwork is bit-identical; the engine suite saves a two-layer painting, reopens it and compares every tile's bytes. And the PNGs are projections: nothing ever reads pixels back from them. Lose them and the painting is untouched.

What is not in the package matters as much. The undo history is a session's, not the document's; the same test checks that a reopened artwork cannot undo. The live selection is not saved, only selections saved on purpose. And three kinds of data that belong to an artwork live beside the library, never inside the package a save writes: the time-lapse's video segments (Application Support/Timelapse/<id>/), the Generate panel's candidates (Application Support/Generations/<id>/) and the reference picture. The time-lapse code says why in one sentence: so saving an artwork never rewrites its video, and the Files app shows only artworks. A save writes the whole package every time, and a growing video inside it would make every save as big as the film. Sharing is the one exception. A shared copy carries the time-lapse and the reference picture inside the package, as Timelapse and Reference entries, and the receiving iPad moves both back out beside its own library before the artwork joins its gallery. Generate's candidates never travel.

The Tile File, and Keys That Arrive Later

A .tiles file is a small binary format: the magic ARDT, a version, a tile count, then for each tile its x, y, a flags word, a length and the payload, every integer little-endian. Flag bit 0 says the payload is LZFSE-compressed, and it is set only when compression actually made the bytes smaller. A tile of flat color shrinks to almost nothing; a tile of noise would grow, so it is stored raw. Paying a few bytes for the flag is cheaper than ever storing a compressed tile larger than the original.

The manifest carries formatVersion 1 and has kept it through every feature since. Clipping, lock, reference, the drawing guide, groups, text, masks, animation, saved selections and DPI all arrived as optional keys, and an absent key means the old behavior. The writer even leaves a key out when it holds the default (clipped is written only when true; dpi only when it is not 300). The reader is strict in the other direction: a manifest whose formatVersion is newer than it knows is refused, and a damaged tile file or manifest is refused rather than half-loaded. A saved selection that cannot be read is dropped, and the picture still opens.

Write Beside, Then Swap

Saving never edits the package in place. ArtworkFile.write creates a hidden sibling, .<name>.<uuid>.saving, in the same folder, writes the entire package into it, and only then swaps it in: replaceItemAt when a package already exists, moveItem for the first save. If anything throws on the way (a full disk, an encoder failure), the catch removes the staging folder and rethrows. The old package was never opened for writing, so it is exactly what it was.

Writing the whole package also removes a class of bug for free. When you delete a layer, its .tiles file cannot linger, because the new package never contained it. The engine test for this saves two layers, deletes one, saves again, and checks that Layers/ holds one file and that no .saving folder is left beside it. The second code block is a sketch of the same moves you can run: its second save "crashes" after one file, the listing shows the old package unchanged with no leftovers, and its third save drops a layer whose file then disappears.

Be exact about one edge. The sketch's crash is a thrown error, which the catch cleans up. A process killed in the middle of the write never reaches the catch, so its hidden staging folder stays behind. The package beside it is still the previous version, and the gallery never lists the leftover (it only lists names ending in .ariandel), but nothing in the code sweeps it up either.

Package writes go through one actor, ArtworkWriter, so two saves never interleave on disk. The freezing happens before that, on the main thread: ArtworkArchive(artwork) freezes every tile (Track 3 told that story), the two preview flattens run there too, PNG encoding runs in a detached utility task, and the package write runs on the actor.

When It Saves

There are five reasons a save starts: created, autosave, closing, leavingForeground and explicit. The interesting ones are the third code block.

  • Autosave is a debounce. Every change except a selection-only one calls scheduleAutosave(), which cancels the pending save and starts a new four-second countdown. If a stroke or an editing session (a transform, an adjustment, Liquify, text) is still open when it fires, it reschedules itself instead of saving half a gesture. If the revision has not moved since the last save, it does nothing.
  • Leaving the canvas saves with closing, and the gallery refreshes only after that save lands, so it never shows the picture as it was before.
  • Leaving the foreground (any scene phase other than active) saves with leavingForeground. These two reasons first end a stroke in progress, commit any open session and finish the time-lapse segment, because there may be no later chance.
  • Every save asks iOS for background time with beginBackgroundTask and ends it when the write finishes. It matters most for the save that starts as the app leaves the screen, which iOS could otherwise suspend half-written.

The architecture describes autosave as running "periodically while he paints". Teach the code instead, because the difference is real: a debounce has no upper bound. Painting with no four-second pause restarts the countdown with every change, so the autosave waits for the first pause. Painters pause constantly, so the effect looks periodic, but the guarantee is "four seconds after the last change", not "every so often".

Never edit the only copy in place. Build the complete new version beside the old one, then make the switch a single step the file system does for you. Whatever goes wrong before the switch costs you the new version; nothing that goes wrong can cost you the old one.

Code

The package, the atomic write and the tile file (AriandelEngine/ArtworkFile.swift, trimmed: the per-layer loop, manifest and preview writes are elided)·swift
/// The `.ariandel` document package.
///
/// ```
/// <id>.ariandel/
///   Artwork.json            manifest: canvas, background, layer stack, properties
///   Layers/<layer-id>.tiles every tile in the engine's native format, LZFSE-compressed
///   Layers/<mask-id>.tiles  a layer mask's paint, the same way
///   Selections/<id>.selection a saved selection's coverage, LZFSE-compressed
///   Preview.png             the flattened picture (projection, regenerable)
///   Thumbnail.png           small flattened picture for the gallery (projection)
/// ```
///
/// Tiles are stored losslessly in libmypaint's own format, so a reopened
/// artwork is bit-identical. PNG files are projections for galleries and
/// Quick Look; nothing reads pixels back from them.
public enum ArtworkFile {
    // ...

    /// Writes the package atomically: everything goes into a sibling temporary
    /// package first, which then replaces `url` in one step.
    public static func write(_ archive: ArtworkArchive, preview: Data? = nil, thumbnail: Data? = nil, to url: URL) throws {
        let manager = FileManager.default
        let parent = url.deletingLastPathComponent()
        try manager.createDirectory(at: parent, withIntermediateDirectories: true)
        let staging = parent.appendingPathComponent(".\(url.lastPathComponent).\(UUID().uuidString).saving",
                                                    isDirectory: true)
        try manager.createDirectory(at: staging.appendingPathComponent(layersDirectory), withIntermediateDirectories: true)
        do {
            // ... every layer, mask and saved selection, then Artwork.json,
            // Preview.png and Thumbnail.png, all written into staging ...
            if manager.fileExists(atPath: url.path) {
                _ = try manager.replaceItemAt(url, withItemAt: staging)
            } else {
                try manager.moveItem(at: staging, to: url)
            }
        } catch {
            try? manager.removeItem(at: staging)
            throw error
        }
    }

    /// `ARDT`, version, count, then per tile: x, y, flags (bit 0 = LZFSE), length, payload.
    /// All integers little-endian.
    static func encodeTiles(_ tiles: [(key: TileKey, tile: FrozenTile)]) throws -> Data {
        var out = Data(tileMagic)
        out.appendLittleEndian(UInt32(1))
        out.appendLittleEndian(UInt32(tiles.count))
        for (key, tile) in tiles {
            let bytes = tile.buffer.data
            let compressed = try (bytes as NSData).compressed(using: .lzfse) as Data
            let useCompressed = compressed.count < bytes.count
            out.appendLittleEndian(key.x)
            out.appendLittleEndian(key.y)
            out.appendLittleEndian(UInt32(useCompressed ? 1 : 0))
            let payload = useCompressed ? compressed : bytes
            out.appendLittleEndian(UInt32(payload.count))
            out.append(payload)
        }
        return out
    }
A minimal sketch: stage, swap, survive a failed save, and lose a deleted layer's file (compiled and run with swiftc in Swift 6 mode on macOS)·swift
// A minimal sketch of an atomic package save in the shape of ArtworkFile.write:
// stage the whole package beside the target, then swap it in with one call.
// Build and run: swiftc -swift-version 6 package.swift -o package && ./package
import Foundation

struct Fail: Error {}

func writePackage(layers: [String: Data], to url: URL, failAfter: Int? = nil) throws {
    let manager = FileManager.default
    let parent = url.deletingLastPathComponent()
    let staging = parent.appendingPathComponent(".\(url.lastPathComponent).\(UUID().uuidString).saving",
                                                isDirectory: true)
    try manager.createDirectory(at: staging.appendingPathComponent("Layers"), withIntermediateDirectories: true)
    do {
        var written = 0
        for (id, raw) in layers.sorted(by: { $0.key < $1.key }) {
            if let failAfter, written == failAfter { throw Fail() }      // a crash, simulated
            // LZFSE only when it actually shrinks the bytes; flag says which.
            let packed = try (raw as NSData).compressed(using: .lzfse) as Data
            let useCompressed = packed.count < raw.count
            var file = Data([useCompressed ? 1 : 0])
            file.append(useCompressed ? packed : raw)
            try file.write(to: staging.appendingPathComponent("Layers/\(id).tiles"))
            written += 1
        }
        try Data(#"{"formatVersion":1}"#.utf8).write(to: staging.appendingPathComponent("Artwork.json"))
        if manager.fileExists(atPath: url.path) {
            _ = try manager.replaceItemAt(url, withItemAt: staging)
        } else {
            try manager.moveItem(at: staging, to: url)
        }
    } catch {
        try? manager.removeItem(at: staging)      // the old package was never touched
        throw error
    }
}

func listing(_ dir: URL) -> [String] {
    (try? FileManager.default.subpathsOfDirectory(atPath: dir.path).sorted()) ?? []
}

let root = FileManager.default.temporaryDirectory.appendingPathComponent("atomic-\(UUID().uuidString)")
try FileManager.default.createDirectory(at: root, withIntermediateDirectories: true)
let url = root.appendingPathComponent("study.ariandel", isDirectory: true)

let flat = Data(repeating: 0x40, count: 32_768)                         // compresses well
let noise = Data((0..<32_768).map { _ in UInt8.random(in: 0...255) })  // does not
try writePackage(layers: ["A": flat, "B": noise, "C": flat], to: url)
print("first save:", listing(url))
for name in ["A", "B"] {
    let file = try Data(contentsOf: url.appendingPathComponent("Layers/\(name).tiles"))
    print("  \(name).tiles flag \(file[0]), \(file.count) bytes")
}

// Second save drops layer C and dies after one layer file.
do { try writePackage(layers: ["A": flat, "B": flat], to: url, failAfter: 1) } catch { print("second save failed") }
print("after the failure:", listing(url))
print("leftovers beside it:", listing(root).filter { $0.hasSuffix(".saving") })

// Third save succeeds: layer C's file is gone because the package was replaced whole.
try writePackage(layers: ["A": flat, "B": flat], to: url)
print("third save:", listing(url))
What starts a save, and the one writer (AriandelMobile, three files, verbatim excerpts)·swift
// App/AriandelApp.swift: leaving the foreground
        .onChange(of: scenePhase) { _, phase in
            if phase != .active {
                studio.saveOpenArtwork(reason: .leavingForeground)
            }
        }

// Canvas/CanvasModel.swift: every change but a selection-only one restarts the countdown
        if !change.isSubset(of: [.selection, .selectedArea]) {
            scheduleAutosave()
            timelapse?.noteChange()
            generationModel?.noteChange()
        }

    private func scheduleAutosave() {
        autosaveTask?.cancel()
        autosaveTask = Task { [weak self] in
            try? await Task.sleep(for: .seconds(4))
            guard !Task.isCancelled else { return }
            self?.save(reason: .autosave)
        }
    }

// Library/ArtworkLibrary.swift: one writer
/// Serializes package writes so two saves never interleave on disk.
actor ArtworkWriter {
    func write(_ archive: ArtworkArchive, preview: Data?, thumbnail: Data?, to url: URL) throws {
        try ArtworkFile.write(archive, preview: preview, thumbnail: thumbnail, to: url)
    }
}

External links

Exercise

Find the save path of an app you maintain (a config file, a document, a cache). Answer three questions in writing: if the process is killed halfway through the write, what is on disk? If an item is removed from the data, can its old file survive the next save? And what starts a save: a timer, a debounce, or an event? Then run the sketch, and change it so the second save is interrupted by a real kill (call exit(1) instead of throwing) and look at what is left beside the package.
Hint
Writing to a temporary file and renaming it over the original covers a single file. A folder of files needs the whole folder staged, or a manifest that is written last and names only the files that belong. With exit(1), the catch never runs, so expect the old package intact and a hidden .saving folder beside it.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.