Lesson 01 of 05 · published

Selections Are Enforced After Each Batch

~14 min · libmypaint, selections, copy-on-write, fixed-point, architecture

Level 0Blank Canvas

0 XP0/37 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

An Engine That Paints Anywhere

libmypaint has no idea what a selection is. Its surface interface is a handful of calls: draw a dab here, tell me the color around there, begin a batch, end a batch. There is no mask parameter anywhere in it, and there never will be, because Ariandel vendors libmypaint byte for byte and never patches it (invariant 1). Yet Dad lassos a region, paints across its edge, and the paint stops at the line. Something has to make that true without the engine's cooperation.

The house sign states the rule for this whole track in one sentence: libmypaint never learns about selections, Clone, dual brushes, shapes, grain, stabilization or taper. Each of those is done in one of three places: before the mapping (the pen's input is moved, or the brush's radius and pressure scaled, before the engine sees them), inside the stamp (Ariandel's own C surface lays a dab where libmypaint asks for one: shaped tips, Photoshop's dual brush, a stamped grain), or after the batch (the engine's pixels are merged with something afterwards). Selections are the purest example of the third kind, so they go first.

The Batch Is the Smallest Thing the Host Sees

Painter.continueStroke feeds one frame's worth of coalesced touches at a time. It opens the surface with surface.begin(), hands each input to brush.stroke(to:on:), and closes it with surface.end(). That bracket is the batch.

Why not clip each dab as it lands? Because, for a round dab, there is no "as it lands" that Swift is ever told about. libmypaint's tiled surface queues round dabs per tile and renders a tile's queue later: inside mypaint_surface_end_atomic, or earlier when get_color flushes the tile to sample it. Swift does get called during a batch, but only as the tile store: libmypaint asks for a tile's buffer and hands it back, and Ariandel's own shaped stamp acquires the tiles it lays into. None of those calls says which dab is writing. To clip inside the dab you would have to patch libmypaint's dab renderer, which the vendoring rule forbids. Or you would have to teach every path that writes pixels about the mask: the round dab, Ariandel's own shaped stamp, each symmetry reflection. Right after surface.end() returns, every path's paint for the batch has landed and Ariandel holds all of it at once, so that is where the rule lives. The Painter's own doc comment says it plainly: "libmypaint never learns about selections; the rule lives here, once."

Recording What the Batch Overwrote

To undo a batch's paint outside the selection, you need what those tiles looked like before the batch. So when a stroke has a selection (or a grain, more on that below), continueStroke calls beginBatchRecording() on the layer's TileStore. From then on, the first write to any tile in this batch passes through noteWrite, which freezes the tile's current buffer and keeps it as that tile's before-state. The write then lands in a copy, because a frozen tile is never written. After surface.end(), endBatchRecording() hands back a TileSnapshot: every tile the batch touched, each with its state before the batch (or nil if the tile did not exist).

Three Kinds of Tile

keepInside (first code block) walks that snapshot and asks the selection one question per tile: coverageClass(of:).

.all: every pixel of the tile is fully selected. Keep the paint; there is nothing to do.
.none: no pixel is selected. Put the before tile back wholesale with setTile. No arithmetic, and the store simply shares the frozen buffer.
.partial: the selection's edge crosses this tile. Extract the tile's 64×64 slice of the 8-bit selection mask and run ard_fix15_masked_merge (second code block).

The kernel computes, per channel, before + (after − before) · m / 255, with rounding toward the nearest value. At m = 255 it skips the pixel; at m = 0 it copies the before pixel back. The channels are premultiplied 15-bit fixed point (Track 3), and a straight weighted blend of two premultiplied pixels is itself a valid premultiplied pixel. So the merge needs no un-premultiply, no division by alpha, nothing clever. That is a quiet benefit of libmypaint's format choice.

What It Costs

Copy-on-write makes an ordinary stroke cheap: the stroke's undo recording freezes a tile at its first write, the write lands in a copy, and every later batch writes that copy in place. One copy per touched tile per stroke.

The batch recording breaks that. It freezes again at the start of every batch, so the next batch's first write to the same tile lands in a fresh copy. With a selection, the cost is one copy per touched tile per batch. A tile is 64 × 64 pixels × 4 channels × 2 bytes, 32 KiB. A stroke that spends 100 batches scrubbing over the same 20 tiles copies 2,000 tiles, 62.5 MiB of memory traffic over its life, where the same stroke without a selection copies 20. The third code block is a compiled model of exactly this bookkeeping: four tiles, ten batches, 4 copies without a selection and 40 with one.

That is the trade Ariandel chose with its eyes open: a cost that scales with batches, paid only while a selection exists, in exchange for an engine that stays untouched and a rule that lives in one function.

When you cannot teach the engine a rule, find the first moment you hold its pixels again, and enforce the rule there, once. The batch boundary is that moment. Everything the engine did inside it is provisional until the host has had its say.

Two Honest Edges

First, the merge works per batch, and the before-state is per batch. A half-selected pixel keeps half of each batch's change, and the next batch starts from that result. Scrub long enough over a feathered edge and the paint keeps building there: the model's half-selected tile climbs from 1000 to 3510 over ten batches while the fully selected one reaches 6000. A feather limits how fast paint arrives, not how much can arrive.

Second, inside a single batch, libmypaint can read back its own not-yet-clipped paint. A smudging brush samples the canvas with get_color, which first flushes the dabs queued on those tiles. So within one batch, a sample taken near the edge can include paint that the merge is about to remove. The code admits this by construction; it is the price of clipping per batch rather than per dab.

Code

Putting back what landed outside (AriandelEngine/Painter.swift, verbatim)·swift

/// Puts back whatever the batch painted outside the selection.
private func keepInside(_ selection: SelectionMask, batch: TileSnapshot, on store: TileStore) {
    for (key, before) in batch {
        switch selection.coverageClass(of: key) {
        case .all:
            continue
        case .none:
            store.setTile(before, at: key)
        case .partial:
            guard store.tile(at: key) != nil else { continue }
            let tile = store.writableTile(at: key)
            coverage.withUnsafeMutableBufferPointer { buffer in
                selection.tileCoverage(key, into: buffer.baseAddress!)
                ard_fix15_masked_merge(tile.pixels, before?.pixels, buffer.baseAddress!, Tile.pixelCount)
            }
        }
    }
}

The masked merge kernel (CAriandelSurface/ariandel_kernels.c, verbatim)·c

void
ard_fix15_masked_merge(uint16_t *tile, const uint16_t *before, const uint8_t *mask, size_t pixels)
{
    for (size_t p = 0; p < pixels; p++) {
        const int32_t m = mask[p];
        if (m == 255) {
            continue;
        }
        uint16_t *t = tile + p * 4;
        if (m == 0) {
            if (before) {
                memcpy(t, before + p * 4, 4 * sizeof(uint16_t));
            } else {
                t[0] = t[1] = t[2] = t[3] = 0;
            }
            continue;
        }
        for (int c = 0; c < 4; c++) {
            const int32_t b = before ? before[p * 4 + c] : 0;
            const int32_t delta = (int32_t)t[c] - b;
            const int32_t value = b + (delta * m + (delta >= 0 ? 127 : -127)) / 255;
            t[c] = (uint16_t)clamp_one(value);
        }
    }
}

A minimal sketch of the batch bookkeeping, counting copies (one-channel, four-pixel tiles; compiled and run with swiftc in Swift 6 mode)·swift

// A minimal sketch of Painter.keepInside: copy-on-write tiles, a stroke
// recording, a batch recording that freezes again every batch, and the
// masked merge. One tile here is four one-channel pixels.

final class Buffer {
    var pixels: [Int32]
    private(set) var isFrozen = false
    init(_ pixels: [Int32]) { self.pixels = pixels }
    @discardableResult func freeze() -> Buffer { isFrozen = true; return self }
}

final class Store {
    var tiles: [Int: Buffer] = [:]
    var copies = 0
    private var recording: [Int: Buffer?]?
    private var batchRecording: [Int: Buffer?]?

    func writable(_ key: Int) -> Buffer {
        note(key)
        if let tile = tiles[key] {
            guard tile.isFrozen else { return tile }
            let copy = Buffer(tile.pixels)
            copies += 1
            tiles[key] = copy
            return copy
        }
        let tile = Buffer([0, 0, 0, 0])
        tiles[key] = tile
        return tile
    }

    func setTile(_ tile: Buffer?, at key: Int) {
        note(key)
        tiles[key] = tile
    }

    func beginRecording() { recording = [:] }
    func beginBatch() { batchRecording = [:] }
    func endBatch() -> [Int: Buffer?] {
        defer { batchRecording = nil }
        return batchRecording ?? [:]
    }

    // As TileStore.noteWrite: holding the current buffer freezes it, so the
    // write that follows lands in a copy.
    private func note(_ key: Int) {
        if recording != nil, recording?.index(forKey: key) == nil {
            recording?.updateValue(tiles[key]?.freeze(), forKey: key)
        }
        if batchRecording != nil, batchRecording?.index(forKey: key) == nil {
            batchRecording?.updateValue(tiles[key]?.freeze(), forKey: key)
        }
    }
}

// ard_fix15_masked_merge's arithmetic: before + (after - before) * m / 255.
func maskedMerge(_ tile: Buffer, before: Buffer?, mask: [Int32]) {
    for p in 0..<tile.pixels.count {
        let m = mask[p]
        if m == 255 { continue }
        let b = before?.pixels[p] ?? 0
        let delta = tile.pixels[p] - b
        tile.pixels[p] = b + (delta * m + (delta >= 0 ? 127 : -127)) / 255
    }
}

// Per tile: .all keeps the paint, .none puts the before tile back, .partial merges.
let masks: [Int: [Int32]] = [
    0: [255, 255, 255, 255],  // inside
    1: [255, 255, 0, 0],      // across the edge
    2: [0, 0, 0, 0],          // outside
    3: [128, 128, 128, 128],  // a feathered edge
]

func paint(batches: Int, selection: Bool) -> Store {
    let store = Store()
    for key in 0..<4 { store.tiles[key] = Buffer([1000, 1000, 1000, 1000]) }
    store.beginRecording()
    for _ in 0..<batches {
        if selection { store.beginBatch() }
        for key in 0..<4 {  // libmypaint paints every tile, selection or not
            let tile = store.writable(key)
            for p in 0..<4 { tile.pixels[p] += 500 }
        }
        guard selection else { continue }
        for (key, before) in store.endBatch() {
            let mask = masks[key]!
            if mask.allSatisfy({ $0 == 255 }) { continue }
            if mask.allSatisfy({ $0 == 0 }) { store.setTile(before, at: key); continue }
            maskedMerge(store.writable(key), before: before, mask: mask)
        }
    }
    return store
}

for selection in [false, true] {
    let store = paint(batches: 10, selection: selection)
    print(selection ? "with a selection:" : "no selection:   ", "copies =", store.copies)
    for key in 0..<4 { print("  tile \(key):", store.tiles[key]!.pixels) }
}

// Output:
// no selection:    copies = 4
//   tile 0: [6000, 6000, 6000, 6000]
//   tile 1: [6000, 6000, 6000, 6000]
//   tile 2: [6000, 6000, 6000, 6000]
//   tile 3: [6000, 6000, 6000, 6000]
// with a selection: copies = 40
//   tile 0: [6000, 6000, 6000, 6000]
//   tile 1: [6000, 6000, 1000, 1000]
//   tile 2: [1000, 1000, 1000, 1000]
//   tile 3: [3510, 3510, 3510, 3510]

External links

Exercise

Pick an image operation you do not control, such as a blur or a sharpen from Pillow, Core Image or any library you use. Apply it to a whole test image, then clip its effect to a soft-edged mask afterwards with before + (after − before) · m / 255. Then apply the operation five times in a row, clipping after each one, and compare a pixel at 50% mask against one at 100%. Does the half-masked pixel converge on the fully masked result?

Hint

Keep the before image of each pass, not just the first one; that is what makes the clip per pass. With an 8-bit mask, round the division (add 127 for a positive delta, subtract it for a negative one) or a long sequence of passes drifts by a level each time.

Progress

Progress is local-only — sign in to sync across devices.

← ✏️ From Pencil to PaintQuiz · 4 questions Next →Clone Through Coverage

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.