~13 min · precision, quantization, premultiplied-alpha, metal, code-review
Level 0Blank Canvas
0 XP0/37 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
Two Precisions of One Painting
Every Ariandel layer exists twice. The truth is its tiles: premultiplied RGBA in 15-bit fixed point, 32,768 steps from nothing to full. The mirror is one slice of the GPU's layer array: premultiplied RGBA in 8 bits, 255 steps. The mirror exists to be looked at, and for looking it is plenty: whatever reaches the screen has passed through 8 bits anyway.
The trouble is what premultiplied does to those 255 steps. A premultiplied channel stores color times alpha, so faint paint gets only a sliver of the range. Take the engine test's own value: a soft blue, straight color (0.3, 0.6, 0.9), at alpha 0.008. In fix15 its alpha is 262 steps and its red about 79, plenty to recover the color by dividing. In 8 bits its alpha is 2, and every channel can only be 0, 1 or 2. Divide back and the color is (0.5, 0.5, 1.0). The hue has moved; the gentle blue is now a lavender. That loss is invisible on screen, because at alpha 2/255 nobody sees the color at all. It only becomes visible if something later makes that paint stronger.
And painting is full of faint paint. Airbrushes build up from it; soft brush edges are made of it; glazes are nothing else.
The Merge That Read the Mirror
On Ariandel's first evening, Merge Down was written the obvious way. The renderer already had both layers on the GPU and a compositor that knew every blend mode, so the merge synchronized the mirror, composited the two layers' mirror slices into a float texture the size of the canvas, converted the result back to fix15 and wrote it into the lower layer. Every pixel went through the 8-bit slice on its way.
The flaw was not the 8 bits. The display reads the same 8 bits on every frame and is fine. The flaw was the direction. The display's 8-bit picture is thrown away on the next frame and rebuilt from the tiles. A merge's result is written into the tiles: the mirror's rounding became the truth, permanently, and every later operation started from it. The 6479f1f review found it by reading, a few hours later, and recorded it in one line: Merge Down "went through the 8-bit premultiplied display mirror, erasing the color of faint paint in the lower layer."
The Merge That Builds Its Own
Today's mergedTiles (first code block) never touches the mirror. It works out the region the two layers actually cover, the union of their tiles, and builds two textures just for that region:
A small rgba16Unorm 2D array: slice 0 the lower layer, slice 1 the upper, then each enabled mask's paint. Each tile is converted with ard_fix15_to_rgba16_premultiplied, which maps fix15's 32,768 steps onto 16 bits' 65,535, so nothing is lost on the way up. Only the covered tiles are uploaded: at each of them, a slice with no tile of its own gets zeros. Tiles of the bounding rectangle that neither layer covers are not written at all. The kernel composites them from whatever memory is there, and the readback skips them, so nothing stale reaches a layer.
An rgba32Float target.
Then it runs the same compositor, with a two-entry stack (the lower layer at full opacity, the upper with its blend, opacity, clipping and mask), reads the float result back row by row, multiplies by 32,768, rounds, clamps, and returns fix15 tiles, dropping any that came out fully transparent. The kernel does not care that its input is 16-bit instead of 8-bit: it reads a texture2d_array<float> either way. One blend formula, two input precisions. The new path also sizes its textures to the region, where the old one allocated a canvas-sized target for every merge.
The test that guards it (second code block) is the faint blue from above, merged with an empty layer, and it checks that the straight color comes back within 0.02 of (0.3, 0.6, 0.9).
Why Flatten May Stay at Eight Bits, Mostly
Exports, the gallery previews and Pippa's canvas look all call flatten, and flatten composites from the very same 8-bit slices. Is that the same bug? Not quite, and the difference is the lesson. Merge writes its result into the very layer the next merge reads, so its rounding becomes the truth and compounds. Most flatten results leave the app instead (a PNG, a JPEG, a PDF page, a preview, Pippa's look), and for those the rounding is a one-time loss. It is not always an invisible one. A transparent PNG stores straight color, so the faint blue that reaches it as (0.5, 0.5, 1.0) is written down exactly that wrong, at a precision the format could have represented correctly.
And flatten's output does come back in. ColorDrop, when its reference is all layers, and the magic wand both read a transparent flatten to decide where to reach, so faint paint's drifted color can move the edge of a fill or a selection. Copy All puts a flatten on the clipboard as a PNG, and Paste turns that into a new layer, mirror rounding and all. What none of them does is re-read its own output in a loop, the way Merge Layers re-reads the layer it keeps merging into. That is why merge was the one that had to leave the mirror first, not proof that the other readers are lossless.
The third code block makes the difference concrete. Merge the faint blue once through 8 bits and its color becomes (0.502, 0.502, 1.0); through 16 bits it stays (0.302, 0.599, 0.901). Then it does what a painter does: forty glazes at alpha 0.01, merged down one at a time. Because each merge re-reads the growing lower layer, the 8-bit path compounds its error: the color drifts to (0.417, 0.656, 0.656) and the alpha overshoots (12,196 against 10,847 in float). The 16-bit path ends at (0.299, 0.600, 0.899). Merge Layers in the app is literally this loop: it merges the selected layers into the lowest one, one after another, re-reading that layer every time.
A lossy copy is safe only while it flows one way. A cache, a preview, an export may round as much as their audience can't see. The moment any of them is written back into the source, its rounding becomes the truth and every later step inherits it. When you find a write-back, check the precision of everything it reads.
Code
Merge at 16 bits into 32-bit float over the covered region (AriandelEngine/CanvasRenderer.swift, trimmed: the mask bookkeeping and the stack are elided)·swift
/// The lower layer's new tiles after `upperID` is merged into it with the
/// upper layer's blend mode, opacity, clipping and mask; the lower
/// layer's own mask is applied too, so the result needs no mask (an
/// enabled mask is baked, a disabled one let everything through). The
/// lower layer keeps its other properties; only its pixels change.
///
/// The layers are uploaded at 16 bits per channel and composited in
/// 32-bit float over just the region they cover, so a merge never drops
/// the lower layer to the display mirror's 8 bits (soft, faint paint
/// would lose its color).
public func mergedTiles(of artwork: Artwork, upper upperID: UUID) throws -> [TileKey: TileBuffer] {
guard let upperIndex = artwork.index(of: upperID), upperIndex > 0 else { return [:] }
let upper = artwork.layers[upperIndex]
let lower = artwork.layers[upperIndex - 1]
let size = artwork.size
let keys = Set(lower.tiles.keys).union(upper.tiles.keys).filter { PixelRect(tile: $0).clipped(to: size) != nil }
guard !keys.isEmpty else { return [:] }
let region = keys.map { PixelRect(tile: $0) }.reduce(PixelRect(tile: keys.first!)) { $0.union($1) }
// Slices: 0 lower, 1 upper, then each enabled mask's paint.
// ... the tile stores of the lower layer, the upper layer and any enabled masks go into `sources` ...
let arrayDescriptor = MTLTextureDescriptor()
arrayDescriptor.textureType = .type2DArray
arrayDescriptor.pixelFormat = .rgba16Unorm
arrayDescriptor.width = region.width
arrayDescriptor.height = region.height
arrayDescriptor.arrayLength = sources.count
arrayDescriptor.usage = [.shaderRead]
arrayDescriptor.storageMode = .shared
let targetDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba32Float, width: region.width,
height: region.height, mipmapped: false)
targetDescriptor.usage = [.shaderRead, .shaderWrite]
targetDescriptor.storageMode = .shared
guard let array = device.makeTexture(descriptor: arrayDescriptor),
let target = device.makeTexture(descriptor: targetDescriptor),
let commandBuffer = queue.makeCommandBuffer() else {
throw RendererError.outOfMemory("merge \(region.width)×\(region.height)")
}
var tile16 = [UInt16](repeating: 0, count: Tile.valueCount)
let zero = [UInt16](repeating: 0, count: Tile.valueCount)
for (slice, store) in sources.enumerated() {
for key in keys {
let origin = key.pixelOrigin
// Every tile of the region is written, empty ones as zeros, so no slice reads stale memory.
if let tile = store.tile(at: key) {
tile16.withUnsafeMutableBufferPointer { buffer in
ard_fix15_to_rgba16_premultiplied(tile.pixels, buffer.baseAddress!, Tile.pixelCount)
}
tile16.withUnsafeBytes { bytes in
array.replace(region: MTLRegionMake2D(origin.x - region.x, origin.y - region.y, Tile.size, Tile.size),
mipmapLevel: 0, slice: slice, withBytes: bytes.baseAddress!,
bytesPerRow: Tile.size * 8, bytesPerImage: Tile.size * Tile.size * 8)
}
} else {
zero.withUnsafeBytes { bytes in
array.replace(region: MTLRegionMake2D(origin.x - region.x, origin.y - region.y, Tile.size, Tile.size),
mipmapLevel: 0, slice: slice, withBytes: bytes.baseAddress!,
bytesPerRow: Tile.size * 8, bytesPerImage: Tile.size * Tile.size * 8)
}
}
}
}
// ... the stack: the lower layer at opacity 1, then the upper layer with its blend, opacity, clip and mask ...
compose(stack: stack, background: nil, region: PixelRect(x: 0, y: 0, width: region.width, height: region.height),
layers: array, into: target, commandBuffer: commandBuffer)
commandBuffer.commit()
commandBuffer.waitUntilCompleted()
var tiles: [TileKey: TileBuffer] = [:]
var row = [Float](repeating: 0, count: Tile.size * 4)
for key in keys {
guard let rect = PixelRect(tile: key).clipped(to: size) else { continue }
let tile = TileBuffer()
let origin = key.pixelOrigin
for y in rect.y..<(rect.y + rect.height) {
row.withUnsafeMutableBytes { bytes in
target.getBytes(bytes.baseAddress!, bytesPerRow: Tile.size * 16,
from: MTLRegionMake2D(rect.x - region.x, y - region.y, rect.width, 1), mipmapLevel: 0)
}
let base = ((y - origin.y) * Tile.size + (rect.x - origin.x)) * 4
for i in 0..<(rect.width * 4) {
let value = row[i] * Float(Tile.fix15One)
tile.pixels[base + i] = UInt16(min(Float(Tile.fix15One), max(0, value.rounded())))
}
}
if !tile.isTransparent {
tiles[key] = tile
}
}
return tiles
}
The test that keeps faint paint's color (Tests/AriandelEngineTests/TransformAndAdjustmentTests.swift, verbatim)·swift
@Test func mergingKeepsTheColorOfFaintPaint() throws {
// At alpha 2/255 the display mirror's premultiplied 8 bits hold almost
// no color; a merge must not go through it.
let renderer = try CanvasRenderer()
let artwork = Artwork(size: CanvasSize(width: 64, height: 64), background: nil)
artwork.fillLayer(artwork.activeLayerID, with: RGBAColor(red: 0.3, green: 0.6, blue: 0.9, alpha: 0.008))
let upper = artwork.addLayer()
artwork.fillLayer(upper.id, with: RGBAColor(red: 1, green: 1, blue: 1, alpha: 0.0))
let merged = try renderer.mergedTiles(of: artwork, upper: upper.id)
artwork.mergeDown(upper.id, merged: merged)
let p = artwork.activeLayer.tiles.tile(at: TileKey(x: 0, y: 0))!.color(atX: 5, y: 5)
#expect(abs(p.r / p.a - 0.3) < 0.02 && abs(p.g / p.a - 0.6) < 0.02 && abs(p.b / p.a - 0.9) < 0.02,
"straight color survives: \(p.r / p.a), \(p.g / p.a), \(p.b / p.a)")
}
A minimal sketch: one faint merge and forty glazes, read at 8 versus 16 bits (compiled and run with swiftc in Swift 6 mode; the printed output is at the bottom)·swift
// A minimal sketch of why merge down must not read the 8-bit display mirror.
// Tiles hold premultiplied fix15 (1.0 = 32768). A merge reads both layers at
// some precision, composites in float, and writes fix15 back into the lower layer.
// Build and run: swiftc -swift-version 6 mirror.swift -o mirror && ./mirror
typealias Pixel = SIMD4<Double> // premultiplied r, g, b, a in 0...1
let one = 32768.0 // fix15's 1.0
func toFix15(_ p: Pixel) -> SIMD4<UInt32> { // what a layer stores
func c(_ v: Double) -> UInt32 { UInt32(min(one, max(0, (v * one).rounded()))) }
return SIMD4(c(p.x), c(p.y), c(p.z), c(p.w))
}
// Upload at `bits` per channel, rounding as ard_fix15_to_rgba8/16_premultiplied do.
func read(_ v: SIMD4<UInt32>, bits: UInt32) -> Pixel {
let maximum = (UInt32(1) << bits) - 1
func q(_ c: UInt32) -> Double { Double((c * maximum + 16384) >> 15) / Double(maximum) }
return Pixel(q(v.x), q(v.y), q(v.z), q(v.w))
}
func over(_ dst: Pixel, _ src: Pixel) -> Pixel { src + dst * (1 - src.w) } // Normal, premultiplied
func straight(_ v: SIMD4<UInt32>) -> String {
guard v.w > 0 else { return "transparent" }
let a = Double(v.w)
func f(_ c: UInt32) -> String { String(Int((Double(c) / a * 1000).rounded())) }
return "rgb/1000 = (\(f(v.x)), \(f(v.y)), \(f(v.z))) alpha/32768 = \(v.w)"
}
let blue = SIMD3<Double>(0.3, 0.6, 0.9)
func glaze(_ alpha: Double) -> Pixel { Pixel(blue.x * alpha, blue.y * alpha, blue.z * alpha, alpha) }
// 1. One faint layer (alpha 0.008, the engine test's value) merged with an empty one.
let faint = toFix15(glaze(0.008))
print("truth ", straight(faint))
for bits: UInt32 in [8, 16] {
let merged = toFix15(over(read(faint, bits: bits), read(SIMD4(0, 0, 0, 0), bits: bits)))
print("merge at \(bits) bits".padding(16), straight(merged))
}
// 2. Forty glazes of alpha 0.01, merged down one at a time: every merge re-reads
// the growing lower layer at the mirror's precision, so the error compounds.
var exact = Pixel(0, 0, 0, 0)
var lower8 = SIMD4<UInt32>(0, 0, 0, 0), lower16 = SIMD4<UInt32>(0, 0, 0, 0)
for _ in 0..<40 {
let upper = toFix15(glaze(0.01))
exact = over(exact, glaze(0.01))
lower8 = toFix15(over(read(lower8, bits: 8), read(upper, bits: 8)))
lower16 = toFix15(over(read(lower16, bits: 16), read(upper, bits: 16)))
}
print("40 glazes, float ", straight(toFix15(exact)))
print("40 glazes, 8-bit ", straight(lower8))
print("40 glazes, 16-bit", straight(lower16))
extension String {
func padding(_ width: Int) -> String { self + String(repeating: " ", count: max(0, width - count)) }
}
// Prints:
// truth rgb/1000 = (302, 599, 901) alpha/32768 = 262
// merge at 8 bits rgb/1000 = (502, 502, 1000) alpha/32768 = 257
// merge at 16 bits rgb/1000 = (302, 599, 901) alpha/32768 = 262
// 40 glazes, float rgb/1000 = (300, 600, 900) alpha/32768 = 10847
// 40 glazes, 8-bit rgb/1000 = (417, 656, 656) alpha/32768 = 12196
// 40 glazes, 16-bit rgb/1000 = (299, 600, 899) alpha/32768 = 10861
Run the sketch, then sweep the single faint merge: step alpha from 0.004 to 0.5 in steps of 0.001 and print, for the 8-bit path, the largest error in straight color at each alpha. Find the largest alpha at which the error is still 1% or more. Then look at a pipeline of your own (image processing, audio, a data export) and find one place where a reduced-precision copy is written back into the source of truth.
Hint
An 8-bit premultiplied channel has about 255 × alpha levels, so the error shrinks roughly like 1 / (255 × alpha): about 23% at alpha 0.01, 5% at 0.05, 2% at 0.1. It is not monotonic, because rounding lands differently at each alpha; for this blue it is still at or above 1% at some alphas up to about 0.28. In your own pipeline, look for round trips: a thumbnail used as an input, a value rounded to a cent and summed again, a compressed file re-encoded.
Progress
Progress is local-only — sign in to sync across devices.