Green Is Not Exercised

"Ask what a skip would hide, and make the skip loud."

Seventy-Four Green Tests and a Crash on the First Tap

The training app moved its dictation code into the shared kit and shipped the result as a TestFlight build. The owner tapped the microphone and the app died, every time. The previous build, with the same logic as app-local code, had worked on the same phone minutes earlier. The suite had passed 74 tests on the Simulator and passed again on a cabled iPad. The crash was the one from the concurrency track: a closure created inside a @MainActor method, compiled under Swift 6 mode in the kit, inherited main-actor isolation and trapped when the audio engine called it on its realtime thread. No suite could have seen it. No test started the audio engine, not on the Simulator and not on the iPad, so the audio tap was never invoked. Green only ever covers what the hardware and the tests actually exercise.

A Skip Reads as a Pass

The same app's device tests had an escape hatch: they skipped when the speech permission was still undetermined. Three device runs in a row came back green while exercising nothing, on the only hardware able to reach the crash above. A suite that runs and skips is indistinguishable from one that passes: in xcodebuild output, in a script's summary line, in the exit status. Measured for this lesson, a package whose one real check skipped printed "Executed 2 tests, with 1 test skipped and 0 failures", marked the suite passed, and exited 0. The fix had two parts. Device tests now request their permissions instead of skipping past them (one prompt per device, silent afterwards), and that same iPad then ran 77 tests with zero failures and zero skips. And any script that can surface a skip says so in its own words.

Split the Assertion Instead of Deleting It

Some skips are honest, and the way to keep them honest is to split what they guard. The Simulator does not implement Data Protection: a write with .completeFileProtection succeeds, nothing is recorded, and attributesOfItem returns no protection class, with no error anywhere. Deleting the assertion leaves the most security-relevant behaviour untested everywhere. Asserting only the constant quietly redefines "we protect files" as "we have a variable named complete". So the policy constants are asserted on every platform, which is the half that fails when someone downgrades the policy, and the recorded class is asserted on a device only, behind a skip whose message says why.

A Round Trip That Skips the Transport

One more green lie. A watch payload with an absent optional field was never delivered, while the watch reported it synced. The wire test built the dictionary and parsed it straight back, and Swift is happy to store Optional.none in a [String: Any]. The real transfer accepts property-list values only. The test that catches it serializes with PropertyListSerialization, and it was run against the broken spelling first to watch it fail. A regression test nobody has watched fail is a guess.

Code

The policy asserted everywhere, the recorded class on a device only, and a skip that explains itself·swift

// Sources/SparkStore/Protection.swift
import Foundation

public enum SparkProtection {
    /// The policy: the class the app writes with. Asserted everywhere.
    public static let writeOptions: Data.WritingOptions = [.atomic, .completeFileProtection]
    public static let expectedClass: FileProtectionType = .complete

    /// Whether this platform records the class at all. The Simulator does not; neither does macOS.
    public static var platformRecordsProtectionClasses: Bool {
        #if os(iOS) && !targetEnvironment(simulator)
        return true
        #else
        return false
        #endif
    }

    public static func recordedClass(of url: URL) -> FileProtectionType? {
        (try? FileManager.default.attributesOfItem(atPath: url.path))?[.protectionKey] as? FileProtectionType
    }
}

// Tests/SparkStoreTests/ProtectionTests.swift
import Foundation
import XCTest
@testable import SparkStore

final class ProtectionTests: XCTestCase {
    /// Runs everywhere: fails if someone downgrades the policy.
    func testPolicyIsCompleteProtection() {
        XCTAssertTrue(SparkProtection.writeOptions.contains(.completeFileProtection))
        XCTAssertEqual(SparkProtection.expectedClass, .complete)
    }

    /// Device only, and it says so: a Simulator run skips LOUDLY instead of passing a check of nothing.
    func testWrittenFileRecordsCompleteProtection() throws {
        try XCTSkipUnless(SparkProtection.platformRecordsProtectionClasses,
                          "this platform does not record protection classes; run on a device")
        let url = FileManager.default.temporaryDirectory.appending(path: "protected-\(UUID()).bin")
        try Data("secret".utf8).write(to: url, options: SparkProtection.writeOptions)
        XCTAssertEqual(SparkProtection.recordedClass(of: url), .complete)
    }
}

A test wrapper that will not let a skip read as a pass·bash

#!/bin/zsh
# Run the suite, keep the log, and refuse to let a skip read as a pass.
set -uo pipefail
swift test 2>&1 | tee test.log
suite_status=${pipestatus[1]}
summary=$(grep -E 'Executed [0-9]+ tests?' test.log | tail -1)
skipped=$(print -r -- "$summary" | sed -nE 's/.*with ([0-9]+) tests? skipped.*/\1/p')
(( suite_status == 0 )) || { print -u2 "HARD_FAIL suite exited $suite_status"; exit 1; }
if [[ -n "$skipped" ]] && (( skipped > 0 )); then
  print "TESTS_GREEN_WITH_SKIPS $skipped skipped: these checks did NOT run here"
  grep 'Test skipped' test.log | sed 's/^.*Test skipped - /  skipped: /'
else
  print "TESTS_GREEN no skips"
fi

Exercise

Build the SparkStore package above and run its tests on your Mac, then wrap the run with loud.sh and confirm it names the skipped check. Next, add a property-list transport test for a payload type with an optional field: first write the naive round trip that passes with Optional<String>.none as Any in the dictionary, then the serializing version, and show the serializing test failing against the broken spelling before you fix the payload.

Hint

PropertyListSerialization.propertyList(_:isValidFor: .binary) answers false for a dictionary holding Optional<String>.none. The fix is to omit the key when the value is nil, which the serializing test then accepts.