Swift Testing, XCTest, and the Self-Test Executable

"Three ways to test is not a mess if each one has a job."

The House Rule: Tests by Purpose

The family's apps once used all three styles more or less by accident. The rule now assigns each one a job:

Swift Testing for SwiftPM package suites. @Test functions, #expect and #require, @Suite types, parameterized tests with arguments:, and traits such as .timeLimit(.minutes(1)). It runs with swift test, in parallel by default, and it is where new tests go.
XCTest for Xcode-hosted tests. Unit tests that run inside an iOS or watch app host, extension tests, and UI tests with XCUITest — the XcodeGen targets. Xcode-hosted suites and the upload pipeline's test gate are built around it.
A self-test executable only for Macs without Xcode. A Mac with only the Command Line Tools has no XCTest, and older tools had no Testing module either. Measured with the 6.1 Command Line Tools, a Swift Testing suite does run there. The family's dictation and text-transformation utilities keep a -selftest executable — plain checks, exit status 1 on failure — so their pure logic can be verified there, and the repository says so.

What Swift 6.3 Added

Swift Testing in 6.3 can record a warning issue (Issue.record("…", severity: .warning)) that is reported without failing the test, cancel a test that cannot meaningfully run (try Test.cancel()), and attach images to results. Exit tests and attachments arrived in 6.2.

Traps the Family Paid For

A hosted test boots the whole app. An iOS unit test target runs inside the app, so Bundle.main is right — but UserDefaults.standard is the app's, and the app launched beside the tests dials its real engine endpoint. Inject the defaults store, the notification centre and transport construction.
AppKit tests share focus. Tests that create windows and key views interfere when run in parallel; a family editor runs its suite with swift test --no-parallel.
A symptom that moves under an inert edit is not a code bug. Appending an unrelated two-assertion @Suite to a file made a real-clock test in the same file abort the whole test process with a task-allocator failure, eight runs out of eight. Moving the new suite to its own file fixed it. The expensive part was bisecting: twice the investigation rewrote working production code because each intermediate result looked causal. Rule: when a "fix" changes the symptom but an inert edit changes it too, stop bisecting and rule things out one at a time, with a fixed number of repetitions per configuration.
An async test that can hang needs a time limit. Without a .timeLimit trait, a regression that never resumes stalls the entire run instead of failing one test.

Code

Swift Testing for the package: parameterized, time-limited, #require·swift

import Testing
import Foundation
@testable import SparkCore

@Suite("Capture")
struct CaptureTests {
    @Test("blank detection", arguments: [("", true), ("   ", true), ("milk", false), (" 우유 ", false)])
    func blank(text: String, expected: Bool) {
        #expect(Capture(id: "x", text: text).isBlank == expected)
    }

    @Test(.timeLimit(.minutes(1)))
    func roundTripsThroughJSON() throws {
        let capture = Capture(id: "01J", text: "hello")
        let data = try JSONEncoder().encode(capture)
        let decoded = try #require(try? JSONDecoder().decode(Capture.self, from: data))
        #expect(decoded == capture)
    }
}

The self-test executable for a Mac with only the Command Line Tools·swift

// Sources/spark-selftest/main.swift — run with: swift run spark-selftest
import SparkCore
import Foundation

var failures = 0

@MainActor func expect(_ condition: Bool, _ message: String, line: Int = #line) {
    if !condition { failures += 1; print("FAIL line \(line): \(message)") }
}

expect(Capture(id: "a", text: "  ").isBlank, "whitespace-only text is blank")
expect(!Capture(id: "b", text: "milk").isBlank, "text with letters is not blank")

print(failures == 0 ? "SPARK_SELFTEST_OK" : "SPARK_SELFTEST_FAILED \(failures)")
exit(failures == 0 ? 0 : 1)

Exercise

Add both code blocks to SparkKit. Run swift test and swift run spark-selftest and record the output. Then break Capture.isBlank on purpose (make it always return false) and confirm that BOTH the Swift Testing suite and the self-test fail and say why. Restore it. Finally, add a .timeLimit trait to a test that awaits a continuation you never resume, and observe that the run fails in bounded time instead of hanging.

Hint

A self-test that never fails proves nothing — the break-it-on-purpose step is the point. For the hanging test, await withCheckedContinuation { _ in } never resumes; Swift may also warn that the continuation leaked, which is fine for this experiment.