The Test Pyramid

Five layers, each with its own question

Voyager is checked by five instruments, and each answers a question the others cannot. The counts below are taken from the snapshot this quest reads, origin/main at 59e0269.

Layer	Size	Runs against	Proves
The parent suite, `VoyagerCoreTests`	242 test functions, 48 suites	no WebKit at all	the rules: tabs, sleep, sessions, identities, trust, the catalog, passwords, history
The engine child's suite, `VoyagerWebKitTests`	63 test functions, 8 suites	real `WKWebView`s	that WebKit behaves the way the contract assumes
Kit drift	1 test, plus a step in the phone's gate	a Sibling Kit checkout	that vendored kit files still match the kit
The smoke run	58 `check` lines over two launches	the verification app, on screen	that the AppKit shell, its windows and its keys work together
Hosted XCTests on the phone	10 methods in 3 classes, 2 files	the iOS app on a dedicated simulator	the widget's links, the shelf, the shipped Info.plist

Track 3 already showed the bottom of this pyramid: why the parent can be tested with a recorder in place of an engine, and which claims the fake can never prove. This lesson is about the whole shape, and about what each layer leaves to the next.

What each layer leaves to the next

The parent suite is fast and deterministic because it never meets WebKit. Its blind spot is the engine itself.
The WebKit suite covers that spot. It loads pages into real web views, brings a scroll position back from WebKit's own bytes, and checks that WebKit really calls the child's delegate methods. One of its 63 functions is an opt-in diagnostic that a normal run skips. Its blind spot is the app around the web views.
The smoke run is the only automated check of that app. The package has no test target for VoyagerApp, the AppKit shell, so the verification build walking its own screens is what catches a lost keystroke or a settings window drawn empty.
The hosted suite holds the phone's promises: a family link carries a catalog id and nothing else, the shelf keeps names and ids only, the phone never shows the dev site or a Mac app, and the Info.plist declares the cwkvoyager scheme with one blanket transport key. Under XCTest the app shows an inert window and builds no browser, so nothing dials the family.
Kit drift answers a question no product test asks: is the code that came from the kit still the kit's?

Drift as a test

Voyager takes shared family code from Sibling Kit. Some arrives as a package, and some is vendored as files, like the TestFlight trio in the next lesson. A vendored file edited by hand is a fork nobody decided to make, so drift is checked inside the ordinary test run (second block). On the fleet it compares the vendored files with the kit and records an issue when they differ. Off the fleet there is nothing to compare, and it says so out loud instead of passing quietly. Sibling Kit Quest teaches the vendoring model and its drift check in full.

A test that hangs says nothing

Every WebKit suite is .serialized and carries a time limit of one or two minutes. The limit comes from a gotcha in the house sign. A WebKit test that awaits something a hidden page never does hangs the whole run in silence: getUserMedia is not answered until the page sits in a visible window, and swift test prints nothing until it ends. The limit turns a silent hang into a failure with a name. To find a culprit by hand, run one test under an alarm:

perl -e 'alarm 70; exec @ARGV' xcrun swift test --skip-build --filter <name>

Counting honestly

Test counts drift through documents. The implementation plan records 59 tests at the first brick, 45 on the fake engine and 14 on WebKit. A daily-use review written later that day records 156 core tests and 52 WebKit tests. Commit ac85754 says 305. Each was true when it was written, and only the repository can say what is true now. So count with a command (first block) and name the snapshot you counted.

These numbers count functions, not cases. A parameterized test, like the one that checks 14 delegate selectors, runs more cases than it adds to the count.

Before a phone upload, the gate runs the layers in a fixed order. scripts/test-mobile.sh runs the drift check, then the package suite, judged by its exit status, then the hosted suite, judged by the log's own verdict line (third block). It never trusts a filtered pipe, whose status is the filter's.

Principle: A layer earns its place by proving something no other layer can. When you add a test, name the blind spot it covers.

Code

The commands behind this lesson's counts (hand-written, run against cwkVoyager origin/main 59e0269)·shell

cd cwkVoyager
git grep -c '@Test' origin/main -- Tests/VoyagerCoreTests | awk -F: '{n += $NF} END {print n}'    # 242
git grep -c '@Test' origin/main -- Tests/VoyagerWebKitTests | awk -F: '{n += $NF} END {print n}'  # 63
git grep -c '@Suite' origin/main -- Tests/VoyagerCoreTests | awk -F: '{n += $NF} END {print n}'   # 48
git grep -c 'func test' origin/main -- clients/apple/VoyagerMobile/Tests                            # 7 and 3
git show origin/main:scripts/smoke.sh | grep -c '^ *check('                                         # 58

cwkVoyager, Tests/VoyagerCoreTests/KitDriftTests.swift: kit drift inside the ordinary test run (verbatim, lines 1-19)·swift

import CwkKitTesting
import Testing

/// Architecture invariant 14: kit-owned code is consumed, not copied. The
/// files cwkSiblingKit vendors into this repo match the kit.
@Suite("Sibling kit drift")
struct KitDriftTests {
    @Test func vendoredKitFilesMatchTheKit() {
        switch KitDriftCheck.run(repo: "cwkVoyager") {
        case .passed:
            break
        case .skipped(let reason):
            // Off the fleet there is nothing to compare against; say so loudly.
            print("note: kit drift check skipped — \(reason)")
        case .failed(let output):
            Issue.record("vendored kit files drifted:\n\(output)")
        }
    }
}

cwkVoyager, scripts/test-mobile.sh: the hosted suite, judged by the log's verdict (verbatim, lines 52-70)·shell

# 3. The hosted app suite. project.yml is the source; regenerate before trusting a green result.
(cd "$mobile" && xcodegen >/dev/null)

results="$mobile/build/mobile-tests.xcresult"
/bin/rm -rf "$results"

xcodebuild test \
  -project "$mobile/VoyagerMobile.xcodeproj" \
  -scheme VoyagerMobile \
  -destination "platform=iOS Simulator,id=$VOYAGER_MOBILE_SIMULATOR_ID" \
  -resultBundlePath "$results" \
  > "$mobile/build/mobile-tests.log" 2>&1 || true

# The log's verdict decides, not a filtered pipe (learning #370).
if ! /usr/bin/grep -q "\*\* TEST SUCCEEDED \*\*" "$mobile/build/mobile-tests.log"; then
  /usr/bin/grep -E "error:|failed" "$mobile/build/mobile-tests.log" | /usr/bin/head -30 >&2 || true
  print -u2 "HARD_FAIL the iOS suite did not pass — see $mobile/build/mobile-tests.log"
  exit 1
fi

Exercise

Draw the test pyramid of an app you maintain. For each layer, write its size (counted with a command, not from memory), what it runs against, and one claim that only this layer can prove. Then find a claim your users depend on that no layer proves, and decide which layer should own it.

Hint

Look for code with no test target at all; Voyager's AppKit shell has none. For that kind of code, an end-to-end run of a separate verification build is often the only honest layer. If a claim needs a visible window or a real device, write that into the test's name or comment, so nobody expects the fast layer to catch it.

Five layers, each with its own question

What each layer leaves to the next

Drift as a test

A test that hangs says nothing

Counting honestly

Code

External links

Exercise

Progress

Comments 0