Honest Uploads

Silence is the bug

Dad's complaint came from slow mobile data abroad: the family apps froze while photos went up and told him nothing. In Spark a capture opened its moment and then said Kept on the phone. or Sending... for minutes, with no bytes, no rate and no seconds. On a slow link a screen that says nothing looks exactly like a broken one, so Dad could not tell whether to wait, retry or give up.

The fix started with a measurement. A scratch engine sat behind the kit's throttling proxy, the dedicated test simulator took photos from a seeded camera, and every status line was logged. Two faults turned up behind the silence. The first was the queue: a photo kept while a drain pass ran waited for the next 60 s sync pass, which is why the drain is now coalesced (the previous lesson). The second was worse, and it is the subject here.

A timer that counted the wrong thing

At 20 KB/s, the 1.65 MB copy for Pippa never landed. The request handed its whole body to the send buffer in about a second, and URLSession counted no bytes while the buffer drained at the link's pace. The idle limit was 60 s, so every pass read a healthy upload as a dead link, cut it, and sent it again from the start: 7 attempts and 9.93 MB on the wire, still unsent after 900 s. The 180 s resource ceiling would have cut a 4 MB original next, because 4 MB at 20 KB/s takes 200 s.

So uploads got a transport and a policy of their own (the first code block). The idle limit is set per request from the body: the whole body's time at 15 KB/s, never under 120 s. A 1.65 MB copy gets the 120 s floor, and a 4.31 MB original gets about 287 s. The resource ceiling is 1800 s, which holds a 12 MB original at 7 KB/s. The code's comment says the ceiling never decides a moving upload, and for camera files at the rates the bench measured, that holds. It is still a ceiling: a file that needs more than 30 minutes on the link, say 13 MB at 7 KB/s, is cut, and the next pass sends it again from the start.

Requests	Idle limit	Ceiling
JSON calls	15 s	30 s
Media coming down	60 s	180 s
A capture's files going up	The body at 15 KB/s, at least 120 s	1800 s
The live lane's one short call	60 s	75 s
A moment's event stream	120 s	900 s

One policy for everything would be wrong for most of these. A JSON call that has said nothing for 15 s is probably lost. A photo that has said nothing for 15 s may simply be waiting in a buffer.

A line that says what is crossing

Each file now goes through the kit's FileUploadRun, and the moment's screen and its row in the queue show the kit's UploadStatusLine. The line names the file, counts megabytes of megabytes, shows the rate and the seconds, and ends with the request that follows the bytes: Asking Pippa after the copy, Keeping it at full size after the original. When the bytes go quiet, the line does not start a countdown. It reports what a check found: this phone's link first, then the engine's health route. The line appears before the moment's record is even sent, so it moves from the first second. A capture behind another one says Kept · next in line.

Failures keep the same honesty. A refusal of the file itself, such as a size the engine will not take, stops the line without a Retry, because retrying cannot help. A link failure, an engine answering 503, or a 4xx the kit counts as transient (401, 403, 408, 425 and 429, from the previous lesson) keeps the photo on the phone and offers Retry. The numbers from the bench, at 61.4 KB/s with three photos: before, 4 fixed labels and never a byte count. After, the first live line at 3.5 s, 299 distinct line states, and nothing sent twice. At 20 KB/s the copy that never landed went up once, followed by the original, both done in 305.9 s.

Principle: A status line is a claim about the world. Every word in it should name something true: a file, a size, a rate, a finding. A progress bar that guesses teaches the user to stop reading it.

One photo, two sizes

Honest numbers were not the end of it. The first titles read 1 of 2 · the copy Pippa reads and then 2 of 2 · the full-resolution original. Dad took a single camera capture and read those lines as two photos going up, an earlier one plus the new one. The engine's record, its log and the turn each held exactly one photo. The two were two passes of the same photo, and the count in the title said otherwise.

The titles no longer count anything (the second code block): This photo · a smaller size first, for Pippa to read, then The same photo · now at full size, to keep. A test fails if either title ever matches a pattern like a count again (the third code block), and a companion test takes two captures in a row and checks that each one creates one moment and sends exactly one copy and one original of its own bytes.

The web client got the same upload line from the kit. The problem was family-wide, so the fix was too: Sibling Kit Quest teaches why a second app needing the same piece is the moment it moves into the kit.

Code

cwkSpark, clients/apple/SparkMobile/Sources/EngineClient.swift: one session policy per kind of request, and the upload idle limit (verbatim, lines 18-38)·swift

    static let jsonPolicy = HTTPSessionPolicy(requestTimeout: 15, resourceTimeout: 30, byteBudget: 8 << 20)
    static let mediaPolicy = HTTPSessionPolicy(requestTimeout: 60, resourceTimeout: 180, byteBudget: 32 << 20)
    /// One file of a capture (cwkSiblingKit todo #545). The resource ceiling
    /// never decides a moving upload: a 4 MB camera original at 20 KB/s is
    /// 200 s, past the 180 s that used to cut it and send it again from the
    /// start; 1800 s holds a 12 MB original at 7 KB/s. The idle limit is
    /// `uploadIdle`, per request.
    static let uploadPolicy = HTTPSessionPolicy(requestTimeout: 120, resourceTimeout: 1800, byteBudget: 1 << 20)

    /// Seconds without a byte counted before a file's request is a dead link.
    /// The body can go into the send buffer at once, and URLSession counts
    /// nothing while the buffer drains: on the throttled bench at 20 KB/s the
    /// 1.65 MB copy was handed over in a second, then cut by the 60 s idle limit
    /// every pass and sent again, and never landed. So the limit is the whole
    /// body's time at 15 KB/s, never under the policy's 120 s.
    static func uploadIdle(bytes: Int) -> TimeInterval {
        max(uploadPolicy.requestTimeout, Double(bytes) / 15_000)
    }
    /// The live lane waits for Pippa's one short call (measured 3-6 s).
    static let livePolicy = HTTPSessionPolicy(requestTimeout: 60, resourceTimeout: 75, byteBudget: 1 << 20)
    static let streamPolicy = HTTPSessionPolicy(requestTimeout: 120, resourceTimeout: 900, byteBudget: 8 << 20)

cwkSpark, clients/apple/SparkMobile/Sources/Outbox.swift: each file's title and its finishing word (verbatim, lines 154-181)·swift

struct MediaUpload: Equatable, Sendable {
    enum Step: String, Sendable {
        case copy, original, document

        /// Which file this is, over its line.
        var title: String {
            switch self {
            case .copy: "This photo · a smaller size first, for Pippa to read"
            case .original: "The same photo · now at full size, to keep"
            case .document: "The document"
            }
        }

        /// The request that follows the bytes (the line's finishing word).
        var finishing: String {
            switch self {
            case .copy: "Asking Pippa"
            case .original: "Keeping it at full size"
            case .document: "Reading the document"
            }
        }
    }

    var entryId: String
    var momentId: String
    var step: Step
    var status: UploadStatus
}

cwkSpark, clients/apple/SparkMobile/Tests/TwoCapturesTests.swift: no upload title may read as a photo count (verbatim, lines 93-100)·swift

    func testNoUploadLineReadsAsAPhotoCount() {
        for step in [MediaUpload.Step.copy, .original] {
            XCTAssertNil(step.title.range(of: #"\d+\s+of\s+\d+"#, options: .regularExpression),
                         "\(step.title) reads as a count of photos")
        }
        XCTAssertTrue(MediaUpload.Step.original.title.contains("same photo"),
                      "the full-size pass names the photo as the one already going up")
    }

Exercise

Take an app of yours that uploads files and run it through a throttling proxy at about 20 KB/s. Record attempts per file, total bytes sent, and every distinct status string the user sees. Then give uploads their own idle limit computed from the body size and a minimum, and replace any count-style title with words that name the file. Run the same measurement again and compare.

Hint

The idle limit is the time between bytes, and a request body can sit in the send buffer with nothing counted. Spark's formula is the body's time at a slow floor rate, never below a fixed minimum. For titles, ask what a tired person on a bad connection would think the words mean.

Silence is the bug

A timer that counted the wrong thing

A line that says what is crossing

One photo, two sizes

Code

External links

Exercise

Progress

Comments 0