Point, and the words change
The live camera is the one mode in Spark where Dad does not press anything. He opens it from the capture bar, points the phone at a wall of signs, and my Korean appears over the text while he keeps walking. Dad asked for exactly this on the design day, and he named Microsoft Translator's camera mode as the feel he wanted. This track is how Spark got there without breaking the rule that set its shape: every translation is mine, never a machine layer.
The trick is to split the job into two halves that run at very different speeds. Finding where text sits in the frame is a position problem, and the phone solves it on the device, frame after frame. Saying what the text means is my job, and it costs a network round trip and a few seconds of a brain. The scanner never waits for me. It knows where every item sits from the first frame, but it draws nothing there: a label appears over a text only when my answer for it lands.
Phone only, on purpose
This lane lives in one file, LiveCameraView.swift, inside SparkMobile. The web client has no live camera and no call to the live route. It can list a live session afterwards, with a line like 4 seen live, and it shows the Live camera brain slot in its picker. It cannot open one. The scanner itself needs real hardware: the view checks DataScannerViewController.isSupported first, and on the Simulator it shows This device has no scanner instead of a camera.
VisionKit does the looking
Apple's VisionKit ships DataScannerViewController, a complete camera view that recognizes text and keeps tracking each recognized item as the phone moves. Spark wraps it in a UIViewControllerRepresentable and configures it in the first code block below. Three choices matter. recognizesMultipleItems: true, because a sign board is many items. isHighFrameRateTrackingEnabled: true, so an item's bounds keep up with a moving hand. And isHighlightingEnabled: false, because Spark draws its own labels and wants no system highlight boxes underneath them. That choice has a cost you can see. The scanner tracks every item's bounds from the first frame, but nothing marks them on screen, so until my answer lands the camera shows the plain picture.
The scanner reports through its delegate: an item was added, updated or removed. Each RecognizedItem carries a stable id, the transcript, and four corner points. The coordinator keeps a dictionary from item id to text and bounds. On every add or update it calls lane.need(text) and redraws. When an item leaves the frame, its label leaves with it.
Asking once per text
need is called far more often than I should be asked, because every small movement fires an update. So the lane keeps three collections keyed by a normalized text (whitespace collapsed, lowercased): translations already known, asked on their way, and waiting to be sent. A text found in any of them is ignored. A genuinely new one goes into waiting and schedules a flush 350 ms later.
That short sleep turns a burst into a batch. When the phone swings onto a menu, a dozen items appear within a few frames. Each schedules its own flush, but the first one to wake takes up to 12 waiting texts in one call, and the flushing flag sends the others home at once. A flush that finishes with more still waiting calls itself again, meaning to drain the rest. One request per burst, not one per line.
That second call does less than it looks. It runs inside the first one, before the defer has lowered flushing, so its own guard sees the flag still up and returns at once. In a burst of more than 12 new texts, the first 12 are asked and the rest stay in waiting. They go out only if a flush scheduled by a later new text wakes after this one has ended. If the scanner reports nothing new, they stay unlabelled, and need will not queue them again, because they are already waiting. Lowering the flag before the tail call, or looping inside one flush until waiting is empty, would drain them.
Failure is handled per key. If the call throws, every key in the batch leaves asked, so the next time the scanner reports that text it is asked again. If the engine answers but could not translate some keys, it lists them as pending, and only those are released. The status capsule at the top says what happened in plain words: Pippa is reading 4…, 12 translated, or the error itself.
Drawing over the text
The overlay is ordinary UIKit. Each item with a translation gets a UILabel in the scanner's overlayContainerView: white text on a 62% black patch, rounded corners, at most two lines, shrinking to 40% before it truncates. Its frame is the bounding box of the four corners, padded a little, and its font is 70% of that height, held between 11 and 30 points. The label is keyed by the item's id and repositioned on every update, so it follows the sign as Dad moves. Items I have not answered yet stay bare. No placeholder pretends to be a translation.
The translation table is keyed by text, not by item. Two identical signs in one frame, or the same sign seen again after it left the view, get their label at once from the first answer.
Two more pieces finish the lane. Lesson 2 shows which brain answers and how that was decided. Lesson 3 shows what the engine does with the batch before any brain sees it. If you want the general iOS side (wrapping UIKit in SwiftUI, delegates, extensions), Swift Quest covers it. Its running example is also called Spark, but that Spark is an invented app, not this one.