The Question Behind the Tier
Every look has to choose how large a picture to send. Bigger reads more detail and costs more tokens; smaller is cheap and loses small text. Game screens are full of small text: menu labels, item counts, subtitles, a health number in a corner. So "which resolution" is really "how much small text can a model read at each size, and what does each size cost?" Oriel answered it the way it answers most questions, by measuring.
The Measurement
Thirty frames of the console's home screen were taken from the 4K game card, one a second. Each was fitted to 1280×720 and to 1920×1080 with exactly the resampling and JPEG settings a look uses, and read with Apple's Vision text recognizer. The reference was the same recognizer on the full 4K frame: 459 lines of text, from 27.9 px to about 45 px tall at 4K, so 9 to 15 px at 720p. Two numbers were measured for each tier: how many reference lines came out exactly, and how similar the characters were overall.
| Tier | Lines read exactly | Character similarity | Smallest quarter of the text: exact / similarity | Image tokens (dearest vendor) |
|---|---|---|---|---|
| 1280×720 | 68.8% | 0.944 | 76.5% / 0.865 | 1,229 |
| 1920×1080 | 94.6% | 0.990 | 76.5% / 0.950 | 1,844 |
1080p reads clearly more of the small labels on game art. It also costs 1,844 tokens for the picture alone, which is over the whole 1,600-token budget of a look before a single character of text. The tempting move is to raise the budget. The architecture document records the decision instead: the look stays 720p, and small text is read another way.
The Other Way: Crop the Original
The insight is that a look and a reading are different jobs. A look answers "what is happening?", and 720p is enough for that. Reading a label answers "what does this say?", and for that you do not need the whole screen sharper; you need one region at full detail. That is the crop action. It takes a region in normalized coordinates (x, y, w, h between 0 and 1) and cuts it from the original frame: the full-resolution 4K frame from the recording, or a full-resolution peek when nothing is recording. The piece is fitted inside 1024×1024 (a large region is scaled down to fit, so a full-frame 4K crop comes back as 1024×576), and a small region is enlarged two times with the same Lanczos resampling so the recognizer has something to work with.
A 1024×1024 crop costs 1,399 tokens at most, inside the same 1,600 budget, and for the small regions it exists for, it is sharper than either tier, because a region that already fits is never scaled down. So the pattern a brain learns is two steps: look at 720p to see where the interesting text is, then crop that region from the original. Two cheap calls replace one expensive one, and the second call is better at its job than the expensive one would have been.
Small Honesty in the Crop
Two details keep the crop honest. A region too small to read (under eight pixels a side) is refused with that reason rather than returning a smear. And a crop from a live peek that turns out to be old is refused with the frame's time, so the caller can ask the recording for that moment explicitly; a crop exists to read fine detail as it is, and an old frame is not that.
There is one more quiet step before any look: some devices send their picture inside black bars (one pocket camera always sends a portrait buffer with the 16:9 picture in a band across the middle). Looks cut bars that take at least a tenth of the frame, so the model is not charged for black. Never on the game feed, though, where black is part of the game.