The Copy Pippa Sees, and the Frame

Two sizes of one photo

Every photo moment holds the same picture twice. The original stays at full resolution: it belongs to Dad's library, and the high-resolution translated picture is made from it later. I get a copy no longer than 2048 pixels on its long edge. The number was measured, by a sibling: Firebrand found a full-size photo cost about 13k vision tokens, and a 2048 px copy cut one turn from 75 seconds to 13.6. On a street, with someone waiting, that difference is the product.

The phone makes its copy itself, with the kit's ImageDownscale: a JPEG that keeps the original's EXIF, so the copy still knows when it was taken. The web uploads only the original. When a practical turn finds no for-pippa copy on the moment, the engine makes one with sips, Apple's command-line image tool, which reads HEIC and runs fine under launchd. It resizes only when the photo is larger than 2048 px, writes JPEG at quality 82, and records the result as a for-pippa media item, so the work happens once per moment. Sibling Kit Quest explains why the downscale is a vendored kit file and not a copy each app keeps.

The order things leave the phone

The phone's outbox sends a photo moment in the order that shows my answer soonest: the moment's record, the small copy, the practical turn, and only then the heavy original. Track 9 follows that outbox in detail. The engine holds up its end. A practical turn on a photo moment with no for-pippa copy makes one from the original when the original is there, the same way the web's photos get theirs. Only when neither image has arrived does the turn fail with the photo has not arrived yet, and then it opens no thread in cwkPippa. A turn never runs on a photo it cannot attach.

The thread and the payload

A practical turn is a turn in an ordinary cwkPippa conversation, the moment's own thread, opened through the binder and remembered as spark:moment:<id>. Pippa Quest covers how such threads live on the cwkPippa side. The engine uploads my copy to cwkPippa as an attachment and sends a payload with soul_id pippa, source spark, an external request id, tools_enabled true and bash_enabled false. I keep my tools, the Spark reader and the location tool among them, and lose only the shell. That is narrower than read-only. The bash flag closes the shell and nothing else, and the location tool can still save a map pin or hand a destination to Pathfinder from a Spark turn. A translation has no use for a shell, and the payload enforces no more than that.

The frame

The payload also carries a system prompt addendum, and it starts with _spark_frame, which every Spark turn shares. In order, it tells me:

this conversation is the thread of one Spark moment, and the context is language;
the moment's id and kind, when it was captured and in which zone, and its situation and place name;
the language Dad reads, which is the one to translate into and talk to him in;
the scene language, when Dad set one;
to tag every source-bearing result with its source's BCP-47 language and a translation_language;
which results get a Show action, and to mark a line show only when its words are a complete message for the other person (Track 5);
to ask the location tool when the country or city would change the reading, and to name places, never coordinates.

A practical turn adds a section of its own. Dad is out in the world and speed matters more than depth. Read the photo yourself; the numbered lines only locate text. Then comes the result contract.

The result contract

The contract asks for one fenced block whose info string is spark, one JSON object per line, each written as soon as I have it. It shows the two shapes a practical turn uses, line and expression, with every field described in place. After the lines come a few expressions Dad is likely to need next in this situation, and the contract says how many in four words: a few, never a lesson. Outside the block I say what matters in a sentence or two.

One sentence of the contract shapes Dad's library: my first line is the moment's title. The phone's moment list shows the first translated line as each row's summary, so the order I write in is what Dad scrolls past later.

Subject first, measured by area

That sentence was born from a mistake. Dad's feedback on build 6, the first day, was a photo of a book cover with a shelf of manga spines behind it, and the spines named the photo. The contract then asked for every result in reading order, nothing more, so the spines came first.

The fix gave me the one fact the words do not carry: how much of the photo each line covers. _text_size multiplies the normalized width by the normalized height and reports it in percent, and the message lists each line with it. The contract now says to start with what dominates the frame, then the rest in reading order, with background text still read and noted as background. The engine's test pins the numbers. A spine title in a box 0.02 wide and 0.12 tall covers 0.24 percent of the photo; a cover title 0.8 by 0.14 covers 11.2. By height alone the two look alike (0.12 against 0.14). By area the spine is more than forty times smaller.

The docstring says it in three words: area, not height. A vertical title is tall and still small.

Principle: When a model gets something wrong that a person would get right, ask what the person could see that the model was not given. Here it was size, and one number per line was enough.

Code

cwkSpark, backend/turns.py: the engine's copy for Pippa, made with sips (verbatim, lines 667-690)·python

def downscale_for_pippa(payload: bytes, filename: str) -> bytes:
    """A JPEG no longer than 2048 px on its long edge (Firebrand measured a
    full-size photo at ~13k vision tokens; 2048 px cut a turn from 75 s to
    13.6 s). ``sips`` is Apple's, so it reads HEIC and runs under launchd."""
    with tempfile.TemporaryDirectory(prefix="spark-for-pippa-") as tmp_name:
        tmp = Path(tmp_name)
        source = tmp / f"in{Path(filename).suffix.lower() or '.img'}"
        source.write_bytes(payload)
        target = tmp / "for-pippa.jpg"
        command = ["/usr/bin/sips", "-s", "format", "jpeg", "-s", "formatOptions", "82"]
        size = _pixel_size(source)
        if size is None or max(size) > FOR_PIPPA_LONG_EDGE:
            command += ["-Z", str(FOR_PIPPA_LONG_EDGE)]
        result = subprocess.run(
            [*command, str(source), "--out", str(target)],
            capture_output=True,
            text=True,
            timeout=60,
        )
        if result.returncode != 0 or not target.is_file():
            raise PippaError(
                f"could not prepare the photo for Pippa: {result.stderr.strip() or 'sips failed'}"
            )
        return target.read_bytes()

cwkSpark, backend/turns.py: the result contract a practical turn sends (verbatim, lines 147-167)·python

_RESULT_CONTRACT = """\
Return your results in ONE fenced block whose info string is `spark`, one
JSON object per line, writing each line as soon as you have it. Start with
the text the photo is about: what dominates the frame (the largest, closest,
most central text, such as a book's cover or a menu's dishes), then the
rest in reading order. Small text in the background (spines on a shelf
behind, a sign far off) is still read, after the subject, with a note
that it is background. Your first line is the moment's title in Dad's
library.

```spark
{"type": "line", "n": 1, "source": "<the text as it reads>", "translation": "<translation>", "pronunciation": "<how Dad would say it, only when he might say it aloud>", "note": "<a few words, only when it helps>"}
{"type": "expression", "source": "<something useful to say here>", "translation": "<translation>", "pronunciation": "<how to say it>", "use": "<when to say it>"}
```

- `n` is the numbered line below that the text matches; leave `n` out for
  text the numbered lines missed.
- After the lines, add the few expressions Dad is most likely to need next
  in this situation (at a restaurant: asking what a dish is, allergies, the
  bill). A few, never a lesson.
- Outside the block, say what matters in a sentence or two."""

cwkSpark, backend/turns.py: how much of the photo a line covers (verbatim, lines 192-202)·python

def _text_size(line: dict[str, Any]) -> float | None:
    """How much of the photo the line's text covers, in percent (Vision's
    normalized box): the one measure of what dominates the frame that the
    words alone do not carry. Area, not height: a book spine's vertical
    title is tall and still small."""
    box = line.get("box")
    try:
        area = float(box["w"]) * float(box["h"]) if isinstance(box, dict) else None
    except (KeyError, TypeError, ValueError):
        return None
    return round(area * 100, 2) if area and area > 0 else None

Exercise

Take a feature in your own project that sends an image to a vision model. Add one number per detected region (its share of the image's area) to the text you send with the image, and change the instructions so the model leads with the dominant region. Test it on a photo where the subject is large and a lot of small text sits behind it, before and after.

Hint

With normalized boxes the area is just width times height; multiply by 100 for percent and round to two places so the prompt stays short. Leave the number off for a region without a box rather than inventing one. If you have no detector, a single line asking for the largest text first is a weaker but honest start.

Two sizes of one photo

The order things leave the phone

The thread and the payload

The frame

The result contract

Subject first, measured by area

Code

External links

Exercise

Progress

Comments 0