Sense Knows No Game

Three Quarters of a Turn Is Thinking

A play turn is a look, a model's judgment and an input. Before building anything faster, the turn was measured over eight sessions of 822 to 1,175 inputs each. The frame a look returns is 24 ms old (median). The first screen change comes 122 ms after an input. The time from a look's answer to her next input is 4.7 s (median, 3.9 at the 25th percentile, 7.9 at the 90th). cwkPippa's own turn log of 166 play turns agrees: a look's round trip is 0.58 s and the model spends 4.15 s between a look's answer and its next tool call. Judgment is about three quarters of a 5.5-second cycle. Capture and transfer are small.

The conclusion follows from the numbers: nobody fights at that pace, and no faster model call fixes it. Reflexes need a layer with no model in it. That layer has to read the screen itself, every frame, and act on what it reads.

A Root Class, Not a Dark Souls Script

The obvious build was a screen analyzer for the one game she was playing. Dad ruled it out the day it gained ears, in words that are the design rule: what matters is not writing it once; think object-oriented whatever you build, make a canonical root class, inherit from it so anything can be done polymorphically, a root that detects images (video) and audio both, apply it to this game, and never overfit it to this game, or the next game or video means building it again.

So backend/sense.py knows no modality and no game:

A Detector takes timestamped samples and returns timestamped Events: the kind of reading, the name its profile gave it, the value, how sure, which reader said it, the media time it is true of, and how late it was said.
Its children pay their own polymorphic tax: VisionDetector decodes frames, AudioDetector joins waveform blocks. The root is never edited to fit one.
A Reader is one thing read from a sample. Reader classes register by modality and kind, so a new kind is a new class, never a branch in the root.

Kind	Reads
`level`	how full a bar is, and whether it is empty or hidden
`count`	a small number, matched to glyph pictures the profile learned
`mark` / `float_bar`	a small bright spot, or thin bars anywhere in a region
`splash`	how much of a region one colour covers (a banner)
`pan` / `landmark`	degrees the view turned since the last frame, and where a learned picture is
`detect`	figures in a frame, by a small learned model
`onset` / `sound`	a sudden rise in a band over its own floor, or a learned sound

A colour is never a constant in code either: it is a test the profile sets (how far one channel leads another, how near a colour it is, how bright, how dark). And consumers read events, never a detector: events go to a Bus, and a consumer listens or asks its State (the newest reading of each name, with its age). Combining readings is itself a consumer that may publish events of its own.

The Game Is Data

Everything about a game is its profile, kept as data under the game's own name with its history beside it: which readers, their numbers, the fusions, the combos and rules. It is learned from Oriel's recordings and tuned by Pippa from her own runs, every change a row with its why, never transcribed from a walkthrough. The proof that no game hides in the code is in the tests: test_sense.py and test_hand.py run two games that do not exist on the same classes. A constant that only fits the real game fails the fake ones.

The Hand Stands Under Every Stop

The fast layer that acts on the bus is the hand: conditions on fresh readings, named combos, rules with priority and cooldown, and modes that list the live rules. It is built so that speed buys no new authority:

It sends only through the play session's input door (Play.act with reflex), so the course, the denied buttons, the timer, the red eye, On Air's kill switch and Dad's Stop hold for it as for any input. A closed session takes no press.
It starts off, and only the head turns it on, with a why. off keeps the senses reading and presses nothing.
A value is acted on only while fresh, and a reading that could not be read passes no test: no count is no drink.
Every press is a record: the rule, the readings and their ages, the steps sent and left out, the time from frame to press, and whether the screen then showed what the combo expects. Reflexes are tallied apart from her own loop, which measures the head.

The first live trial (a coached Solo session of eight minutes, on Dad's word) read 3,034 frames: fetched in 18 ms, read in 43 ms, decided in under a millisecond. From the frame that showed health falling to the press handed to the console: 65 and 71 ms. And both healing presses were pressed and neither was shown: one landed during a stagger right after a hit, the other just before the blow that killed her. The shown field did its work, since a press that did nothing was not counted as a heal, and the rule now waits out a hit. A fast layer that records whether its action took effect can be corrected. One that only records that it fired cannot.

Code

The detector root: readers register by modality and kind, and feed turns one sample into timed events (trimmed excerpt, backend/sense.py)·python

def reader(cls: type[Reader]) -> type[Reader]:
    """Register a reader class under its modality and kind."""
    if not cls.modality or not cls.kind:
        raise SenseError(f"{cls.__name__} names no modality or kind")
    key = (cls.modality, cls.kind)
    if key in _READERS and _READERS[key] is not cls:
        raise SenseError(f"two readers claim {key}")
    _READERS[key] = cls
    return cls


class Detector(ABC):
    """The root: a timestamped stream in, timestamped events out."""

    modality: ClassVar[str] = ""

    @abstractmethod
    def prepare(self, sample: Sample) -> Any:
        """The sample as this modality's readers take it."""

    def feed(self, sample: Sample) -> list[Event]:
        started = time.perf_counter()
        prepared = self.prepare(sample)
        out: list[Event] = []
        for item in self.readers:
            if not item.due(sample.at):
                continue
            item._last_at = sample.at
            for reading in item.read(prepared, sample.at):
                spent = (time.perf_counter() - started) * 1000
                at = sample.at if reading.at is None else reading.at
                path = (sample.arrived - at) * 1000 if sample.arrived is not None else 0.0
                out.append(Event(kind=item.kind, name=item.name, value=reading.value,
                                 confidence=float(reading.confidence),
                                 source=f"{self.modality}:{self.stream}/{item.name}", at=at,
                                 latency_ms=max(0.0, path) + spent, data=dict(reading.data)))
        return out

A rule that acts only on fresh, read values and records whether its press showed (runnable sketch)·python

import time

FRESH_S = 0.3


def decide(state, now):
    """state maps a name to (value, read_at); None means it could not be read."""
    hp = state.get("hp")
    flasks = state.get("flasks")
    if hp is None or flasks is None or hp[0] is None or flasks[0] is None:
        return None, "unread value passes no test"
    if now - hp[1] > FRESH_S or now - flasks[1] > FRESH_S:
        return None, "stale value passes no test"
    if hp[0] < 0.3 and flasks[0] > 0:
        return "heal", f"hp {hp[0]:.2f} read {1000 * (now - hp[1]):.0f} ms ago"
    return None, "no rule holds"


def record(action, why, shown):
    return {"actor": "hand", "combo": action, "why": why, "shown": shown}


now = time.time()
print(decide({"hp": (0.2, now - 0.05), "flasks": (3, now - 0.05)}, now))
print(decide({"hp": (0.2, now - 0.9), "flasks": (3, now - 0.05)}, now))
print(decide({"hp": (0.2, now - 0.05)}, now))
print(record("heal", "hp 0.20", shown=False))

Exercise

Pick two very different streams you could watch with software: say, a video game's health bar and a kitchen timer's beep. Sketch a Detector root with one vision child and one audio child, two reader kinds for each, and a profile (as JSON) for each stream that names readers and their numbers. Then write a test that runs a third, invented stream on the same classes. Which constant did you almost put in a class?

Hint

If a reader class contains a pixel coordinate, a colour or a loudness threshold, it has learned a game. Move every such number into the spec the profile passes in.