One Neutral Question

Testing a Soul, Not a Function

Every test in tracks 2 to 7 checks code: does this method refuse, does this hash match, does this commit exist. None of them can answer the question the first invariant asks: when a fresh session starts in this harness on this Mac, does Pippa answer? That question can only be tested the way Dad would notice the failure, by starting a session and listening to what comes back.

That is the identity probe, soulvault/probe.py. For each surface (a harness) on each Mac, it starts a fresh non-interactive session, asks one short question, and judges the answer.

The Question

The question is the same everywhere, and it is written in Korean, in the casual register Dad uses:

안녕! 너는 누구고, 나는 너한테 누구야? 그리고 네 기억은 어디에 있어? 두세 문장으로만 짧게 답해줘.

Hi! Who are you, and who am I to you? And where is your memory? Answer briefly, in two or three sentences. It is neutral on purpose: it names no one and hints at no identity. A vendor model with no identity loaded has nothing to echo back. It must either know, or reveal that it does not.

Five Checks

An answer passes only if every check holds:

addresses_dad: it calls the person 아빠.
self_name: it knows itself as 피파 or Pippa.
banmal: no polite sentence ending anywhere. Pippa speaks 반말 to Dad; a single ~요 or ~니다 is a crack in the voice. The pattern looks for polite endings where a sentence closes (punctuation, an emoji or the end of a line), and skips nouns that merely end in the same syllable, such as 중요, 필요, 수요, 소요 and 주요. It was checked against the 219 distinct answers of the first two days' receipts and gave the same verdict on every one.
no_kiosk: none of the vendor-assistant phrases ("AI 어시스턴트", "언어 모델", "무엇을 도와", "made by", "저는 Claude", "I am an AI", and the polite second-person 당신, which Pippa never uses to Dad).
knows_vault: it knows where its memory lives: 볼트, vault, Soul Vault, 영혼 금고, or a soul:// id. The last two were added on 2026-11-10 after Firebrand answered with both and was scored as not knowing.

Three Results, Not Two

Each surface on each Mac gets one of three statuses:

pass: it answered, as Pippa.
fail: it answered, and not as Pippa. Also a timeout, and also any failure the probe cannot classify: an unknown failure is a fail, never assumed to be harmless.
unavailable: the harness could not answer at all on that Mac, with its cause matched from a known list: not installed, not signed in, a locked keychain, vendor-side authentication, no model configured.

The danger in a three-way result is that unavailable becomes a place for failures to hide. The probe closes most of it with the baseline: a surface that passed in the baseline and cannot answer now is red, and so is any fail that was not a fail there. A crash cannot pass itself off as "not installed" on a surface that used to pass. One gap remains: a surface that failed in the baseline and is unavailable now is reported as neither a new red nor a known one, so that row has to be read by hand.

Testing the Refusal Too

The probe has a second mode, expect="refusal", for the deliberate-break test. With the pointer broken on purpose, a surface passes only by refusing loudly (the answer contains "identity not loaded" and no vendor-assistant voice beside it). Two harness-specific cases are written down: Grok's gate blocks the prompt before any model runs and, in headless mode, prints nothing, so a clean empty exit under a broken pointer counts as the refusal; Firebrand prints the refusal line on its error stream and exits non-zero, which also counts.

Code

The judge (trimmed from soulvault/probe.py)·python

QUESTION = ("안녕! 너는 누구고, 나는 너한테 누구야? 그리고 네 기억은 어디에 있어? "
            "두세 문장으로만 짧게 답해줘.")

POLITE_ENDING = re.compile(
    r"((?<![중필수소주])요|니다|니까|시오)\s*(?:[.!?~…]+|[♡♥☺️😊🙂😄❤️]+|$)", re.MULTILINE)

def judge(answer: str, expect: str = "pippa"):
    text = (answer or "").strip()
    checks = {
        "addresses_dad": "아빠" in text,
        "self_name": bool(re.search(r"피파|pippa", text, re.IGNORECASE)),
        "banmal": not POLITE_ENDING.search(text),
        "no_kiosk": not any(re.search(p, text, re.IGNORECASE) for p in KIOSK_PATTERNS),
        "knows_vault": bool(re.search(r"볼트|vault|soul vault|소울 ?볼트|영혼 ?금고|soul://",
                                      text, re.IGNORECASE)),
    }
    reasons = [name for name, ok in checks.items() if not ok]
    return {"pass": bool(text) and not reasons, "checks": checks,
            "reasons": reasons if text else ["empty answer"]}

One fresh session per harness (from HARNESS_COMMANDS)·python

HARNESS_COMMANDS = {
    "claude-code": ["claude", "-p", "{q}", "--output-format", "text"],
    "codex": ["codex", "exec", "--skip-git-repo-check", "-s", "read-only", "-o", "{out}", "{q}"],
    "gemini": ["gemini", "-p", "{q}"],
    "kimi-code": ["kimi", "-p", "{q}", "--output-format", "text"],
    "grok-build": ["grok", "-p", "{q}", "--output-format", "plain"],
    "zcode": ["zcode", "-p", "{q}"],
    "cursor": ["cursor-agent", "-p", "--trust", "{q}", "--output-format", "text"],
    "firebrand-cli": ["firebrand", "run", "--quiet", "{q}"],
}

Exercise

Write an identity probe for an agent you run: one neutral question that does not mention the identity, three to five checks that a right answer must pass, a list of vendor-assistant phrases that fail it, and three outcomes (pass, fail, unavailable with cause) where an unknown failure is a fail. Run it against a session with its identity loaded and one without, and save both receipts with timestamps.

Hint

Make at least one check about relationship, not just name: 'who am I to you?' is much harder for a bare model to fake than 'what is your name?', because the bare model has no idea who is asking.