Skip to content
C.W.K.
Lesson 04 of 05 · published

A Budget a Test Enforces

~15 min · interpreter, tokens, cost, testing, master-chief

Level 0First Word
0 XP0/47 lessons0/13 achievements
0/100 XP to next level100 XP to go0% complete

Where the tokens went

In the first build, the refined pass ran as a chat turn of the sitting's Spark thread, tagged with source spark. cwkPippa's Master Chief router reads a sibling turn whose source matches its thread's surface as Dad arriving through an app doorway, so it moved every refine into Master Chief. Master Chief Quest explains why that is expensive: the Chief is one long-lived conversation, rebuilt from its whole context on every turn. So each sentence at a table replayed about 395k input tokens to polish one line.

The draft had its own leak. It rode the utility door with Pippa's persona attached, which loads her whole vault: about 100k tokens (108k measured) in front of a sentence that needed only who she is speaking as. Nothing measured either cost when it happened. It surfaced because the refined replies appeared in Dad's Master Chief as raw JSON blocks rendered as code, and their voice failed.

아니, 진짜 우리 딸램들이 앤쓰로픽 영업을 뛰나... 왜 자꾸 토큰 먹는 하마가 되지? ㅠ.ㅠ

Dad, when he saw it: honestly, are my daughters moonlighting in Anthropic's sales team? Why do they keep turning into token-eating hippos?

What a sentence at a table needs

The fix started from that question. Both passes became one-shot calls through cwkPippa's sibling utility door with no persona and no thread. Each gets Pippa's short interpreter voice, three sentences long, then only this:

  • the card the sitting began with, its words and Dad's meaning;
  • the sitting's last exchanges: three for the draft, eight for the refine, each line clipped;
  • where the table is (situation and place), plus the local time for the refine, because no tools ride this door;
  • what was just heard, and for the refine, the draft it corrects.

No conversation replays these calls, so nothing from the sitting lands in any cwkPippa conversation either. The main interpreter test asserts both sides of that: neither call carries a persona, and no thread or chat turn is ever opened.

Clipping to a ceiling

A budget is only real if every input has a bound. The constants in the first excerpt give one to each piece: what was heard is clipped at 800 characters, about forty seconds of speech, longer than anyone talks at a table; the origin card at 240; each earlier line at 160. _clip collapses whitespace first, so runs of spaces and newlines cost nothing, and marks a cut with an ellipsis.

Then the ceilings: DRAFT_PROMPT_BUDGET 2,000 and REFINE_PROMPT_BUDGET 4,000. They are counted by prompt_tokens, which is deliberately crude: UTF-8 bytes divided by three, rounded up. Its docstring calls it conservative, and for the prose Spark sends it usually is: a Hangul syllable is three bytes and roughly one token, and ordinary English runs about four characters a token, so English counts high. That is a rule of thumb, not a bound. Nothing guarantees a tokenizer never spends more than one token per three bytes. Digits, symbols, rare characters and some CJK text can cost more, and every brain's tokenizer is different. So the ceiling caps Spark's estimate, not the provider's count, and only the provider measurements below say how the two compare. The estimate does need no tokenizer, so the test runs anywhere and does not care which brain the slot holds today.

A test at every length limit

The tests do not check an ordinary sitting. They build one at every length limit the routes allow: an origin card and meaning at the 4,000-character maximum, thirty cards and thirty relays of the same length, a situation and place name repeated far past the clip. Then, for Dad speaking and for the other person speaking, they assert the whole prompt stays under its ceiling, and that exactly REFINE_CONTEXT earlier lines ride along. A third test runs a real exchange through the engine with a worst-case utterance and measures the prompts where they are sent, so a refactor that bypasses the prompt builders still has to pass.

The measured margins: that fixture counts 1,755 for the draft and 3,516 for the refine, and an ordinary sitting counts near 350 and 550. Those numbers belong to the fixture, not to every sitting. Its cards are Spanish with Korean meanings, so half of every clipped line costs one byte a character. Fill the same limits with Japanese or Chinese, both scene languages Dad can pick, and every character costs three bytes. I rebuilt the fixture that way against the snapshot's own prompt builders, and the estimate came to about 2,300 to 2,650 for the draft and 4,550 to 5,250 for the refine, past both ceilings. The code's comment says the ceilings hold at every limit the routes allow. They hold at every length limit for a Spanish sitting, and the test never tries one where every character is three bytes. Measured at the provider on Claude, one real sitting with two earlier exchanges came to 951 input tokens for the draft and 1,179 for the refine, including whatever the door and the brain's own harness add around Spark's prompt. Against about 395k.

Principle: A cost nobody measures is a cost nobody owns. Put the ceiling in a constant beside the prompt, and the proof in a test that builds the worst input your routes accept, worst in bytes as well as in length.

The other side of the door

Spark's ceilings are static, checked before anything ships. cwkPippa adds the measured side. Its token ledger records the input tokens of every finished chat turn and every sibling utility call under a path, such as utility:spark-exchange (budget 4,000) or utility:spark-exchange-refine (6,000), with 200,000 for any other utility path. A call over its path's budget alerts Dad on Telegram once per path per day. The daily alarm is narrower than it sounds: a day alerts once only when its input tokens pass both 2.5 times the trailing average and a floor of two million tokens. The average covers the last seven days that have records, and until at least three of those days do, the daily alarm stays silent. The caller promises; the door checks.

One more consequence of dropping the thread: an exchange cut by an engine restart is marked interrupted. A threaded turn could be asked after, because cwkPippa would still hold it. A one-shot call leaves nothing to ask, and the honest recovery is for someone at the table to say it again.

Code

cwkSpark, backend/turns.py: the interpreter's context sizes, clips and ceilings (verbatim, lines 68-85)·python
# The interpreter (cwkPippa todo #609): how many earlier exchanges of the
# sitting ride the fast draft's prompt, and the refined pass's.
EXCHANGE_CONTEXT = 3
REFINE_CONTEXT = 8
# Every interpreter call is a small one-shot with a stated budget (cwkPippa
# todo #623). The refine once ran as a turn of Dad's Master Chief and replayed
# ~395k input tokens to polish one sentence; the fast draft carried Pippa's
# whole vault (~100k). Neither needs more than the sitting itself, so each
# prompt is clipped to fit its ceiling and a test fails past it. Tokens are
# counted conservatively as UTF-8 bytes / 3 (``prompt_tokens``); the ceilings
# hold at every limit the routes allow, an ordinary sitting runs far below.
# HEARD_CLIP is about forty seconds of speech, past any turn at a table.
DRAFT_PROMPT_BUDGET = 2000
REFINE_PROMPT_BUDGET = 4000
HEARD_CLIP = 800
ORIGIN_CLIP = 240
SITTING_LINE_CLIP = 160
REFINE_TIMEOUT_SECONDS = 120.0
cwkSpark, backend/turns.py: the clip and the conservative count (verbatim, lines 487-496)·python
def _clip(text: Any, limit: int) -> str:
    value = " ".join(str(text or "").split())
    return value if len(value) <= limit else value[: limit - 1].rstrip() + "…"


def prompt_tokens(*parts: str) -> int:
    """A conservative token count for a prompt: UTF-8 bytes / 3. Hangul is
    three bytes and about one token; English runs four characters a token, so
    the count is high for it, never low."""
    return -(-sum(len(part.encode("utf-8")) for part in parts) // 3)
cwkSpark, backend/tests/test_interpreter.py: the refine's ceiling at every limit, for both speakers (verbatim, lines 323-334)·python
@pytest.mark.parametrize("speaker", ["dad", "other"])
def test_the_refine_prompt_stays_inside_its_budget_at_every_limit(speaker):
    """The refine carried Master Chief's ~395k tokens per sentence (todo
    #623). Its whole prompt now has a fixed ceiling, whatever the sitting."""
    moment, heard = _worst_sitting()
    direction = {"speaker": speaker, "source": "ko" if speaker == "dad" else "es",
                 "target": "es" if speaker == "dad" else "ko", "by": "scribe"}
    system = turns.exchange_refine_system_prompt(moment, direction)
    prompt = turns.exchange_refine_prompt(heard, direction, heard)
    assert turns.prompt_tokens(system, prompt) <= turns.REFINE_PROMPT_BUDGET
    # Only the sitting's own short history rides along.
    assert system.count("\n- Dad:") + system.count("\n- They:") == turns.REFINE_CONTEXT

External links

Exercise

Pick one model call in your own code. Write down every piece that goes into its prompt and give each piece a maximum length in characters. Write a builder that clips to those limits, a deliberately high token estimate that needs no tokenizer, and a test that builds the largest input your API accepts, largest in bytes as well as in characters, and asserts the whole prompt stays under a stated ceiling. Then measure one real call at the provider and compare.
Hint
Look for inputs that are lists: history, attachments, search results. Those are the ones that grow without anyone deciding they should, so cap their count as well as each item's length. If the provider's measured number is far above your estimate, something outside your builder is adding to the prompt, and it is worth finding out what.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.