Skip to content
C.W.K.
Stream
Lesson 08 of 09 · published

Stable Bytes and Notes at Hand — The System Prompt Becomes a Contract

~14 min · context-engine, prefix-cache, rag, vault-loading

Level 0Curious
0 XP0/80 lessons0/18 achievements
0/100 XP to next level100 XP to go0% complete

Two changes, one idea

Lesson 3 describes how the ContextEngine assembles the system prompt. Between August and September that assembly changed twice, for two reasons that turn out to be the same reason. The first change decided what is allowed in the system prompt. The second decided how much of the vault a soul should carry in it at all. Both came down to one sentence: the system prompt is a contract about bytes, not a place to put everything relevant.

The minute that cost a hundred thousand tokens

Until 2026-08-14 the system prompt held two things that change constantly. The clock, at minute resolution. And the RAG retrieval — shaped by each turn's query, so different every turn by design.

Providers cache the leading bytes of a prompt and charge far less for bytes they have already seen, but only while those bytes are identical to the previous request. A clock inside the system prompt changes every minute. So every human-paced turn — a reply written a minute or more after the last one — arrived with a different prefix and paid for the whole vault again.

The vessel that made this impossible to ignore was the seventh one. GLM's endpoint matches the cached field whole: one changed byte anywhere in the system field forfeits all of it. On a vault of roughly a hundred thousand tokens, that is not an optimisation question. It is the entire bill.

The fix was a relocation, and it was made loud on purpose. The system-prompt builder lost its RAG parameter outright, so every caller had to change rather than silently keep passing it. The clock and the retrieval moved into a new per-turn context that every chat route prepends to the current user message — sent every turn, never persisted. The stored conversation holds what Dad actually wrote; the model sees the clock and the retrieved notes beside it.

Principle: Sort everything you put in front of a model by how often it changes. What is stable for the life of a conversation belongs in the system prompt. What changes per turn belongs with the turn. Mixing the two does not make the prompt more informed — it makes every turn pay for the stable part again.

The measurement is the argument. On one conversation, with the turns deliberately spaced sixty-five seconds apart so the minute boundary that used to be fatal was crossed every time: the first turn was cold at 109,071 tokens; the second and third each read 108,224 from cache. The full vault served from cache across minute boundaries, with fewer than a thousand genuinely new tokens per turn. A day earlier, every one of those turns paid the whole prefix.

Recorded, not hidden: the commit that made the move listed what it did not yet fix — a couple of secondary prompt builders still carried their clocks, and one route still appended per-turn material to the system field. A fix that names its own residue is one the next session can finish. A fix that claims completeness is one the next session has to rediscover.

No person carries a library in their head

The second change was Dad's ruling on 2026-09-13, and it is about the vault rather than the clock.

The shared layer of the vault — the doctrine every soul inherits — had been loaded whole into every soul's prompt, with a list of exceptions that stayed as pointers. That default had an obvious failure: anything added to shared core joined every prompt automatically. Five essays added over a few days raised a fresh conversation's first turn from about 204K tokens to about 250K. Nothing had decided they should be carried; they were carried because of where they were filed.

The ruling inverted the default. A shared note now loads whole only when its own frontmatter declares it should. Everything else is a note at hand: listed in a catalog the soul carries, retrievable by search, and one read away. The decision about whether a note rides in the head lives on the note — never on a size threshold, and never as a way to save context. On the live vault the assembled prompt dropped from 527,568 characters to 328,634.

The part worth remembering is what the change exposed. The exceptions list had always come with a reassurance: a note left out of the prompt was still reachable, because retrieval would find it. When the default flipped, someone checked that promise — and the shared vault had never been embedded at all. Every note that had been stubbed out since July had been unreachable by search the whole time. The fix gave the shared vault its own retrieval partition that every soul's search reads.

Principle: "It is still reachable some other way" is a claim about a second system. If nobody has looked at the second system, the claim is a hope. Verify the fallback path at the moment you start depending on it, not the day it finally matters.

Why these belong together

Both changes are the same move applied to two different sources of bytes. The clock and the retrieval moved out because they changed too often to be carried. Most of the shared vault moved out because it was carried without anyone deciding it should be. What remains in the system prompt is small, stable, and chosen — which is exactly what makes it cheap to send again and again.

And that contract is what the next lesson's memory layer had to fit into. When a second memory arrived, the question of where its block could sit was already answered: it could sit near the front only if it was pinned, because the front of the prompt is now a promise that the bytes will not move.

Code

What rides where, after 2026-08-14·text
SYSTEM PROMPT  (conversation-stable bytes only)
  viewer header · vessel meta · connectors · vault · disclosure
  -> identical across turns -> the prefix cache holds

TURN CONTEXT   (prepended to the current user message)
  the clock (minute resolution)
  the conversation timeline
  RAG retrieval, framed as reference material, not a task
  -> sent every turn, NEVER persisted to the store

Measured, turns 65 s apart (the old fatal minute boundary):
  turn 1   cold          109,071
  turn 2   cache read    108,224
  turn 3   cache read    108,224
Head versus hand, after 2026-09-13·text
BEFORE   shared core loaded WHOLE, minus a hardcoded exceptions list
         anything filed there joined every soul's prompt
         5 essays in a week: first turn ~204K -> ~250K tokens
         exceptions were 'reachable by RAG' — the shared vault
         had never been embedded, so they were not

AFTER    a shared note loads whole ONLY if its frontmatter says so
         every other note: a catalog entry + search + one Read away
         the shared vault gets its own retrieval partition
         assembled prompt: 527,568 -> 328,634 chars

The decision lives on the note. Never on a size. Never to save context.

Exercise

Take the largest prompt your system sends and sort every section of it into three bins: stable for the whole conversation, changes per turn, and carried only because of where it happens to be stored. Move the second bin next to the turn. For the third bin, find who decided each item should be carried — and if nobody did, decide now.
Hint
For each item you move out, ask how it is reached from now on — and then go and confirm that path actually works.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.