One Window, Not a Version

The Deploy Order, and Why It Is an Order

When a wire contract changes — a field's vocabulary, a response envelope, a route's shape — the deploy runs in one sequence: the owning service first, then the shared layer, then each consumer engine, then all of them restarted in the same window.

The order follows the direction of the dependency, and the last step is the one people skip. Everything can be on disk and still be wrong, because a running process holds the code it imported at startup. Deploying without restarting produces a system where the files say one thing and the behavior says another, which is the most confusing state available.

Why There Is No Mixed-Version Grace Period

Versioned APIs are designed for exactly this window: old and new coexist by contract, consumers upgrade on their own schedule, and the mixed state is supported.

This family has none of that, and does not want it. Version skew is the defect the shared repository exists to remove — a version pin here would make skew a supported feature rather than a bug. The consequence is that the mixed window is not a supported state. During it, every component behaves correctly against a contract that half the system has not adopted yet.

That is worse than a broken window, because a broken window announces itself. In a half-deployed window, things mostly work: the request goes through, the response parses, one field means something slightly different than the receiver assumes. Test in that window and the results describe a configuration that will not exist in a few minutes.

A half-deployed window manufactures false confidence, so never test in one and never demonstrate from one. Everything is individually correct and collectively wrong, which produces plausible results with no relationship to the system that will exist afterward. Finish the window, restart everything, then look. A green result taken mid-deploy is not weak evidence — it is evidence about a system nobody is going to run.

Two Practical Consequences

First: keep the window short and scripted. Anything that makes the sequence longer — a manual step, a confirmation, a service that has to be found before it can be restarted — extends the interval in which the system is in an unsupported state. The sequence should be one command that does all of it.

Second: a landed change is not a deployed change. A commit that alters module-level data — a registry, a route table, a policy map — reaches a long-running process only when that process restarts. Everything can be committed, pushed, and correct on disk while the running system continues to behave the old way indefinitely. The failure that follows is disorienting, because reading the code tells you the opposite of what you are observing, and the code is what everybody trusts.

Code

The window, scripted — and the state that is not supported·bash

#!/usr/bin/env bash
# One command, one window. Every manual step lengthens the interval
# during which the system is in a state nobody designed for.
set -euo pipefail

# 1. the OWNER first - it defines the contract
deploy owner-service && restart owner-service

# 2. the shared layer - vendored copies into every consumer
python sync/kit_sync.py

# 3. each consumer engine
for app in app-a app-b app-c; do deploy "$app"; done

# 4. RESTART THEM TOGETHER. The step people skip, and the one that
#    makes the previous three real: a running process holds the code
#    it imported at STARTUP, so files on disk change nothing until
#    this line runs.
for app in app-a app-b app-c; do restart "$app"; done

# Only now is it meaningful to look at anything.
for app in app-a app-b app-c; do healthcheck "$app"; done


# THE STATE THAT IS NOT SUPPORTED - and why it is worse than broken:
#
#   owner    deployed   speaks the NEW vocabulary
#   shared   deployed   translates for the NEW vocabulary
#   app-a    deployed   but NOT RESTARTED -> still runs the OLD code
#   app-b    not yet deployed
#
# Nothing crashes. Requests go through, responses parse, and one
# field means something slightly different than the receiver
# assumes. Any test run here describes a configuration that will
# not exist in five minutes.


# THE DIAGNOSTIC for "landed but not deployed", which is the same
# failure at rest: compare the process start time against the
# modification time of the file whose behavior you are expecting.
#
#   ps -o lstart= -p "$(pgrep -f app-a)"
#   stat -f '%Sm' app-a/registry.py
#
# A file NEWER than the process is the whole explanation, and it is
# the fastest way out of an argument with your own source code.

Exercise

For a service you operate, find one piece of configuration that is loaded at import or startup rather than per request — a registry, a route table, a feature map. Then check how a change to it currently reaches production, and whether anything in your deploy would restart the process. If not, write down what the symptom would be, and where somebody would look first.

Hint

The symptom is always 'the code says X and the system does Y', and the first place people look is the code — which is why this class of failure survives long past when it should. Make the diagnostic step explicit in your runbook so the second question is about the process, not the source.

The Deploy Order, and Why It Is an Order

Why There Is No Mixed-Version Grace Period

Two Practical Consequences

Code

External links

Exercise

Progress

Comments 0