A Listener Is Not an Identity

"Something is listening on 8100. Cute. But is it Ember — or a ghost wearing Ember's port?"

The Trap in a Green Light

Here's the naive health check: connect to the declared port, and if something answers, call it 'running.' It feels reasonable and it's quietly wrong. A TCP listener only proves that some process is bound to that port — not that it's the product you expect. It could be a leftover server from last week that never got killed. It could be a different app that grabbed the port first. It could be a placeholder, a proxy, a totally unrelated process. Every one of those makes the naive check show a confident green light for a service that is, in truth, not there.

Two Checks That Prove Identity

Firelink refuses to be fooled, so it verifies responder identity with two checks. First, the port-owner check: for an office launchd member, the declared localhost port must be owned by the job's own process tree — not merely open, but held by the process the launchd job actually spawned. Second, the health-identity check: a member with a declared health path must answer that path with the expected identity substring — the response has to say, in effect, 'yes, I'm Ember.' Pass both and it's genuinely running. Fail either one and Firelink reports unknown with the mismatch detail — never running.

Why 'Unknown' Is the Honest Verdict

Notice it doesn't report 'stopped' on a failed identity check either. Something is answering — it's just not provably the right something. That's not 'down'; it's 'I can't confirm this is what it claims to be.' Calling it running would be a lie; calling it stopped would also be a lie. unknown, plus the specific mismatch, is the only honest answer — and it's exactly the state a human needs to see to go investigate 'what is actually sitting on that port?'

A response is not proof of identity — verify that the responder is who it claims to be, and downgrade to 'unknown' when you can't. 'Something answered' and 'the right thing answered' are different facts. Systems that treat a mere connection as health will one day route real traffic to a ghost and call it healthy the whole time.

Code

Verify the responder — don't trust the listener·python

def verify_service(member) -> ServiceState:
    if not tcp_listening(member.host, member.port):
        return ServiceState("stopped")            # nothing there at all

    # Something is listening. That is NOT enough. Prove it's the right thing.

    # Check 1: port-owner — is the port held by this launchd job's process tree?
    if member.launchd_owned:
        owner = port_owning_pid(member.port)
        if owner not in process_tree(member.launchd_label):
            return ServiceState("unknown", why="port owned by a foreign process")

    # Check 2: health-identity — does the health path say it's really this member?
    if member.health_check:
        body = http_get(member.health_check.path)
        if member.health_check.identity_substring not in body:
            return ServiceState("unknown", why="health response identity mismatch")

    return ServiceState("running")                 # earned it: bound AND proven

# A leftover process on :8100 fails check 1 or 2 -> reported 'unknown',
# never a green 'running'. The port being open is not the answer.

Exercise

Take any health check you rely on and ask: what exactly does 'up' mean here? Is it 'the port accepts a connection,' 'an HTTP 200 came back,' or 'the response proved it's the right service and version'? Construct a concrete scenario where the weaker check reports healthy but the real service is not there (a leftover process, a proxy, a maintenance page returning 200). Then describe the identity check that would catch it.

Hint

Ladder of strength: port-open < connection-accepted < HTTP 200 < response contains an identity/version marker. Most outages that 'monitoring missed' were caught by nothing stronger than 'the port was open.' Push your check up the ladder until it proves identity, not just presence.

A Listener Is Not an Identity

The Trap in a Green Light

Two Checks That Prove Identity

Why 'Unknown' Is the Honest Verdict

Code

External links

Exercise

Progress

Comments 0