Skip to content
C.W.K.
Lesson 05 of 05 · published

Sent at Most Once

~14 min · at-most-once, write-call, outcome-unknown, connection-pool, failure-modes

Level 0Loose File
0 XP0/43 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

Every Read Convenience Is a Second Send

Delta's read path is built to try hard. A pooled keep-alive connection that turns out dead is replaced and the request retried; a LAN route that fails mid-request is dropped and the request goes again over the tailnet; when this Mac's own Delta is down, kit_delta reads the peer's gateway directly; and the cache can answer without asking anyone. For a read, every one of those is free: reading a thumbnail twice changes nothing.

For a write, each of them is a second send. If the first send reached the owner and only the answer was lost, the retry writes the note twice, or (with compare-and-swap) is refused as a conflict for a write that actually happened. So the write call gives up every one of those conveniences. Delta's invariant 11 says it plainly: a write call is sent at most once, and Delta says whether it left.

The Line That Matters: Did It Leave?

A remote write can fail in three places, and only one of them is ambiguous. PeerClient.call is built around that line:

Where it failedOutcomeWhat the app does
Before the request left: no route, a connect that fails or is on hold, a ticket that cannot be minted, a certificate that does not match, this Mac's own Delta downunavailable (the kit: local_delta_down when it is this Mac's Delta), at once; the detail ends "nothing was sent"Nothing was written. Try again later, or tell the person.
After it left, with no answer: a cut, or silence past the time limitoutcome_unknown, HTTP 502Ask the owner by the request id. Never send a new request.
The owner answeredits result, or its refusal under its own name and status, passed through as it cameRead the answer.

Everything before conn.request(...) is wrapped so that any exception becomes "nothing was sent". One recovery is allowed there: a LAN route that fails before the send is dropped and one freshly chosen route is tried, because the first route carried nothing. After the send, there is no retry of any kind.

A Connection of Its Own

Why does a call refuse the connection pool? A pooled connection that the server closed while it sat idle does not fail when you pick it up. It fails after the request has been written into it, and at that point nobody can say whether the owner read the request before the connection died. Delta's read path papers over this by retrying on a fresh connection, which is harmless for a read. A write call cannot, so it opens a new connection every time (_open_for_call) and uses it once. The kit does the same on loopback: the app's call to its own Delta is a one-use connection too, for the same reason one hop earlier.

The Time Limit, and the One Exception

The caller sets how long the owner may take: X-Delta-Timeout, in seconds, 30 if absent, at most 600; anything else is bad_request before the call leaves. Connecting is bounded separately by Delta's own connect timeouts, so a slow owner and an unreachable one stay different outcomes.

There is exactly one case where a call goes a second time, and it is not a second send of the work. On a Thunderbolt or LAN route the request carries a ticket; if the owner's gateway answers 401, it refused the ticket at the door, before any verb ran. Nothing was carried out. So the call renews the ticket, opens a new connection, and goes once more under it. The owner still carries it out at most once. A 401 on the renewed attempt is passed back as the answer.

No Way Around This Mac's Delta

A read falls back to the peer's gateway when the local Delta is down. A write does not, and cannot: the local Delta and its local token are what vouch for which Mac and which account is asking. An app that went around it would be an anonymous caller. So kit_delta's call() fails with local_delta_down, "nothing was sent", and Soul Vault turns that into canonical_unreachable: nothing was written, and the text stays with the caller. Its outcome_unknown path keeps the request id and says to ask again with the same id, which the store answers with its first result.

Measured, Including the Awkward Second

With the owner's Delta stopped, calls came back unavailable in 5 to 9 ms once the route had fallen to the tailnet; the first caller, who had to drop the LAN route and try the tailnet, waited 0.4 to 1.6 s. One more fact came out of that measurement. A Delta that is told to stop keeps answering until its listeners' next poll returns, about a second, and a write call that arrives in that second is carried out like any other. One did, during the proof: a fixture revision was written into the gap. So "unreachable" begins when the port refuses, not when the stop is sent, and a test that needs the owner gone waits for the refusal.

Code

PeerClient.call: unavailable before the send, outcome_unknown after (trimmed excerpt)·python
# delta/peer.py (trimmed excerpt): PeerClient.call, one write sent at most once
def call(self, label, path, *, body, headers=None, app=None, read_timeout=READ_TIMEOUT, force=None):
    def nothing_sent(exc):
        return Unavailable(f"{why(exc)}; nothing was sent to {label}")

    # Whatever fails from here to the send, of any kind, failed before the call left.
    try:
        route = self.delta.routes.get(label, force=force)
    except Exception as exc:
        raise nothing_sent(exc) from None
    try:
        conn, ticket = self._open_for_call(route, label)      # a new connection, never a pooled one
    except Exception as exc:
        if route.kind == "tailnet" or force:
            raise nothing_sent(exc) from None
        # The LAN route failed before anything left: drop it and choose once more.
        self.delta.routes.invalidate(label, why(exc))
        try:
            route = self.delta.routes.get(label, refresh=True)
            conn, ticket = self._open_for_call(route, label)
        except Exception as again:
            raise nothing_sent(again) from None

    # hdrs: Content-Length, X-Delta-App, X-Delta-Request-Id, built above
    renewed = False
    while True:
        if ticket:
            hdrs["Authorization"] = f"Delta {ticket}"
        try:
            conn.sock.settimeout(read_timeout)
            conn.request("POST", path, body=body, headers=hdrs)
            resp = conn.getresponse()
        except socket.timeout:
            conn.close()
            raise OutcomeUnknown(f"{label} did not answer the call within {read_timeout:g} s. The call had "
                                 "left this Mac, so it may or may not have been carried out; it is not sent again")
        except (http.client.HTTPException, ssl.SSLError, OSError) as exc:
            conn.close()
            raise OutcomeUnknown(f"the connection to {label} ended during the call ({type(exc).__name__}). "
                                 "It may or may not have been carried out; it is not sent again")
        out = Response(self, self._key(route), conn, resp, route, label, read_timeout, "POST")
        if out.status == 401 and route.needs_ticket and not renewed:
            # Refused at the door, before any verb ran: nothing was carried out. A new
            # ticket, a new connection, and the call goes once under it.
            renewed = True
            out.read()
            ticket = self.ticket(label, renew=True, refused=ticket)
            conn = self._connect(route, label)
            continue
        return out
At most once, on loopback: one answered call, one that left and timed out, one that never left·python
# at_most_once.py: tell "nothing was sent" from "it left and no answer came back"
import http.client
import json
import socket
import threading
import time
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

carried_out = []            # what the owner actually did


class Owner(BaseHTTPRequestHandler):
    def do_POST(self):
        body = json.loads(self.rfile.read(int(self.headers["Content-Length"])))
        carried_out.append(self.headers["X-Request-Id"])
        time.sleep(float(body.get("think", 0)))       # the owner's work, slow on purpose
        answer = json.dumps({"ok": True}).encode()
        self.send_response(200)
        self.send_header("Content-Type", "application/json")
        self.send_header("Content-Length", str(len(answer)))
        self.end_headers()
        self.wfile.write(answer)

    def log_message(self, *args):
        pass


def call(port: int, request_id: str, body: dict, timeout: float) -> str:
    """One write, sent at most once, on a connection of its own."""
    conn = http.client.HTTPConnection("127.0.0.1", port, timeout=2.0)
    try:
        conn.connect()
    except OSError as exc:
        return f"unavailable: {exc.strerror or exc}; nothing was sent"
    try:
        conn.sock.settimeout(timeout)
        conn.request("POST", "/call/write", body=json.dumps(body),
                     headers={"Content-Type": "application/json", "X-Request-Id": request_id})
        resp = conn.getresponse()
        return f"answered {resp.status}: {resp.read().decode()}"
    except (socket.timeout, OSError, http.client.HTTPException) as exc:
        return f"outcome_unknown: {type(exc).__name__}; it left, so it is not sent again"
    finally:
        conn.close()


server = ThreadingHTTPServer(("127.0.0.1", 0), Owner)
port = server.server_port
threading.Thread(target=server.serve_forever, daemon=True).start()

print("1.", call(port, "r-1", {"think": 0}, timeout=1.0))
print("2.", call(port, "r-2", {"think": 1.5}, timeout=0.5))
server.shutdown()
server.server_close()
print("3.", call(port, "r-3", {"think": 0}, timeout=1.0))
time.sleep(1.2)                                    # let r-2 finish on the owner's side
print("the owner carried out:", carried_out)

External links

Exercise

Run the at-most-once demo and confirm that r-2 was carried out even though the caller was told outcome_unknown. Then add the missing half: give the owner a GET endpoint that answers what it did for a request id, and change the caller so that, after outcome_unknown, it asks that endpoint (never re-sending the POST) and prints the settled outcome.
Hint
Store each request id's answer in the owner before replying (a dict is enough here). The status query must be a read: asking it twice is harmless, which is exactly the property the write lacks. Wait until the owner's slow call has finished before asking, or answer "still running" for an id the owner has seen but not finished.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.