Error Recovery & Graceful Deployment

Connection draining

During a deploy, the new server boots with zero connections; the old server has thousands. Killing the old server outright dumps all of them at code 1006. The graceful pattern: tell clients to reconnect (they will reach the new server), wait briefly for them to disconnect on their own, then close the rest with code 1001 ("going away"). cwkPippa's lifespan example in Track 4 lesson 5 is exactly this pattern.

Client-side message buffering

During the brief disconnect window, the client may try to send. A robust client buffers locally and flushes on reconnect. This is the Track 5 ACK pattern but local: queue when not OPEN, send on next OPEN, never lose a user-typed message because of a deploy.

Rolling deploy

Behind a load balancer with sticky sessions, take down one server at a time. Connections on that server reconnect to the remaining servers. Repeat for each server. Zero global downtime; transient per-user reconnect.

Code

Server: graceful drain on SIGTERM·python

from contextlib import asynccontextmanager
import asyncio, signal

shutdown_event = asyncio.Event()

@asynccontextmanager
async def lifespan(app):
    # Trap SIGTERM so we can drain instead of crashing.
    loop = asyncio.get_running_loop()
    for sig in (signal.SIGTERM, signal.SIGINT):
        loop.add_signal_handler(sig, shutdown_event.set)
    yield
    # Drain phase
    log.info('draining %d connections', manager.total_connections)
    for room in list(manager.rooms):
        await manager.broadcast(room, {
            'type': 'system.reconnect',
            'data': {'reason': 'deploy'},
        })
    await asyncio.sleep(2.0)  # let clients reconnect on their own
    close_tasks = [
        ws.close(code=1001, reason='deploy')
        for ws in list(manager.ws_meta)
    ]
    await asyncio.gather(*close_tasks, return_exceptions=True)
    log.info('drain complete')

app = FastAPI(lifespan=lifespan)

Client: buffered send·javascript

class BufferedSocket {
  constructor(url) {
    this.url = url;
    this.buffer = [];
    this._connect();
  }
  _connect() {
    this.ws = new WebSocket(this.url);
    this.ws.addEventListener('open', () => {
      while (this.buffer.length) this.ws.send(this.buffer.shift());
    });
    this.ws.addEventListener('close', () => {
      // schedule reconnect (Track 2 backoff pattern)
      setTimeout(() => this._connect(), 1_000);
    });
  }
  send(msg) {
    const wire = JSON.stringify(msg);
    if (this.ws?.readyState === WebSocket.OPEN) this.ws.send(wire);
    else this.buffer.push(wire);
  }
}

Exercise

Implement the lifespan drain + the BufferedSocket. Trigger a deploy via kill -TERM <pid> while three test clients are connected. Confirm: (a) clients receive system.reconnect, (b) buffered messages flush after reconnect, (c) the new server gets the reconnects within ~2 seconds, (d) total user-facing disruption is < 1 second.