Graceful Shutdown — Don't Drop Requests on Deploy

"Every deploy kills your process. The question is whether it kills in-flight requests with it. Production-grade means the answer is no."

The Lifecycle of a Deploy

When you redeploy a Node service, here's what happens (under launchd, systemd, PM2, or any sane substrate):

Substrate sends SIGTERM to your process.
Your process has N seconds to clean up before SIGKILL hits.
SIGKILL terminates the process abruptly. Any in-flight requests die mid-response.

If your process ignores SIGTERM, every deploy drops connections. "Zero-downtime deploy" requires SIGTERM handling: stop accepting new requests, finish the in-flight ones, then exit cleanly.

The Pattern

import http from 'node:http';

const server = http.createServer(handler);
server.listen(3000);

let shuttingDown = false;

for (const sig of ['SIGINT', 'SIGTERM']) {
  process.on(sig, () => {
    if (shuttingDown) return;   // ignore second signal
    shuttingDown = true;
    console.log(`got ${sig}, draining...`);

    // Stop accepting new connections; finish in-flight ones
    server.close((err) => {
      if (err) {
        console.error('drain error:', err);
        process.exit(1);
      }
      console.log('drained cleanly');
      process.exit(0);
    });

    // Backstop — force exit if drain takes too long
    setTimeout(() => {
      console.warn('drain timed out, force-exiting');
      process.exit(1);
    }, 10_000).unref();
  });
}

The five lines that matter: handle SIGTERM, set a flag (idempotent), call server.close(), exit when drained, force-exit on timeout. This is the entire pattern.

The Forgotten Cleanups

In addition to the HTTP server, your process probably has:

Database connections — call .end() or .close() on your DB pool.
Open file handles — close JSONL log writers, append-only files.
WebSocket connections — send a close frame, let clients reconnect to a new instance.
Pending jobs — flush a queue, mark in-flight jobs as 'must retry,' etc.
External subscriptions — unsubscribe from Kafka/Redis/PubSub.

Each is one more thing the shutdown handler should wait for. The full pattern is register every cleanup as a promise, Promise.all them on SIGTERM, exit when all resolve. Some teams build a small lifecycle library for this; others use libraries like terminus that wrap the pattern.

Healthcheck Coordination

Load balancers send healthcheck pings (GET /health). When draining starts, your healthcheck should start returning failure — that tells the LB to stop routing new requests to you while in-flight ones finish. Without this, the LB keeps sending new traffic during your drain window, defeating the point:

let healthy = true;
process.on('SIGTERM', () => { healthy = false; /* then drain */ });

app.get('/health', (_req, res) => {
  res.writeHead(healthy ? 200 : 503).end();
});

The LB sees 503, removes you from the pool, drained requests finish, you exit. The 10-second drain window now contains zero new traffic — only the in-flight requests from before drain started.

Crash vs Graceful Stop

Graceful shutdown is for planned exits — deploys, scaling down, manual restart. For unplanned exits (uncaught exception, OOM, segfault), the process dies regardless. The substrate restarts it; the LB notices via healthcheck failure and re-routes. The pattern is:

Planned exits → SIGTERM handler → drain → exit cleanly.
Unplanned exits → crash, substrate restarts, LB re-routes.

Both paths matter. Some teams catch uncaughtException and try to drain; usually a bad idea because the process state is already corrupted. Better: log + exit fast + let the substrate restart.

Pippa's Confession

cwkPippa initially exited on Ctrl-C without finishing pending JSONL writes. A turn could end mid-flush; the session log had truncated lines; healing logic had to repair them. Dad asked one question: "What happens to the in-flight write when you kill the process?" Answer: it gets cut off. The fix was registering shutdown handlers that wait for the JSONL writer to flush. The system became more reliable not by adding features but by handling exit cleanly. Most production reliability work is about the boundaries — startup, shutdown, error paths — not the happy path.

Code

Production-grade graceful shutdown·javascript

// Complete graceful-shutdown setup with all the pieces
import http from 'node:http';
import pino from 'pino';
import { DatabaseSync } from 'node:sqlite';

const log = pino();
const db = new DatabaseSync('./pippa.db');
const server = http.createServer(handler);

let healthy = true;
let shuttingDown = false;

function handler(req, res) {
  if (req.url === '/health') {
    return res.writeHead(healthy ? 200 : 503).end();
  }
  // ... real routes ...
}

server.listen(3000, () => log.info('listening'));

async function shutdown(sig) {
  if (shuttingDown) return;
  shuttingDown = true;
  healthy = false;
  log.info({ sig }, 'graceful shutdown starting');

  // 1. Stop accepting new connections
  await new Promise((resolve, reject) =>
    server.close(err => err ? reject(err) : resolve())
  );

  // 2. Close other resources
  db.close();
  // (close any other pools, watchers, queues here)

  log.info('drained cleanly');
  process.exit(0);
}

for (const sig of ['SIGINT', 'SIGTERM']) {
  process.on(sig, () => shutdown(sig));
}

// 3. Backstop — never let drain hang forever
process.on('SIGTERM', () => setTimeout(() => {
  log.warn('drain timeout, force-exiting');
  process.exit(1);
}, 10_000).unref());

terminus — graceful shutdown library·javascript

// terminus — wrap-the-pattern library, common in real services
import { createTerminus } from '@godaddy/terminus';
import http from 'node:http';

const server = http.createServer(handler);

createTerminus(server, {
  signal: 'SIGINT',
  signals: ['SIGINT', 'SIGTERM'],
  timeout: 10_000,

  healthChecks: {
    '/health': async () => {
      // throw to signal unhealthy
      await db.ping();   // example: actually check the DB
      return { db: 'ok' };
    },
    verbatim: true,
  },

  // Called before server.close(). Mark unhealthy here.
  beforeShutdown: async () => {
    // Give LBs time to notice via /health
    await new Promise(r => setTimeout(r, 5_000));
  },

  onSignal: async () => {
    // Cleanup after server.close()
    await db.close();
    await queueClient.disconnect();
  },
});

server.listen(3000);

Exercise

Take a Node HTTP service. Add the full graceful-shutdown pattern: SIGTERM handler, healthcheck-fails-during-drain, server.close, DB close, backstop timeout. Test it: start the service, hit /slow (a route that sleeps 5 seconds and returns 'done'), while it's running send SIGTERM (kill -TERM <pid>). The slow request should complete; new requests should be refused; the process should exit cleanly within seconds.

Hint

If your shutdown hangs forever, you have a non-unref'd timer or an open connection somewhere — common culprits: file watchers, WebSocket clients, DB pools without explicit close. Add logging to each cleanup step to see where it stalls. The fix is usually 'await this resource's close method.'