Recovery After Restart

"Restart recovery is a state transition with evidence, not a cleanup shortcut."

Recover from durable rows

A process can die while a job says running. Current Inkwell runs one local worker lane. On that process's startup, every persisted running job is an orphan from the previous process.

Record the orphaned attempt

Startup marks any running attempt failed, creates a legacy failed attempt only when none exists, then marks each running job failed with an error and finished_at. A separate retry moves that job to queued.

Make startup recovery repeat safely

Leaving running rows untouched strands a single-worker queue. Repeating recovery is safe because only running rows qualify; after the first pass there are none.

Operational invariant. Restart recovery is a state transition with evidence, not a cleanup shortcut.

Put crashes on the timeline

Test “Recovery After Restart” on a timeline where the process may disappear before acceptance, after commit, after claim, after the external call, or after output storage. At every cut, state what the database knows. A repeated request must not duplicate cost or prose.

Design it

Specify startup recovery for a dead worker for the single worker that died. Name statuses, timestamps, errors, and attempt rows. Explain which evidence authorizes recovery instead of letting a cleanup script guess.

Observability is recoverability

An operator should distinguish queued, running, failed, and done with one query and know the next safe action. A silent unknown called a queue has already lost the work.

Code

Recover only expired claims·sql

UPDATE job_attempts
SET status = 'failed',
    error = 'worker restarted',
    finished_at = CURRENT_TIMESTAMP
WHERE status = 'running'
  AND job_id IN (SELECT id FROM jobs WHERE status = 'running');

INSERT INTO job_attempts(
  job_id, attempt_number, status, error, started_at, finished_at
)
SELECT j.id, 1, 'failed', 'worker restarted',
       j.started_at, CURRENT_TIMESTAMP
FROM jobs AS j
WHERE j.status = 'running'
  AND NOT EXISTS (
    SELECT 1 FROM job_attempts AS a WHERE a.job_id = j.id
  );

UPDATE jobs
SET status = 'failed',
    error = 'worker restarted',
    finished_at = CURRENT_TIMESTAMP
WHERE status = 'running';

Recover from durable rows

Record the orphaned attempt

Make startup recovery repeat safely

Put crashes on the timeline

Design it

Observability is recoverability

Code

External links

Exercise

Progress

Comments 0