A batch is a list of intents
When generate() returns, nothing has been sent to Ember. The batch and its tasks exist as rows in SQLite, every task queued, and the HTTP answer is a 202: the work has been accepted and is still to be done. A background worker inside the same engine process turns those intents into Ember jobs, one state at a time, and writes each state down before it acts on it. That habit is the whole design: if the process dies at any line, the row says how far it got.
| State | What is true |
|---|---|
queued | Planned, nothing sent. |
preparing | References are being read and checked. Still nothing sent. |
submitting | The request to Ember is in flight. Its outcome is unknown until Ember answers. |
running | Ember accepted it and gave a job id, recorded on the task. |
copying | Ember finished; its answer is saved on the task and the original is being copied in. |
done | The original and its history are in Avatars. |
The table follows an image task. A clip whose submit Ember accepted before its answer was lost goes back through queued and preparing under the same key, so for a clip those two rows promise only that nothing new will be paid for (the Clips Are Paid track).
Off the main line sit failed, cancelled, and three states that need a person or a reconciliation: uncertain, history_missing and copy_failed. The next two lessons are about those. The worker module names the two groups that drive it: ACTIVE (queued, running, copying) is what it picks up, and RECONCILABLE is what it must never resubmit on its own.
One owner per database
Two workers on one database would each pick up the same queued task and pay for it twice. So the first thing start() does is take an exclusive, non-blocking flock on worker.lock in the data directory. If another process holds it, the second worker refuses to start with Another Avatars worker already owns this database, and the engine's startup fails rather than running beside it.
A lock taken with flock belongs to the open file. When the process exits, however it exits, the kernel closes the file and drops the lock. There is no PID file to go stale after a crash and no cleanup step for anyone to forget. The lock's job is narrow: one live worker per database, and a dead one never blocks the next.
Waking up after a death
With the lock held, start() looks at what the last process left behind, and treats each state by what it can prove:
- preparing goes back to
queued. For an image nothing left the building, so starting over is free. A clip there may already have reached Ember once, and its unchanged key makes starting over free as well. - submitting becomes
uncertain, with the phase Reconcile before retry and the sentence The previous submit may have reached Ember. Check history before submitting again. The POST may or may not have arrived, and the worker refuses to guess. A clip is the one exception: its request carries an idempotency key, so resending it is safe (the Clips Are Paid track). - running needs no special case. The task has a job id, so the pump picks it up and simply resumes polling that job.
- copying with Ember's answer already saved goes straight to the copy again, with no new generation.
Shutting down is not cancelling. stop() cancels the worker's own polling and releases the lock, and leaves every job already sent to Ember running. The comment on that line says it plainly: never cancel authorized upstream work. The next start picks the jobs up by id.
The pump
The pump wakes once a second, clears finished coroutines, lists the active tasks and starts new ones until it holds workers of them at once, four by default. That once-a-second query runs over an index on the task state, which the store adds with a comment: the poll must not grow with the history of every finished task.
The transitions that can race a person carry the store's optimistic version check. Store.update compares versions only when the caller passes expected. The worker passes it on its first two moves, queued to preparing and preparing to submitting, and cancel passes it when it stops a queued or preparing task. Later writes, from running on, go through without a version. That pair is what makes cancelling safe before anything is sent. A test starts a task, holds it inside reference preparation, cancels it, and lets it continue. The worker's next write carries the old version, the store raises a conflict, the worker stops, and the task ends cancelled with nothing submitted.
Into Ember's job door
In preparing, the worker reads each frozen reference from disk and hashes it again; a mismatch stops the task with A frozen reference failed its checksum. It builds the payload with attribution in source_metadata: source_app is cwkAvatars, purpose is exploration, plus the batch id, the task id, the attempt number, the subjects, the frame and the candidate number. Then it writes submitting with a timestamp, and only then posts.
The post goes to POST /api/v1/api-generators/jobs, Ember's asynchronous door. Ember answers 202 with a job id at once, and the worker records running, the job id and a new entry in attempts in its very next write. Polling then goes through the Sibling Kit's shared Ember-jobs client: every 3 seconds, for up to 1800 seconds, a bound its own comment calls a transport limit, since a Pro job at 4K can take minutes.
The image-gen skill's diagram draws a different arrow, from Avatars to /api/v1/api-generators/generate. Ember has that route too, and it is synchronous: it answers 201 with the finished image. Avatars does not call it. A minutes-long synchronous call would leave the only copy of the answer on one open connection, and if that connection dropped there would be no job id to come back to. The code is the authority here, and the code posts to /jobs.