Skip to content
C.W.K.
Stream
Lesson 04 of 05 · published

Accept in One Transaction

~17 min · transactions, sqlite, atomicity, outbox

Level 0Loose Note
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

The One Place Work Changes Hands

Everything so far left the todo open: the row, the launch, the discovery, the enrollment, the invitation. Engine.accept is the one place a managed assignment's work actually changes hands, and it does it inside one engine command. One BEGIN IMMEDIATE, then one commit or one rollback.

Before the Transaction Opens

The adapter calls accept only for a completed final that carries COOP_ACCEPT with the right nonce. Even then, accept refuses before opening anything unless the receipt names the exact native message that said yes: "A correlated native receipt is required." A yes with no message id is a rumor.

Inside: The Steps, in Order

  1. The session. It must be at the generation the adapter holds and not exited: "Stale or exited session generation."
  2. The dispatch. Its session, generation, nonce and status must equal this session, this generation, the nonce that was answered, and invited. Cancelled, expired, already accepted, or someone else's invitation: one refusal covers them all, "Invitation is stale or belongs to another receiver."
  3. Each item, one after another:
    • still open at the revision the dispatch recorded, or "Work changed before acceptance";
    • for a todo whose invitation carried a reply review, acknowledge exactly those reply ids, as the session's actor;
    • take it with the domain's own take operation, as the session's actor;
    • insert the binding: item, session, generation, dispatch;
    • insert the outcome obligation, due in two hours (OUTCOME_REVIEW_SECONDS);
    • record the binding's work_revision, the item's revision after the take.
  4. The dispatch becomes accepted, with the receipt stored on it.
  5. The work envelope is built (dispatch id, bindings, write intents, shared_checkout, the work itself, and the scope when there is one) and enqueued as a work row in the outbox, still inside the same transaction.

Notice the items are interleaved: item one can be taken before item two's revision check fails. That is fine, because nothing commits until the last step succeeds. When item two fails, the rollback takes item one's take, binding and obligation with it.

Why It Has to Be One

The architecture states the rule as step four of dispatch: validate every selected row and revision, bind the attempt, take exactly that selection, append the events and the outcome obligation, all in one transaction. "Any conflict claims none of the selection." Split it and look at each crash point. A take committed without its binding is owned work no session is bound to. A binding without its obligation is owned work nobody is owed an outcome for, which is the original failure this whole track exists to prevent. A work envelope sent before the commit lets a worker start on work the ledger may still refuse. In one transaction, a crash anywhere leaves all of it or none of it.

The envelope deserves its own note. It is not sent from inside accept; it is written there, as an outbox row the adapter delivers after the commit. If the commit fails, the envelope never existed. That is the transactional outbox pattern, and it is why the invitation promised "The engine will commit the claim before sending the task."

Old Helpers, Borrowed Transaction

The take itself is Coop's original domain operation, the same take_todo a CLI todo take calls: one guarded UPDATE verified by its row count, never read-then-write. It runs no transaction of its own. Outside the engine, connect() opens a connection in autocommit mode, so its update and its event row each commit as they run, and it closes that connection when it is done. Inside the engine, connect() hands it a borrowed connection instead: the engine's own, already inside the accept's BEGIN IMMEDIATE, so the take lands in that one transaction and its close() does nothing. The same wrapper covers the helpers that do run their own transaction, such as take_scope: their BEGIN IMMEDIATE becomes a harmless SELECT 1, their commit does nothing, and a rollback makes the engine refuse the whole command. The context module's docstring says it in one line: "Legacy helpers may close/commit; the enclosing engine owns the transaction." That is how a helper written for standalone use joins an atomic accept without being rewritten.

The Store Backs It Up

Two unique indexes make the important invariants true even if code were wrong: engine_active_binding allows at most one active binding per work item, and engine_binding_obligation allows one obligation of each kind per binding. And the command key, accept: plus the invitation's outbox id, makes the whole thing idempotent: an adapter that crashes after the commit and asks again gets the stored result back, not a second take.

A launch, a socket write, a queue receipt, a worker's sentence: none of them takes a record. On the assignment path only this transaction does, and it takes everything it was offered or nothing at all.

Code

Verbatim from engine.py: Engine.accept·python
def accept(self,key,dispatch_id,session_id,generation,nonce,receipt,metadata):
    if not receipt or not receipt.get('native_message_id'):
        raise ValueError('A correlated native receipt is required')
    def run(conn):
        session=self._session(conn,session_id,generation)
        dispatch=self.row(conn,'engine_dispatches',dispatch_id)
        if (dispatch['session_id'],dispatch['generation'],dispatch['nonce'],dispatch['status']) != (session_id,generation,nonce,'invited'):
            raise Conflict('Invitation is stale or belongs to another receiver')
        bindings=[]
        for item in json.loads(dispatch['items']):
            table=KINDS[item['kind']]
            row=self.row(conn,table,item['id'])
            if row['status'] != 'open' or self.revision(conn,table,item['id']) != item['revision']:
                raise Conflict('Work changed before acceptance')
            if item['kind'] == 'todo':
                reviewed = next((r for r in json.loads(dispatch.get('reply_review') or '[]') if r['id']==item['id']), None)
                if reviewed:
                    domain.OPERATIONS['ack_todo_replies'](item['id'], [m['id'] for m in reviewed['messages']], who=session['actor'])
            fn=domain.OPERATIONS['take_'+item['kind']]
            fn(item['id'],who=session['actor'])
            bid=uid()
            conn.execute('INSERT INTO engine_bindings(id,kind,item_id,session_id,generation,dispatch_id,created) VALUES (?,?,?,?,?,?,?)',(bid,item['kind'],item['id'],session_id,generation,dispatch_id,time.time()))
            conn.execute('INSERT INTO engine_obligations(id,binding_id,kind,due,created) VALUES (?,?,?,?,?)',(uid(),bid,'outcome',time.time()+OUTCOME_REVIEW_SECONDS,time.time()))
            conn.execute('UPDATE engine_bindings SET work_revision=? WHERE id=?',(self.revision(conn,table,item['id']),bid))
            bindings.append(self.row(conn,'engine_bindings',bid))
        conn.execute("UPDATE engine_dispatches SET status='accepted',receipt=? WHERE id=?",(dump(receipt),dispatch_id))
        payload={'dispatch_id':dispatch_id,'bindings':bindings,'write_intents':json.loads(dispatch['intents']),
                 'shared_checkout':self.shared_checkout(conn,dispatch),
                 'work':[domain.OPERATIONS['get_todo'](i['id']) if i['kind']=='todo' else self.row(conn,KINDS[i['kind']],i['id']) for i in json.loads(dispatch['items'])]}
        if dispatch['scope']:
            payload['scope']=json.loads(dispatch['scope'])
        self._enqueue(conn,session_id,generation,'work',payload)
        return payload
    return self.command(key,'accept',[dispatch_id,session_id,generation,nonce,receipt],{**metadata,'session':session_id},run)
A minimal sketch you can run: take, bind and obligate atomically, and roll back on a revision mismatch·python
# A minimal sketch, not Coop's code: accept takes, binds and obligates in one
# SQLite transaction, and any conflict claims nothing at all.
import json
import sqlite3
import time
import uuid

OUTCOME_REVIEW_SECONDS = 2 * 3600

db = sqlite3.connect(':memory:', isolation_level=None)  # we issue BEGIN ourselves
db.executescript('''
CREATE TABLE todos(id INTEGER PRIMARY KEY, status TEXT NOT NULL, taken_by TEXT,
                   revision INTEGER NOT NULL DEFAULT 1);
CREATE TABLE dispatches(id TEXT PRIMARY KEY, session TEXT NOT NULL, nonce TEXT NOT NULL,
                        status TEXT NOT NULL DEFAULT 'invited', items TEXT NOT NULL);
CREATE TABLE bindings(id TEXT PRIMARY KEY, todo_id INTEGER NOT NULL, session TEXT NOT NULL,
                      state TEXT NOT NULL DEFAULT 'active');
CREATE UNIQUE INDEX one_owner ON bindings(todo_id) WHERE state='active';
CREATE TABLE obligations(id TEXT PRIMARY KEY, binding_id TEXT NOT NULL,
                         kind TEXT NOT NULL, due REAL NOT NULL);
''')


class Conflict(Exception):
    pass


def prepare(session, todo_ids):
    """Invite a session: record expected revisions and a nonce. Nothing is taken."""
    items = [[i, db.execute('SELECT revision FROM todos WHERE id=?', (i,)).fetchone()[0]]
             for i in todo_ids]
    did, nonce = str(uuid.uuid4()), str(uuid.uuid4())
    db.execute('INSERT INTO dispatches(id, session, nonce, items) VALUES (?,?,?,?)',
               (did, session, nonce, json.dumps(items)))
    return did, nonce


def accept(dispatch_id, session, nonce):
    db.execute('BEGIN IMMEDIATE')  # take the write lock before the first check
    try:
        row = db.execute('SELECT session, nonce, status, items FROM dispatches WHERE id=?',
                         (dispatch_id,)).fetchone()
        if row is None or row[:3] != (session, nonce, 'invited'):
            raise Conflict('Invitation is stale or belongs to another receiver')
        for todo_id, expected in json.loads(row[3]):
            status, revision = db.execute('SELECT status, revision FROM todos WHERE id=?',
                                          (todo_id,)).fetchone()
            if status != 'open' or revision != expected:
                raise Conflict(f'Work changed before acceptance (todo {todo_id})')
            db.execute("UPDATE todos SET status='taken', taken_by=?, revision=revision+1 WHERE id=?",
                       (session, todo_id))
            bid = str(uuid.uuid4())
            db.execute('INSERT INTO bindings(id, todo_id, session) VALUES (?,?,?)',
                       (bid, todo_id, session))
            db.execute("INSERT INTO obligations VALUES (?,?,'outcome',?)",
                       (str(uuid.uuid4()), bid, time.time() + OUTCOME_REVIEW_SECONDS))
        db.execute("UPDATE dispatches SET status='accepted' WHERE id=?", (dispatch_id,))
        db.execute('COMMIT')
    except BaseException:
        db.execute('ROLLBACK')  # todo 1's take, binding and obligation vanish too
        raise


def ledger():
    todos = db.execute('SELECT id, status, taken_by FROM todos ORDER BY id').fetchall()
    bindings = db.execute('SELECT count(*) FROM bindings').fetchone()[0]
    owed = db.execute('SELECT count(*) FROM obligations').fetchone()[0]
    return f'{todos}  bindings={bindings}  obligations={owed}'


db.executemany('INSERT INTO todos(id, status) VALUES (?, ?)', [(1, 'open'), (2, 'open')])
did, nonce = prepare('session-A', [1, 2])

# Dad edits todo 2 after the invitation went out: its revision moves on.
db.execute('UPDATE todos SET revision=revision+1 WHERE id=2')
try:
    accept(did, 'session-A', nonce)
except Conflict as exc:
    print('refused:', exc)
print('after refusal:', ledger())

# Retire that invitation and send a fresh one at the current revisions.
db.execute("UPDATE dispatches SET status='cancelled' WHERE id=?", (did,))
did, nonce = prepare('session-A', [1, 2])
try:
    accept(did, 'session-B', nonce)   # the wrong receiver presents the nonce
except Conflict as exc:
    print('refused:', exc)
accept(did, 'session-A', nonce)
print('after accept: ', ledger())
# refused: Work changed before acceptance (todo 2)
# after refusal: [(1, 'open', None), (2, 'open', None)]  bindings=0  obligations=0
# refused: Invitation is stale or belongs to another receiver
# after accept:  [(1, 'taken', 'session-A'), (2, 'taken', 'session-A')]  bindings=2  obligations=2

External links

Exercise

Run the sketch, then break it on purpose. First make it commit after each todo (COMMIT, then BEGIN IMMEDIATE again) instead of once at the end, and rerun: count the bindings and obligations left behind by the refused accept. Then find a multi-step state change in your own project (claim plus assign plus schedule a reminder, say) and wrap it the way the sketch does. Inject an exception between two of its steps and confirm nothing partial survives.
Hint
With a per-item commit, todo 1 stays taken with a binding and an obligation even though the dispatch was refused, which is exactly the half-claimed state Coop's accept can never produce. In your own code, look for helpers that commit internally; they are the ones that need a borrowed connection or a refactor before they can join a larger transaction.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.