@ultimat3/jobs 9.0.0 → 11.0.0

package/CLAUDE.md

@@ -194,12 +194,38 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 - The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`) and `task.enqueue()`'s
   is the job's plain key. Deliberate: the first stops two schedulers double-firing a tick, the
   second is a manual run with no occurrence to scope to.
-- **One shutdown hook per worker, and `stop()` is what hands it back.** `start()` keeps the
-  unregister `onShutdown` returns; the teardown releases it in a `finally`, so a close that threw
-  still gives it up. Discarding it was a hook per `start()` — the `start()` guard reads a
-  standstill, so start -> stop -> start stacked a second registration retaining a stopped
-  worker's driver, and the next process-wide drain ran all of them. `start()` refuses while
-  draining for the same reason: a claim loop back on a driver the drain is about to close.
+- **TWO shutdown hooks per worker — one per PHASE — and `stop()` is what hands both back**
+  (`As of 2026-08-23`). `accept` is `stopAccepting()`: flip the state, clear the poll timer, return.
+  `close` is the teardown: wait out the rounds and the in-flight jobs, close the driver.
+
+  It was ONE hook, at `accept`, doing all of it — so a single 10-minute job spent the entire drain
+  deadline inside the phase whose whole purpose is to be over immediately, and every hook behind it
+  was invoked with **0 ms** left: `@ultimat3/http`'s "stop listening" and `listenSyncNode`'s
+  `stopAccepting` are both `accept` hooks, so the pod went on serving requests and upgrading
+  websockets for the whole of the drain the load balancer had already been told about. Reproduced —
+  4 hooks started, none finished.
+
+  `start()` keeps both unregisters; the teardown releases them in a `finally`, so a close that threw
+  still gives them up. Discarding them was a hook per `start()` — the `start()` guard reads a
+  standstill, so start -> stop -> start stacked a second registration retaining a stopped worker's
+  driver, and the next process-wide drain ran all of them. `start()` refuses while draining for the
+  same reason: a claim loop back on a driver the drain is about to close.
+- **A claimed job is counted with core's `beginWork()`, so the DRAIN does the waiting**
+  (`As of 2026-08-23`). The wait for in-flight jobs belongs to the phase between `accept` and
+  `inflight`, which exists for exactly this and is where `@ultimat3/http` already puts a request —
+  not to a hook, where one role's work starves every other role's teardown. `/readyz`'s `inflight`
+  becomes truthful on a worker node as a consequence.
+- **The teardown's wait is BOUNDED on the SIGTERM path and unbounded on a manual `stop()`**
+  (`As of 2026-08-23`). The `close` hook is handed `ShutdownReason.deadlineAt` and passes it to
+  `settleAllBy` (`drain-wait.ts`); a manual `stop()` passes nothing and waits as long as its
+  jobs take, because a caller that asked has no budget to spend. Nothing in JS can kill a body that
+  ignores `ctx.signal`, so the unbounded version was a teardown that never returned: the driver was
+  never closed, `state` never left `'draining'`, and the memoised `stopping` promise every later
+  `stop()` joins never settled — `x dev`'s role rollback awaits that promise. Abandoning the wait
+  costs a lapsed lease and a job the queue delivers again, which is what at-least-once already
+  promises; `jobs.worker.drain-abandoned` names it, with the `configureLifecycle({ deadlineMs })`
+  raise as its fix. **A worker always REACHES `'stopped'`**, which is what makes `stop()`'s
+  `state === 'stopped'` early return an answer rather than a wedge.
 - **One teardown, joined.** `stop()` shares the in-flight teardown promise, so a SIGTERM landing
   on a manual stop waits out the same in-flight jobs instead of closing the driver underneath
   it. The promise is cleared as it settles, so a worker that started again tears down again
@@ -223,6 +249,13 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   a climbing `queue_depth` as the only symptoms. The `catch` releases the lease and rethrows; the
   claimed row goes back to the queue by its visibility timeout, as it does for any round that dies.
 
+  `fleetSlots.release(jobId)` is AWAITED in the settle's `.finally`, never `void`ed: the slot is a
+  DELETE in `x_job_leases`, so a fire-and-forget one was still on the wire when the teardown's
+  `allSettled` returned and `driver.close()` took the connection out from under it — the row then
+  held its slot for a full TTL, and a `concurrency: 1` job was unclaimable by the pod replacing this
+  one for a whole visibility window after every deploy. `release` swallows its own failures, so
+  awaiting it cannot turn a finished job into a rejected one.
+
   `LeaseStore.renew` answering `false` means the row is another holder's — two runs live under a cap
   of one — and it was discarded by `.catch(noop)`. It now stops the timer, logs
   `jobs.worker.slot-lost` and aborts the run through `X_JOB_SLOT_LOST`. Its own code, not
@@ -316,10 +349,14 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   waits that round out BEFORE `leader.release()` — a lock handed back mid-dispatch promotes a
   standby onto an occurrence this node is still enqueueing for, which is the double-fire leader
   election exists to prevent — and the round re-reads the drain state before each task, so "stop
-  dispatching" means this round too. Same hook rules as the worker: one `onShutdown` at the
-  `accept` phase while it runs, handed back in the teardown's `finally` so a `release()` that
-  threw still gives it up, `isLeader` cleared there too because a lock this process could not
-  hand back is never treated as still held.
+  dispatching" means this round too. Same hook rules as the worker, `As of 2026-08-23`: **TWO**
+  `onShutdown` registrations — `accept` stops dispatching and returns, `close` waits the round out
+  and releases the lease under the deadline it is handed (`settleAllBy`) — both handed back in the
+  teardown's `finally` so a `release()` that threw still gives them up, `isLeader` cleared there
+  too because a lock this process could not hand back is never treated as still held. **An
+  ABANDONED round does not release**: it is still enqueueing, so handing the lock over is that same
+  double-fire delivered by the shutdown, and a lease row expiring on its own is the safe end —
+  `jobs.scheduler.drain-abandoned` is the line that says so.
 - **`backfill()` is a FACTORY over `job()`, never a ninth primitive.** Same rule `llm()` follows
   in `@ultimat3/ai`: a new capability arrives as a factory over an existing primitive, so a
   backfill inherits `.enqueue()`, the retry policy, the cancellation, the dead-letter path and
@@ -363,6 +400,14 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   the CHANGELOG, `CLAUDE.md` and `x g backfill`'s generated source all say the handler must be
   idempotent. Never invert it: checkpointing first would report a page as swept that nobody wrote,
   and a lost page is unrecoverable where a repeated one is the handler's problem.
+- **A REPLAYED batch writes no ledger row** (`As of 2026-08-23`). `ledger.progress` sits outside
+  `step.run` — it has to, the ledger is not transactional with the steps — so a resumed pass
+  re-issued one `x_backfills` UPDATE per already-completed batch before it read a single new row:
+  4,800 statements on a 5M-row sweep killed at batch 4,800, on every attempt, inside the visibility
+  lease the heartbeat is renewing. The flag is set INSIDE the step body, because that is the only
+  thing that can tell a replay from a run — `step.run` answers the same shape either way. Nothing is
+  lost: the value is ABSOLUTE, so the first batch that does run reports every replayed one behind
+  it, and `finish` writes the total regardless.
 - **`x_backfills` is what has already been SWEPT, and the step checkpoints are where a pass
   resumes. Never the other way round.** The ledger row (`backfill-ledger.ts`) is a report an
   operator reads and the record that a completed name is done; the checkpoints are written in step
@@ -528,7 +573,11 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   branch. `jobs.lease.lost` at error plus `recordLeaseLost(queue)` is the one signal meaning the
   queue re-delivered a job this process was still running, so a false one is a page for a
   non-event, and the window widens exactly when the pool is slow. Re-read the flag AFTER every
-  await and inside the reporter, the way `settleWithin`'s `decided` does in core.
+  await and inside the reporter, the way `settleWithin`'s `decided` does in core. **The interval is
+  `unref`ed**, like all three of `sync-node.ts`'s: it is armed from inside a job run, so a drain
+  that ABANDONS the worker's hook leaves the run — and this timer — with nobody left to call
+  `stop()`, and a refed interval is then the one thing holding a drained process open until the
+  kubelet's SIGKILL.
 - **`stepTimeout` and `eventPoll` are DECLARED on the job, and `execute.ts` is the only place they
   are forwarded** (`As of 2026-08`). `StepRunnerOptions` carried both, `withStepTimeout`
   implemented the ceiling and `steps.test.ts` exercised it by building a runner BY HAND — while the
@@ -617,11 +666,18 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 - **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which REFUSES rather than answering
   a silent no-op.** An operator cancelling a 40M-row sweep has to know whether they stopped it or
   missed it, so a finished job is `X_JOB_NOT_CANCELLABLE` and a driver with no `cancel` is too.
-- **The scheduler asks `leader.acquire()` EVERY round, not only while it thinks it is not the
-  leader.** A lease-backed election expires, so `acquire()` is its renewal and a cached
-  `isLeader = true` would keep dispatching past a lease another node already took. `soleLeader`
-  answers true every time and `createPgLeader` holds its grant behind an internal flag, so the
-  extra call is a no-op for both — the flag also stops Postgres refcounting a second advisory
+- **The scheduler asks `leader.acquire()` EVERY round AND before EVERY task in it, not only while
+  it thinks it is not the leader.** A lease-backed election expires, so `acquire()` is its renewal
+  and a cached `isLeader = true` would keep dispatching past a lease another node already took.
+  Per-task as well as per-round `As of 2026-08-23`: a round walks its tasks serially with an enqueue
+  per job, so a 30s lease and a slow queue leave the tail of the walk running under a lease node B
+  already holds — and **the occurrence key does not absorb that**, because `SQL_ENQUEUE`'s conflict
+  target is the PARTIAL index over the live states, so a second dispatch landing after that
+  occurrence's job completed or dead-lettered inserts a new row and the handler runs twice. Argued
+  from the index definition, not reproduced — `stillLeading()` is the one place the question is
+  asked and `break` is the answer, the same shape the drain's `dispatching()` check already had.
+  `soleLeader` answers true every time and `createPgLeader` holds its grant behind an internal flag,
+  so the extra calls are a no-op for both — the flag also stops Postgres refcounting a second advisory
   grant that `release()`'s single unlock would never hand back.
 - **`createPgLeader` is correct only on a DEDICATED connection and boot has a pool.** A
   session-level `pg_try_advisory_lock` is released when its connection returns to the pool, so
@@ -707,6 +763,7 @@ picture from the other side.
 | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
 | `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
 | `worker.ts` | `worker` role, claim loop, drain |
+| `drain-wait.ts` | the drain's wait, shared by both roles: everything a teardown holds, settled — or abandoned at the budget the `close` hook was handed |
 | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
 | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |
 | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back. The claim loop asks "may I start this one?"; this answers it across the fleet |

package/README.md

@@ -475,6 +475,24 @@ debugging a stuck queue can read and run the exact statement.
 | `worker` | `createWorker({ driver, context, queues, concurrency })` | per-queue pools, lease heartbeat, SIGTERM drain: stop claiming → finish in-flight → close |
 | `scheduler` | `createScheduler({ driver, leader, state })` | one dispatch round at a time, catch-up policy, SIGTERM drain: stop dispatching → finish the round → release the lock |
 
+Both roles register the same **pair** of hooks and bound the `close` half the same way. The
+scheduler's abandoned case differs in one respect: a round still enqueueing keeps the lease rather
+than handing it back, because promoting a standby onto an occurrence this node is mid-dispatch for
+is the double-fire leader election exists to prevent. The lease row expires on its own.
+
+The worker's drain is **two shutdown hooks**, one per phase: `accept` stops claiming and returns
+immediately, `close` waits out the in-flight jobs and closes the driver. A running job is counted
+with core's `beginWork()`, so the wait for it happens in the drain's own in-flight phase rather than
+inside a hook — one hook doing all of it spends the whole `configureLifecycle({ deadlineMs })`
+budget in `accept`, where every other role's "stop taking work" is still queued behind it.
+
+**The `close` hook's wait is bounded by that same budget**, because nothing can kill a handler that
+ignores `ctx.signal`: at the deadline the worker logs `jobs.worker.drain-abandoned`, closes the
+driver and reaches `stopped`, and the job's lease lapses so the queue delivers it again — which is
+what at-least-once already promises. Raise the budget past your slowest job rather than relying on
+the wait: `configureLifecycle({ deadlineMs: 600_000 })`, with a `terminationGracePeriodSeconds` at
+least as large. A manual `await worker.stop()` has no budget and waits as long as its jobs take.
+
 `driver` and `context` are the two required keys on `WorkerOptions`; everything else defaults.
 `context: () => Ctx` supplies the ambient `Ctx` a job run executes under — the app wires its ALS
 and its tenant there, and a worker with no way to build one would run every handler as nobody.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "9.0.0",
+  "version": "11.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "9.0.0",
-    "@ultimat3/entity": "9.0.0",
-    "@ultimat3/schema": "9.0.0",
-    "@ultimat3/time": "9.0.0"
+    "@ultimat3/core": "11.0.0",
+    "@ultimat3/entity": "11.0.0",
+    "@ultimat3/schema": "11.0.0",
+    "@ultimat3/time": "11.0.0"
   }
 }

package/src/backfill-errors.ts

@@ -6,7 +6,6 @@
 // `backfill-pending.ts` and `backfill-registry.ts`.
 
 import { UltimateError } from '@ultimat3/core';
-import { docsFor } from './errors';
 
 /**
  * The seven backfill codes below all answer one question — "why is this sweep not running?" — and
@@ -31,7 +30,6 @@ export class BackfillPendingError extends UltimateError {
       code: 'X_BACKFILL_PENDING',
       cause: `backfill "${input.backfill}" is declared and x_backfills holds no completed pass for it in ${input.environment}`,
       fix: `x db backfill ${input.backfill} --write --json`,
-      docs: docsFor('X_BACKFILL_PENDING'),
     });
   }
 }
@@ -43,7 +41,6 @@ export class BackfillAppliedError extends UltimateError {
       code: 'X_BACKFILL_APPLIED',
       cause: `backfill "${input.backfill}" completed as run ${input.runId} at ${input.completedAt}; a forced rerun writes a NEW ledger row and never edits that one`,
       fix: `x db backfill ${input.backfill} --write --force --json`,
-      docs: docsFor('X_BACKFILL_APPLIED'),
     });
   }
 }
@@ -68,7 +65,6 @@ export class BackfillEnvironmentError extends UltimateError {
         target === undefined
           ? 'x db backfill --pending --json'
           : `ULTIMATE_ENV=${target} x db backfill ${input.backfill} --write --json`,
-      docs: docsFor('X_BACKFILL_ENVIRONMENT'),
     });
   }
 }
@@ -84,7 +80,6 @@ export class BackfillMigrationPendingError extends UltimateError {
       code: 'X_BACKFILL_MIGRATION_PENDING',
       cause: `backfill "${input.backfill}" requires migration ${input.migration}, which x_migrations does not record as applied`,
       fix: 'x db migrate --json',
-      docs: docsFor('X_BACKFILL_MIGRATION_PENDING'),
     });
   }
 }
@@ -100,7 +95,6 @@ export class BackfillRunningError extends UltimateError {
       code: 'X_BACKFILL_RUNNING',
       cause: `backfill "${input.backfill}" already has a live pass queued as ${input.jobId}, and one name holds one live pass; its step trace names the batch it is on, and a pass that is not advancing is a worker that lost its lease`,
       fix: `x jobs show ${input.jobId} --json`,
-      docs: docsFor('X_BACKFILL_RUNNING'),
     });
   }
 }
@@ -116,7 +110,6 @@ export class BackfillStalledError extends UltimateError {
       code: 'X_BACKFILL_STALLED',
       cause: `backfill "${input.backfill}" swept ${input.swept} rows, exhausted its source, and count() still matches ${input.remaining} — a WHERE the sweep narrows and the count does not is what leaves rows behind`,
       fix: `make count() select on exactly what source() selects on in backfill("${input.backfill}")`,
-      docs: docsFor('X_BACKFILL_STALLED'),
     });
   }
 }
@@ -131,7 +124,6 @@ export class BackfillUnknownError extends UltimateError {
           ? `no backfill named "${input.backfill}" is declared, and this app declares none at all`
           : `no backfill named "${input.backfill}" is declared (declared: ${input.known.join(', ')})`,
       fix: 'x db backfill --pending --json',
-      docs: docsFor('X_BACKFILL_UNKNOWN'),
     });
   }
 }

package/src/backfill-pass.ts

@@ -13,7 +13,7 @@
 // in step with the work and decides where a resumed pass restarts, while the `x_backfills` row is
 // a report an operator reads and the record that a completed name has already been swept.
 
-import { appVersion, assert, logger, resolveEnvironment } from '@ultimat3/core';
+import { appVersion, assert, logger, renderThrowable, resolveEnvironment } from '@ultimat3/core';
 import type { BatchIterator } from '@ultimat3/entity';
 import type { BackfillDefinition, BackfillInput, BackfillReport } from './backfill';
 import { BackfillStalledError } from './backfill-errors';
@@ -101,7 +101,7 @@ async function markFailed(
   } catch (error) {
     logger.warn('jobs.backfill.ledger-failed', {
       runId,
-      error: error instanceof Error ? error.message : String(error),
+      error: renderThrowable(error),
     });
   }
 }
@@ -223,6 +223,12 @@ export async function backfillPass<Row>(
       try {
         for (let index = 0; ; index += 1) {
           const stepName = `${STEP_PREFIX}${index}`;
+          /**
+           * Whether this batch's BODY ran, or whether the runner served its checkpoint from
+           * storage. Only the body can be asked: `step.run` answers the same shape either way,
+           * which is the whole point of a replay — so the flag is set inside it.
+           */
+          let swept = false;
           const checkpoint = asCheckpoint(
             await step.run(stepName, async (signal): Promise<Checkpoint> => {
               // INSIDE the body, which is the whole of it: a completed step is served from storage
@@ -237,6 +243,7 @@ export async function backfillPass<Row>(
               const next = await iteration.pull.next();
               if (next.done === true) return { cursor: null, rows: 0 };
               await definition.handle({ rows: next.value, ctx, signal, index });
+              swept = true;
               return { cursor: iteration.batches.cursor, rows: next.value.length };
             }),
             stepName,
@@ -248,8 +255,14 @@ export async function backfillPass<Row>(
           // whose last write is `finish` either way.
           if (checkpoint.rows > 0) {
             batches += 1;
-            // Absolute, so a replayed batch reports the position it reported the first time.
-            await ledger?.progress(runId, { rows, cursor });
+            // Only for a batch this attempt actually SWEPT. A replayed checkpoint runs no body and
+            // touches no row, so writing here re-issued one `x_backfills` UPDATE per completed
+            // batch before a resumed pass read a single new row — 4,800 statements on a 5M-row
+            // sweep killed at batch 4,800, on every attempt, inside the visibility lease the
+            // heartbeat is renewing. Nothing is lost by skipping them: the value is ABSOLUTE, so
+            // the first batch that does run reports every replayed one behind it, and `finish`
+            // writes the total whatever happens.
+            if (swept) await ledger?.progress(runId, { rows, cursor });
           }
           if (cursor === null) break;
         }

package/src/drain-wait.ts

@@ -0,0 +1,50 @@
+// The wait a role's teardown makes, under the drain's own budget — the worker's in-flight jobs,
+// the scheduler's dispatch round. One shape for both, because the race closes over nothing either
+// of them holds: the same seam `lifecycle-deadline.ts` takes in core, and for the same reason —
+// an unbounded wait inside a shutdown hook is a process the kubelet ends with SIGKILL.
+
+import { systemClock } from '@ultimat3/core';
+
+/**
+ * Everything in `pending`, settled — or abandoned once `deadlineAt` (real monotonic ms, the clock
+ * `ShutdownReason.deadlineAt` is measured on) has passed. Answers `true` when everything settled.
+ *
+ * `deadlineAt` is `undefined` for a MANUAL `stop()`, which waits as long as its work takes: a
+ * caller that asked a role to stop has no budget to spend, and closing the queue under a live job
+ * — or handing the lease back under a live dispatch — is exactly what draining exists to prevent. The bound belongs to the SIGTERM path, where the
+ * budget is real and a handler that ignores `ctx.signal` would otherwise hold the teardown — and
+ * with it the memoized `stopping` promise every later `stop()` joins — open forever.
+ *
+ * `allSettled`, so work that rejected is work that finished: each caller observes its own failures
+ * already, and a teardown that rethrew here would skip the close behind it.
+ */
+export async function settleAllBy(
+  pending: readonly Promise<unknown>[],
+  deadlineAt: number | undefined,
+): Promise<boolean> {
+  if (pending.length === 0) return true;
+  const settled = Promise.allSettled(pending);
+  if (deadlineAt === undefined) {
+    await settled;
+    return true;
+  }
+  const remainingMs = Math.max(0, deadlineAt - systemClock.monotonic());
+  return await new Promise<boolean>((resolve) => {
+    let decided = false;
+    const timer = setTimeout(() => {
+      if (decided) return;
+      decided = true;
+      resolve(false);
+    }, remainingMs);
+    // Never the thing keeping a drained process alive — the rule `lifecycle-deadline.ts` states
+    // for its own timer. A spent budget still gives the already-settled case its turn, because a
+    // resolved promise settles on a microtask and this timer on a macrotask.
+    timer.unref?.();
+    void settled.then(() => {
+      if (decided) return;
+      decided = true;
+      clearTimeout(timer);
+      resolve(true);
+    });
+  });
+}

package/src/driver-memory.ts

@@ -83,6 +83,22 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
     jobs.set(id, { ...existing, ...patch, updatedAt: nowMs(clock) });
   };
 
+  /**
+   * `update`, plus the lease columns `SQL_ACK`/`SQL_NACK` set to `null`.
+   *
+   * A settlement RELEASES the claim, and a `Partial<JobRecord>` cannot say so: both fields are
+   * optional, so `visibleAt: undefined` would keep the key and `{ ...existing }` keeps the value.
+   * Left stamped, a `done` row still named the worker that finished it and carried that attempt's
+   * lease deadline — which `x jobs show` prints, and which is the very pair the claim scan's
+   * lease-expiry branch reads to decide a row was abandoned.
+   */
+  const settle = (id: string, patch: Partial<JobRecord>): void => {
+    const existing = jobs.get(id);
+    if (existing === undefined) return;
+    const { visibleAt: _visibleAt, claimedBy: _claimedBy, ...released } = existing;
+    jobs.set(id, { ...released, ...patch, updatedAt: nowMs(clock) });
+  };
+
   const introspect: JobIntrospection = {
     job(jobId) {
       return Promise.resolve(jobs.get(jobId));
@@ -215,7 +231,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
     // would otherwise overwrite a row it no longer owns.
     ack(jobId: string): Promise<void> {
       if (jobs.get(jobId)?.state !== 'running') return Promise.resolve();
-      update(jobId, { state: 'done' });
+      settle(jobId, { state: 'done' });
       return Promise.resolve();
     },
 
@@ -241,7 +257,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
         attempt: counts ? record.attempt : Math.max(0, record.attempt - 1),
         ...(nackOptions.error === undefined ? {} : { lastError: nackOptions.error }),
       };
-      update(jobId, patch);
+      settle(jobId, patch);
       return Promise.resolve();
     },
 

package/src/errors.ts

@@ -93,8 +93,12 @@ registerErrorRetry({
   X_BACKFILL_APPLIED: 'terminal',
 });
 
-/** Shared with `backfill-errors.ts`, which holds the seven `X_BACKFILL_*` classes. */
-export const docsFor = (code: JobErrorCode): string => `https://ultimate.dev/errors/${code}`;
+// No `docs:` on any class below, here or in `backfill-errors.ts`. `UltimateError` fills it from
+// `describeErrorCode(code).docs`, which is `@ultimat3/core`'s `ERROR_DOCS_URL` — one page for every
+// code, never one per code, because `wiki/` is the framework's only public documentation surface
+// and a code lives there in a TABLE ROW, which has no anchor. The `docsFor` that stood here built
+// `https://ultimate.dev/errors/<code>`, which answered 404, host included, on every job failure
+// this package has ever put in a dead-letter row.
 
 /** An enqueue collided with a live job holding the same idempotency key under `onConflict: 'error'`. */
 export class JobDuplicateError extends UltimateError {
@@ -103,7 +107,6 @@ export class JobDuplicateError extends UltimateError {
       code: 'X_JOB_DUPLICATE',
       cause: `job "${input.job}" already queued as ${input.existingId} with idempotencyKey "${input.idempotencyKey}"`,
       fix: 'pass onConflict: "dedupe" to enqueue, or make idempotencyKey narrower',
-      docs: docsFor('X_JOB_DUPLICATE'),
     });
   }
 }
@@ -124,7 +127,6 @@ export class JobNameTakenError extends UltimateError {
       code: 'X_JOB_DUPLICATE',
       cause: `two ${input.kind}s claim the name "${input.name}"`,
       fix: `x jobs ls --json names the one already seated; rename the other's export, or its "name:" if it declares one — a ${input.kind} name is a durable queue key and is globally unique`,
-      docs: docsFor('X_JOB_DUPLICATE'),
     });
   }
 }
@@ -151,7 +153,6 @@ export class JobRowStatusUnknownError extends UltimateError {
         `${input.table}.${input.column} holds "${input.value}", which this build does not know — ` +
         `it reads ${input.known.join(', ')}`,
       fix: `x jobs show --json   # then drain the older workers: a status this build cannot read was almost certainly written by a newer deploy`,
-      docs: docsFor('X_JOB_ROW_STATUS_UNKNOWN'),
     });
   }
 }
@@ -175,7 +176,6 @@ export class ActionJobUnbridgedError extends UltimateError {
       code: 'X_ACTION_JOB_UNBRIDGED',
       cause: `export "${input.export}" is the action projection "${input.job}", which is not a job handle and cannot be registered as one`,
       fix: `wrap it: agentJob(${input.export}, { name: '${input.export}', tenant, retry }) from @ultimat3/ai — that composes job() and returns a handle the queue accepts`,
-      docs: docsFor('X_ACTION_JOB_UNBRIDGED'),
     });
   }
 }
@@ -187,7 +187,6 @@ export class StepDuplicateError extends UltimateError {
       code: 'X_STEP_DUPLICATE',
       cause: `job "${input.job}" used step name "${input.step}" twice in one run`,
       fix: `rename one of them, e.g. step.run('${input.step}-2', ...) — step names are the replay key`,
-      docs: docsFor('X_STEP_DUPLICATE'),
     });
   }
 }
@@ -201,7 +200,6 @@ export class JobTimeoutError extends UltimateError {
           ? `job "${input.job}" exceeded its ${input.timeoutMs}ms timeout`
           : `job "${input.job}" step "${input.step}" exceeded its ${input.timeoutMs}ms timeout`,
       fix: `raise timeout on the job definition, or split the work into step.run() calls`,
-      docs: docsFor('X_JOB_TIMEOUT'),
     });
   }
 }
@@ -225,7 +223,6 @@ export class JobAbortedError extends UltimateError {
           ? `job "${input.job}" was cancelled — this attempt no longer owns the run`
           : `job "${input.job}" was cancelled before step "${input.step}" could be recorded`,
       fix: 'add throwIfAborted(ctx) before expensive work, or pass fetch(url, { signal: ctx.signal }) — the queue re-runs the job, so stop at the deadline instead of running past it',
-      docs: docsFor('X_ABORTED'),
     });
   }
 }
@@ -237,7 +234,6 @@ export class JobMaxAttemptsError extends UltimateError {
       code: 'X_JOB_MAX_ATTEMPTS',
       cause: `job "${input.job}" failed ${input.attempts} times, last error: ${input.lastError}`,
       fix: `x jobs retry ${input.jobId}`,
-      docs: docsFor('X_JOB_MAX_ATTEMPTS'),
     });
   }
 }
@@ -248,7 +244,6 @@ export class DriverUnavailableError extends UltimateError {
       code: 'X_DRIVER_UNAVAILABLE',
       cause: `jobs driver "${input.driver}" is unavailable: ${input.cause}`,
       fix: input.fix,
-      docs: docsFor('X_DRIVER_UNAVAILABLE'),
     });
   }
 }
@@ -263,7 +258,6 @@ export class IdempotencyRequiredError extends UltimateError {
       code: 'X_IDEMPOTENCY_REQUIRED',
       cause: `job "${input.job}" has no idempotencyKey — at-least-once delivery would run it twice`,
       fix: `add idempotencyKey: (input) => \`${input.job}:\${input.id}\` to the job definition`,
-      docs: docsFor('X_IDEMPOTENCY_REQUIRED'),
     });
   }
 }
@@ -290,7 +284,6 @@ export class JobTenantRequiredError extends UltimateError {
       // tenant, and the pass opens the cross-tenant scope for exactly that declaration. Half the
       // callers of this code arrive through `backfill()`, which forwards its `tenant` to `job()`.
       fix: `add tenant: (input) => input.orgId to job("${input.job}") — or tenant: 'none', which declares NO org: right for a job that touches no tenant-scoped table, and the spelling a backfill() uses to sweep every tenant`,
-      docs: docsFor('X_JOB_TENANT_REQUIRED'),
     });
   }
 }
@@ -307,7 +300,6 @@ export class LeaseLostError extends UltimateError {
       code: 'X_JOB_LEASE_LOST',
       cause: `job "${input.job}" (${input.jobId}) is no longer claimed by this worker — it was cancelled, or its visibility lease lapsed and the queue re-delivered it`,
       fix: `x jobs show ${input.jobId} --json`,
-      docs: docsFor('X_JOB_LEASE_LOST'),
     });
   }
 }
@@ -325,7 +317,6 @@ export class JobSlotLostError extends UltimateError {
       code: 'X_JOB_SLOT_LOST',
       cause: `job "${input.job}" (${input.jobId}) no longer holds fleet concurrency slot ${input.slot} — its lease expired and another worker took it`,
       fix: `x jobs show ${input.jobId} --json`,
-      docs: docsFor('X_JOB_SLOT_LOST'),
     });
   }
 }
@@ -344,7 +335,6 @@ export class JobNotCancellableError extends UltimateError {
           ? `no job ${input.jobId} exists in this queue`
           : `job ${input.jobId} is "${input.state}" and only a job that has not finished can be cancelled`,
       fix: `x jobs ls --state running --json`,
-      docs: docsFor('X_JOB_NOT_CANCELLABLE'),
     });
   }
 }
@@ -356,7 +346,6 @@ export class CancelUnsupportedError extends UltimateError {
       code: 'X_JOB_NOT_CANCELLABLE',
       cause: `the "${input.driver}" jobs driver cannot cancel a single job`,
       fix: 'call setJobDriver(createPgDriver()) at boot — only the pg driver implements introspect.cancel — then: x jobs cancel <id> --json',
-      docs: docsFor('X_JOB_NOT_CANCELLABLE'),
     });
   }
 }
@@ -373,7 +362,6 @@ export class ConcurrencyUnenforceableError extends UltimateError {
       code: 'X_JOB_CONCURRENCY_UNENFORCEABLE',
       cause: `${input.jobs.join(', ')} declare concurrency and the "${input.driver}" jobs driver has no lease store, so the cap would hold per process and the fleet would run concurrency x replicas`,
       fix: `remove concurrency from job("${input.jobs[0] ?? 'the job'}"), or call setJobDriver(createPgDriver()) at boot — the pg driver is the one with a lease store`,
-      docs: docsFor('X_JOB_CONCURRENCY_UNENFORCEABLE'),
     });
   }
 }
@@ -385,7 +373,6 @@ export class OutboxNoTxError extends UltimateError {
       code: 'X_OUTBOX_NO_TX',
       cause: `ctx.jobs.enqueue(${input.job}) ran outside a transaction with outbox: 'required'`,
       fix: 'wrap the call in ctx.tx(async (tx) => ...), or enqueue with { outbox: false }',
-      docs: docsFor('X_OUTBOX_NO_TX'),
     });
   }
 }
@@ -396,7 +383,6 @@ export class JobsNotImplementedError extends UltimateError {
       code: 'X_NOT_IMPLEMENTED',
       cause: `${input.feature} is declared but not implemented in @ultimat3/jobs`,
       fix: input.fix,
-      docs: docsFor('X_NOT_IMPLEMENTED'),
     });
   }
 }

package/src/events-pg.ts

@@ -7,7 +7,7 @@
 // resumes at 12:00:30 must still see an event published at 12:00:10.
 
 import type { Clock } from '@ultimat3/core';
-import { logger, systemClock, uuid } from '@ultimat3/core';
+import { logger, renderThrowable, systemClock, uuid } from '@ultimat3/core';
 import type { DurationInput } from './clock';
 import { nowMs, toMs } from './clock';
 import type { PgExecutor } from './driver-pg';
@@ -54,7 +54,7 @@ export function createPgEventBus(options: PgEventBusOptions): EventBus {
     void exec.query(SQL_EVENT_PURGE, []).catch((error: unknown) => {
       // Housekeeping never costs a publish: an unpurged row is filtered out of every read.
       logger.warn('jobs.event.purge-failed', {
-        error: error instanceof Error ? error.message : String(error),
+        error: renderThrowable(error),
       });
     });
     return 0;

package/src/execute.ts

@@ -10,6 +10,7 @@ import {
   anonymousActor,
   isUltimateError,
   logger,
+  renderThrowable,
   reportError,
   runWithContext,
   useContext,
@@ -22,6 +23,7 @@ import { eventBus } from './events';
 import type { AnyJobHandle } from './job';
 import type { JobStopReason } from './retry-classification';
 import { nextRetryForError, recordedFailure } from './retry-classification';
+import { createRunSignal } from './run-signal';
 import type { EventLookup, StepRecord } from './steps';
 import { createStepRunner, isStepSuspension } from './steps';
 import { jobRunActor } from './tenant';
@@ -100,7 +102,14 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
   // where every other body does, with no jobs-only parameter to know about. Composed with the
   // caller's signal rather than replacing it: a ctx that was already going away still is.
   const cancel = new AbortController();
-  const signal = AbortSignal.any([callerSignal(options.ctx), cancel.signal]);
+  // `createRunSignal` and never `AbortSignal.any` — the second of the two sites this package's
+  // `CLAUDE.md` states the rule as absolute for. In the worker path `callerSignal` is per-run and
+  // nothing leaks; on `@ultimat3/testing`'s job-fixture path, which calls `executeJob` directly,
+  // the caller's `ctx.signal` may be process-lifetime, and a composite cannot be undone — so every
+  // job a fixture ran left a dependent signal on it for the life of the process. Disposed in the
+  // `finally` at the bottom of the try, beside `cancel.abort`.
+  const runSignal = createRunSignal([callerSignal(options.ctx), cancel.signal]);
+  const signal = runSignal.signal;
   const ctx: Ctx = Object.freeze({
     ...options.ctx,
     signal,
@@ -178,7 +187,7 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
       });
     }
 
-    const message = error instanceof Error ? error.message : String(error);
+    const message = renderThrowable(error);
     // The ERROR decides too, not only the attempt count. A `terminal` code — a rotated password,
     // a schema mismatch, a permission denial — fails identically on every remaining attempt, so
     // spending them is a queue slot, a provider bill and, at a site that locks an account after
@@ -238,6 +247,9 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
     // attempt already owns. The runner fences its writes on this signal. A second `abort()` keeps
     // the first reason, so a timed-out run still reports the timeout, not this.
     cancel.abort(new JobAbortedError({ job: handle.name }));
+    // AFTER the abort, so the runner's fence still sees it: `dispose` stops following the sources,
+    // it never aborts, and the signal keeps whatever state the abort above left it in.
+    runSignal.dispose();
   }
 
   // Only reachable when the BODY succeeded, and settlement is deliberately outside the catch
@@ -287,9 +299,7 @@ function raceTimeout(
         job,
         timeoutMs,
         ended,
-        ...(error === undefined
-          ? {}
-          : { error: error instanceof Error ? error.message : String(error) }),
+        ...(error === undefined ? {} : { error: renderThrowable(error) }),
       });
     };
     work.then(

package/src/heartbeat.ts

@@ -4,7 +4,7 @@
 // hardest bug in a queue to see from the outside and the easiest to name from in here.
 
 import type { Clock } from '@ultimat3/core';
-import { logger, recordLeaseLost } from '@ultimat3/core';
+import { logger, recordLeaseLost, renderThrowable } from '@ultimat3/core';
 import { nowMs } from './clock';
 import type { ClaimedJob, JobDriver } from './driver';
 import { LeaseLostError } from './errors';
@@ -76,9 +76,7 @@ export function startLeaseHeartbeat(options: LeaseHeartbeatOptions): LeaseHeartb
       attempt: claimed.attempt,
       visibilityTimeoutMs,
       reason: reason ?? 'expired',
-      ...(error === undefined
-        ? {}
-        : { error: error instanceof Error ? error.message : String(error) }),
+      ...(error === undefined ? {} : { error: renderThrowable(error) }),
     });
     recordLeaseLost(claimed.queue);
     // Cancel LAST, so the loss is logged and counted before the body it unwinds starts throwing.
@@ -134,7 +132,7 @@ export function startLeaseHeartbeat(options: LeaseHeartbeatOptions): LeaseHeartb
         job: claimed.name,
         jobId: claimed.id,
         attempt: claimed.attempt,
-        error: error instanceof Error ? error.message : String(error),
+        error: renderThrowable(error),
       });
       if (lapsed()) reportLost(error);
     } finally {

package/src/outbox.ts

@@ -24,7 +24,7 @@
 // test and `x dev` must enqueue with nothing wired — but it is a fallback, not the guarantee.
 
 import type { Clock } from '@ultimat3/core';
-import { currentSpanContext, logger, traceparent, uuid } from '@ultimat3/core';
+import { currentSpanContext, logger, renderThrowable, traceparent, uuid } from '@ultimat3/core';
 import type { Tx } from '@ultimat3/entity';
 import { nowMs } from './clock';
 import type { EnqueueResult, JobDriver } from './driver';
@@ -421,7 +421,7 @@ export function createOutboxRelay(options: RelayOptions): OutboxRelay {
           id: record.id,
           published,
           remaining: batch.length - published,
-          error: error instanceof Error ? error.message : String(error),
+          error: renderThrowable(error),
         });
         // Hand the rest of the batch back rather than sit on a claim nobody is publishing. The
         // claim is a lease now, so without this a single pool timeout parks every committed row
@@ -456,7 +456,7 @@ export function createOutboxRelay(options: RelayOptions): OutboxRelay {
           .then((): void => undefined)
           .catch((error: unknown) => {
             logger.error('jobs.outbox.tick-failed', {
-              error: error instanceof Error ? error.message : String(error),
+              error: renderThrowable(error),
             });
           })
           .finally(() => {

package/src/renewal-timer.ts

@@ -4,6 +4,8 @@
 // the interval, which does nothing to the request already on the wire. So a flag is what every
 // branch after an `await` re-reads — the shape `settleWithin`'s `decided` uses in core.
 
+import { logger, renderThrowable } from '@ultimat3/core';
+
 export interface RenewalTimer {
   /**
    * True once `stop()` has been called. Read AFTER every await in the renewal body: a clean
@@ -22,8 +24,28 @@ export function startRenewalTimer(
 ): RenewalTimer {
   let stopped = false;
   const timer = setInterval(() => {
-    void renew();
+    // `Promise.resolve().then(renew)` and never `void renew()`: a `renew` that throws SYNCHRONOUSLY
+    // escapes before any `.catch` its body chained exists. `worker-fleet-slots.ts` guards the
+    // promise chain and cannot guard this — `LeaseStore.renew` is an injected seam, and a store
+    // that throws on a closed pool throws on the call, not in the chain. Nothing sits above a
+    // `setInterval` callback, so that throw is an uncaught exception in the timer that was going
+    // to keep the lease alive. The shape `outbox.ts`'s tick loop already uses.
+    void Promise.resolve()
+      .then(renew)
+      .catch((error: unknown) => {
+        // A renewal that FAILS is each caller's own business and both handle it. Reaching here
+        // means the seam broke its contract, which is a different fact and is worth its own line —
+        // swallowed, it would be a lease that stops renewing with nothing anywhere saying so.
+        logger.error('jobs.renewal.raised', { error: renderThrowable(error) });
+      });
   }, intervalMs);
+  // Never the thing keeping a drained process alive. This interval is armed from inside a job run,
+  // so a drain that ABANDONS the worker's hook leaves the run — and this timer — with nobody left
+  // to call `stop()`: refed, it holds the event loop open past every phase of the shutdown, and
+  // the kubelet's SIGKILL becomes the exit. `sync-node.ts` unrefs all three of its timers and
+  // `lifecycle-deadline.ts` its own for the same reason. A renewal is bookkeeping for work that is
+  // already over by then; nothing is lost by letting the process go.
+  timer.unref?.();
   return {
     stopped: () => stopped,
     stop(): void {

package/src/scheduler.ts

@@ -6,16 +6,25 @@
 // advisory lock is held by the SESSION, not by this process — it outlives every transaction and is
 // released only by an explicit unlock, the pool's reset on release, or the connection dying, and
 // the next round may run on a different connection. So a node can neither renew it nor prove it
-// still holds one, and leadership passes to a second node while the first is still dispatching. Two schedulers
-// double-enqueue every task; the idempotency key would absorb it, but leader election means
-// the queue never sees the duplicate at all. One ROUND at a time is the same rule inside one
-// process: the loop re-arms on the round it just finished, and any other caller joins that
-// round rather than opening a second one over the same `lastFiredAt`.
+// still holds one, and leadership passes to a second node while the first is still dispatching.
+//
+// **The occurrence key is not a second line of defence, and the lease is therefore re-asserted
+// before EVERY task, not once per round.** `SQL_ENQUEUE`'s conflict target is the PARTIAL index
+// over the live states (`ready`, `delayed`, `running`, `suspended`), so a duplicate enqueue is
+// absorbed only while the first job is still one of those: a second dispatcher landing after that
+// occurrence's job completed, dead-lettered or was retried past its row inserts a NEW row, and the
+// handler runs twice. A round walks its tasks serially with an enqueue per job, so a 30s lease and
+// a slow queue leave the tail of the walk running under a lease another node already took. Argued
+// from the index definition, not reproduced — say it that way, as `outbox.ts` does for the same
+// mechanism. One ROUND at a time is the same rule inside one process: the loop re-arms on the
+// round it just finished, and any other caller joins that round rather than opening a second one
+// over the same `lastFiredAt`.
 
 import type { Clock } from '@ultimat3/core';
-import { isUltimateError, logger, onShutdown } from '@ultimat3/core';
+import { isUltimateError, logger, onShutdown, renderThrowable } from '@ultimat3/core';
 import { instant, nextCronOccurrence } from '@ultimat3/time';
 import { nowMs } from './clock';
+import { settleAllBy } from './drain-wait';
 import type { JobDriver } from './driver';
 import type { TaskHandle, TaskJobResult } from './task';
 import { registeredTasks } from './task';
@@ -26,7 +35,7 @@ import { registeredTasks } from './task';
  * stable code to search on and the `fix:` to run, not a sentence.
  */
 function failureFields(error: unknown): Record<string, unknown> {
-  const message = error instanceof Error ? error.message : String(error);
+  const message = renderThrowable(error);
   return isUltimateError(error)
     ? { error: message, code: error.code, cause: error.cause, fix: error.fix }
     : { error: message };
@@ -114,8 +123,11 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
   let state: 'idle' | 'running' | 'draining' | 'stopped' = 'idle';
   /** The dispatch round in flight — what a second caller joins and what `stop()` waits out. */
   let round: Promise<readonly DispatchedOccurrence[]> | undefined;
-  /** The `onShutdown` registration this scheduler holds while it runs. Handed back by `stop()`. */
-  let releaseShutdownHook: (() => void) | undefined;
+  /**
+   * The two `onShutdown` registrations this scheduler holds while it runs, both handed back by
+   * `stop()`. Two, for the phases they answer — the worker's rule, and `listenSyncNode`'s.
+   */
+  let releaseShutdownHooks: (() => void)[] = [];
   /** The teardown in flight, so a SIGTERM landing on a manual stop joins it. */
   let stopping: Promise<void> | undefined;
 
@@ -176,24 +188,31 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
   /** The drain's one question: may this scheduler still dispatch an occurrence? */
   const dispatching = (): boolean => state !== 'draining' && state !== 'stopped';
 
+  /**
+   * Asked EVERY round AND before every task in it, never only while `isLeader` is false. A
+   * lease-backed election (the one a pooled executor can use — `createPgLeaseLeader`) expires on a
+   * wall clock, so `acquire()` is also its renewal and a node that cached `isLeader = true` keeps
+   * dispatching past a lease another node has already taken. `soleLeader` answers true every time
+   * and `createPgLeader` holds its grant behind an internal flag, so the extra calls are a no-op
+   * for both.
+   */
+  const stillLeading = async (): Promise<boolean> => {
+    if (await leader.acquire()) {
+      isLeader = true;
+      return true;
+    }
+    // Demoted, or never elected. Nothing to release — a lease we no longer hold is not ours to
+    // hand back, and `teardown` reads this same flag before it calls `release()`.
+    if (isLeader) logger.warn('jobs.scheduler.leadership-lost', { at: nowMs(options.clock) });
+    isLeader = false;
+    return false;
+  };
+
   const runRound = async (): Promise<readonly DispatchedOccurrence[]> => {
     // Never take the lock a drain is on its way to releasing: a round that acquired it here
     // would still be enqueueing after `stop()` handed the occurrence to the next node.
     if (!dispatching()) return [];
-    // Asked EVERY round, not only while `isLeader` is false. A lease-backed election (the one a
-    // pooled executor can use — `createPgLeaseLeader`) expires, so `acquire()` is also its
-    // renewal, and a node that cached `isLeader = true` would keep dispatching past a lease
-    // another node has already taken. `soleLeader` answers true every time, and `createPgLeader`
-    // holds its grant internally, so this is a no-op for both.
-    const held = await leader.acquire();
-    if (!held) {
-      // Demoted, or never elected. Nothing to release — a lease we no longer hold is not ours to
-      // hand back, and `teardown` reads this same flag before it calls `release()`.
-      if (isLeader) logger.warn('jobs.scheduler.leadership-lost', { at: nowMs(options.clock) });
-      isLeader = false;
-      return [];
-    }
-    isLeader = true;
+    if (!(await stillLeading())) return [];
 
     const at = nowMs(options.clock);
     const tasks = options.tasks ?? registeredTasks();
@@ -204,6 +223,10 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // at the next round. A task not reached simply fires next time — its `lastFiredAt` is
       // untouched — while the occurrence this round already began is the one `stop()` waits for.
       if (!dispatching()) break;
+      // The lease, on the same rule and for the same reason the drain state is re-read: it expires
+      // on a wall clock in the middle of this walk, not between rounds. See the file header for
+      // what a second dispatcher costs — the occurrence key does NOT absorb it in general.
+      if (!(await stillLeading())) break;
       const last = await schedulerState.lastFiredAt(handle.name);
       if (last === undefined) {
         // First sight of this task: arm it, never fire retroactively for all of history.
@@ -275,10 +298,20 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
     }, tickIntervalMs);
   };
 
-  const teardown = async (reason: string): Promise<void> => {
+  /**
+   * The whole of the `accept` phase: stop dispatching, and nothing else. Synchronous on purpose —
+   * the hook behind this one is somebody else's "stop taking work", and a phase that waits is a
+   * phase that spends the budget those hooks were going to need.
+   */
+  const stopDispatching = (): void => {
+    if (state === 'stopped') return;
     state = 'draining';
     if (timer !== undefined) clearTimeout(timer);
     timer = undefined;
+  };
+
+  const teardown = async (reason: string, deadlineAt?: number): Promise<void> => {
+    stopDispatching();
     logger.info('jobs.scheduler.draining', { reason, dispatching: round !== undefined });
     try {
       // The round this stop races runs to the end first. Releasing the lease under a
@@ -286,8 +319,24 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // then own the same occurrence — the exact double-fire leader election exists to prevent.
       // Settled, not awaited: a round that failed is its own caller's to see, and the lease still
       // has to go back.
-      await Promise.allSettled([round]);
-      if (isLeader) await leader.release();
+      //
+      // Bounded on the SIGTERM path, `undefined` on a manual stop: a round parked in `enqueue` on
+      // a queue that is not answering cannot be cancelled from here, and an unbounded wait is a
+      // teardown that never ends — hooks never handed back, `state` never past 'draining', and the
+      // memoised `stopping` every later `stop()` joins never settling.
+      const dispatched = await settleAllBy(round === undefined ? [] : [round], deadlineAt);
+      // A round we ABANDONED is a round still enqueueing, so the lock is deliberately NOT handed
+      // back: promoting a standby onto an occurrence this process is mid-dispatch for is the
+      // double-fire above, delivered by the shutdown. A lease row expires on its own, which is
+      // what an expiry is for — and `isLeader` is cleared below either way.
+      if (!dispatched) {
+        logger.warn('jobs.scheduler.drain-abandoned', {
+          reason,
+          fix: 'raise the drain budget past a dispatch round — configureLifecycle({ deadlineMs: 60_000 }) — and set terminationGracePeriodSeconds to at least as many seconds',
+        });
+      } else if (isLeader) {
+        await leader.release();
+      }
     } finally {
       // Whatever the release did, this scheduler is done. `isLeader` false because a lock this
       // process no longer holds — or failed to hand back — must never be re-used as if it did,
@@ -295,17 +344,20 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // the next process-wide drain, and keeps this closure and its driver alive with it.
       isLeader = false;
       state = 'stopped';
-      releaseShutdownHook?.();
-      releaseShutdownHook = undefined;
+      for (const release of releaseShutdownHooks) release();
+      releaseShutdownHooks = [];
     }
   };
 
-  const stop = async (reason = 'stop'): Promise<void> => {
+  const stop = async (reason = 'stop', deadlineAt?: number): Promise<void> => {
+    // Answered immediately once this scheduler is done: the teardown always REACHES 'stopped', so
+    // a caller landing after an abandoned drain gets an answer rather than joining a promise that
+    // never settles.
     if (state === 'stopped') return;
     // One teardown, joined rather than repeated — the worker's rule, for the same reason: a
     // SIGTERM landing on a manual stop must wait out the same round, not release the lock a
     // second time behind it. Cleared as it settles, so a scheduler started again stops again.
-    stopping ??= teardown(reason).finally(() => {
+    stopping ??= teardown(reason, deadlineAt).finally(() => {
       stopping = undefined;
     });
     await stopping;
@@ -317,14 +369,19 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // about to release, and stack a second shutdown hook on the one still running.
       if (state !== 'idle' && state !== 'stopped') return;
       state = 'running';
-      // 'accept' phase: stop dispatching before core waits on in-flight work — an occurrence
-      // enqueued during the drain is work nothing in this process is left to run. The
-      // unregister is kept, never discarded: `stop()` hands it back, so start -> stop -> start
-      // holds ONE hook rather than one per start.
+      // TWO hooks, for the two phases. `accept` stops dispatching and returns — an occurrence
+      // enqueued during the drain is work nothing in this process is left to run, and every hook
+      // behind this one still has the whole budget. `close` waits the round out and hands the
+      // lease back, bounded by the deadline it is given. Both unregisters are kept, never
+      // discarded: `stop()` hands them back, so start -> stop -> start holds one pair rather
+      // than one per start.
       if (options.drainOnShutdown !== false) {
-        releaseShutdownHook = onShutdown('jobs.scheduler', () => stop('SIGTERM'), {
-          phase: 'accept',
-        });
+        releaseShutdownHooks = [
+          onShutdown('jobs.scheduler.accept', stopDispatching, { phase: 'accept' }),
+          onShutdown('jobs.scheduler', (reason) => stop('SIGTERM', reason.deadlineAt), {
+            phase: 'close',
+          }),
+        ];
       }
       schedule();
       logger.info('jobs.scheduler.started', { tasks: (options.tasks ?? registeredTasks()).length });

package/src/steps.ts

@@ -7,10 +7,11 @@
 // catches it and re-queues the job for `resumeAt` instead of holding a process for three days.
 
 import type { Clock } from '@ultimat3/core';
-import { logger } from '@ultimat3/core';
+import { logger, renderThrowable } from '@ultimat3/core';
 import type { DurationInput } from './clock';
 import { nowMs, toMs } from './clock';
 import { JobAbortedError, JobTimeoutError, StepDuplicateError } from './errors';
+import { createRunSignal } from './run-signal';
 
 /**
  * The runtime list is the declaration and `StepStatus` is derived from it, the shape
@@ -283,10 +284,16 @@ export function createStepRunner(options: StepRunnerOptions): StepRunner {
     // The step's own ceiling, folded into the run's cancellation so the body reads ONE signal and
     // sees whichever deadline lands first. Composed only when there is a second one to compose.
     const deadline = new AbortController();
-    const signal =
-      options.stepTimeoutMs === undefined
-        ? runSignal
-        : AbortSignal.any([runSignal, deadline.signal]);
+    // `createRunSignal` and never `AbortSignal.any`, which is the rule this package's `CLAUDE.md`
+    // states as absolute: a composite cannot be undone, so ONE dependent signal per STEP stayed on
+    // the run's signal for the whole attempt. A `backfill()` at `batch: 1000` over 5M rows is
+    // 5,000 of them held at once, and an app whose `WorkerOptions.context()` carries a
+    // process-lifetime signal keeps them past the run. Composed only when there is a second signal
+    // to compose, and DISPOSED in the `finally` below, which is the whole reason `run-signal.ts`
+    // exists — `worker-run.ts` disposes the run's own the same way.
+    const composed =
+      options.stepTimeoutMs === undefined ? null : createRunSignal([runSignal, deadline.signal]);
+    const signal = composed?.signal ?? runSignal;
     try {
       const output = await withStepTimeout(
         fn(signal),
@@ -320,12 +327,16 @@ export function createStepRunner(options: StepRunnerOptions): StepRunner {
           status: 'failed',
           startedAt,
           attempts,
-          error: error instanceof Error ? error.message : String(error),
+          error: renderThrowable(error),
         };
         await store.put(failure);
         remember(failure);
       }
       throw error;
+    } finally {
+      // Nothing of the run's is held past the step. Idempotent, and it never aborts: a step that
+      // settled leaves its signal in whatever state it ended in.
+      composed?.dispose();
     }
   }
 

package/src/worker-fleet-slots.ts

@@ -3,7 +3,7 @@
 // Apart from `worker.ts` because the claim loop's question is "may I start this one?" — which job
 // holds which slot, and who gives it back, is bookkeeping of its own.
 
-import { logger } from '@ultimat3/core';
+import { logger, renderThrowable } from '@ultimat3/core';
 import type { ClaimedJob } from './driver';
 import { getJob } from './job';
 import type { HeldLease, LeaseStore } from './leases';
@@ -121,7 +121,7 @@ export function createFleetSlots(options: FleetSlotOptions): FleetSlots {
         logger.warn('jobs.worker.lease-release-failed', {
           workerId: options.workerId,
           jobId,
-          error: error instanceof Error ? error.message : String(error),
+          error: renderThrowable(error),
         });
       });
     },

package/src/worker.ts

@@ -4,8 +4,17 @@
 // deploy turns "at least once" into "always twice", so draining is on by default.
 
 import type { Clock, Ctx } from '@ultimat3/core';
-import { logger, onShutdown, recordJob, recordQueueDepth, uuid } from '@ultimat3/core';
+import {
+  beginWork,
+  logger,
+  onShutdown,
+  recordJob,
+  recordQueueDepth,
+  renderThrowable,
+  uuid,
+} from '@ultimat3/core';
 import { nowMs } from './clock';
+import { settleAllBy } from './drain-wait';
 import type { ClaimedJob, JobDriver, QueueStats } from './driver';
 import { DEFAULT_QUEUE, DEFAULT_VISIBILITY_TIMEOUT_MS } from './driver';
 import { ConcurrencyUnenforceableError } from './errors';
@@ -106,8 +115,12 @@ export function createWorker(options: WorkerOptions): Worker {
   const rounds = new Set<Promise<unknown>>();
   let state: WorkerStats['state'] = 'idle';
   let loop: ReturnType<typeof setTimeout> | undefined;
-  /** The `onShutdown` registration this worker holds while it runs. Handed back by `stop()`. */
-  let releaseShutdownHook: (() => void) | undefined;
+  /**
+   * The two `onShutdown` registrations this worker holds while it runs, both handed back by
+   * `stop()`. Two, because they answer different questions in different PHASES — the split
+   * `listenSyncNode` and `@ultimat3/http` already have.
+   */
+  let releaseShutdownHooks: (() => void)[] = [];
   /** The teardown in flight, so a second `stop()` joins it instead of running a second one. */
   let stopping: Promise<void> | undefined;
   let processed = 0;
@@ -141,7 +154,7 @@ export function createWorker(options: WorkerOptions): Worker {
       // Instrumentation never costs a tick: a queue that cannot be measured must still be worked.
       logger.warn('jobs.worker.depth-failed', {
         workerId,
-        error: error instanceof Error ? error.message : String(error),
+        error: renderThrowable(error),
       });
     }
   };
@@ -255,6 +268,10 @@ export function createWorker(options: WorkerOptions): Worker {
           continue;
         }
 
+        // The claimed job is the process's in-flight work, counted where the DRAIN can see it:
+        // core's own in-flight wait sits between `accept` and `inflight` and exists for exactly
+        // this. Counted nowhere, the worker had to wait for its own jobs inside a hook.
+        const finishWork = beginWork();
         const running = runClaimed(job)
           .then((execution) => {
             if (execution.outcome === 'completed') processed += 1;
@@ -269,9 +286,20 @@ export function createWorker(options: WorkerOptions): Worker {
             if (label !== null) recordJob(queue, label);
             return execution;
           })
-          .finally(() => {
+          .finally(async () => {
             lease.release();
-            void fleetSlots.release(job.id);
+            // AWAITED, never `void`: the slot is a row in `x_job_leases`, so the DELETE was still
+            // on the wire when the teardown's `allSettled` returned and `driver.close()` took the
+            // connection out from under it — a `concurrency: 1` job unclaimable by the pod
+            // replacing this one for a whole visibility window after every deploy. `release`
+            // swallows its own failures, so awaiting it cannot reject a job that finished; the
+            // `finally` is for the one that could, because a lost `finishWork()` is an in-flight
+            // count that never returns to zero and a drain that waits out its whole budget.
+            try {
+              await fleetSlots.release(job.id);
+            } finally {
+              finishWork();
+            }
           });
 
         started.push(running);
@@ -290,7 +318,7 @@ export function createWorker(options: WorkerOptions): Worker {
               workerId,
               job: job.name,
               jobId: job.id,
-              error: error instanceof Error ? error.message : String(error),
+              error: renderThrowable(error),
             });
           },
         );
@@ -332,7 +360,7 @@ export function createWorker(options: WorkerOptions): Worker {
         .catch((error: unknown) => {
           logger.error('jobs.worker.tick-failed', {
             workerId,
-            error: error instanceof Error ? error.message : String(error),
+            error: renderThrowable(error),
           });
         })
         .finally(() => {
@@ -341,18 +369,42 @@ export function createWorker(options: WorkerOptions): Worker {
     }, pollIntervalMs);
   };
 
-  const teardown = async (reason: string): Promise<void> => {
+  /**
+   * The whole of the `accept` phase: stop taking work, and nothing else. Synchronous on purpose —
+   * a phase whose job is to be over before the load balancer's next health check must not contain
+   * a wait, and the hook behind this one is somebody else's "stop listening".
+   */
+  const stopAccepting = (): void => {
+    if (state === 'stopped') return;
     state = 'draining';
     if (loop !== undefined) clearTimeout(loop);
     loop = undefined;
+  };
+
+  const teardown = async (reason: string, deadlineAt?: number): Promise<void> => {
+    stopAccepting();
     logger.info('jobs.worker.draining', { workerId, reason, inFlight: inFlight.size });
     try {
       // Stop claiming, finish what we hold, then close. Anything else re-runs work on deploy.
       // Rounds first: one that passed the guard before the flag flipped is still awaiting its
       // `claim()`, and the jobs it starts join `inFlight` after any snapshot taken here — so a
       // drain that waited on `inFlight` alone closed the driver under a job that had just begun.
-      await Promise.allSettled([...rounds]);
-      await Promise.allSettled([...inFlight]);
+      //
+      // Both waits share ONE deadline on the SIGTERM path (`undefined` on a manual stop, which
+      // waits as long as its jobs take). Nothing can kill a body that ignores `ctx.signal`, so an
+      // unbounded wait here is a teardown that never ends: driver never closed, state never past
+      // 'draining', and the memoized `stopping` every later `stop()` joins never settling.
+      // Abandoning costs a lapsed lease and a redelivered job — at-least-once, as promised.
+      const rounded = await settleAllBy([...rounds], deadlineAt);
+      const drained = (await settleAllBy([...inFlight], deadlineAt)) && rounded;
+      if (!drained) {
+        logger.warn('jobs.worker.drain-abandoned', {
+          workerId,
+          reason,
+          inFlight: inFlight.size,
+          fix: 'raise the drain budget past the slowest job — configureLifecycle({ deadlineMs: 600_000 }) — and set terminationGracePeriodSeconds to at least as many seconds',
+        });
+      }
       await options.driver.close?.();
     } finally {
       // Whatever the close did, this worker is done: a state left at 'draining' is a drain that
@@ -362,19 +414,22 @@ export function createWorker(options: WorkerOptions): Worker {
       // through a driver already closed — and keeps this closure, its driver and its in-flight
       // set alive with it.
       state = 'stopped';
-      releaseShutdownHook?.();
-      releaseShutdownHook = undefined;
+      for (const release of releaseShutdownHooks) release();
+      releaseShutdownHooks = [];
     }
   };
 
-  const stop = async (reason = 'stop'): Promise<void> => {
+  const stop = async (reason = 'stop', deadlineAt?: number): Promise<void> => {
+    // Answered immediately once this worker is done: the teardown always REACHES 'stopped' (its
+    // waits are bounded and the state is set in a `finally`), so a caller landing after an
+    // abandoned drain gets an answer rather than joining a promise that never settles.
     if (state === 'stopped') return;
     // One teardown, joined rather than repeated: a SIGTERM landing on a manual stop must wait out
     // the same in-flight work, not close the driver a second time underneath it. Cleared as it
     // settles, so a worker that started again tears down again instead of joining a promise that
     // settled a lifetime ago. A close that threw still stopped this worker — the failure is the
     // caller's to see on the promise it awaited, not a teardown to run twice.
-    stopping ??= teardown(reason).finally(() => {
+    stopping ??= teardown(reason, deadlineAt).finally(() => {
       stopping = undefined;
     });
     await stopping;
@@ -400,13 +455,22 @@ export function createWorker(options: WorkerOptions): Worker {
       }
       state = 'running';
       logger.info('jobs.worker.started', { workerId, queues });
-      // 'accept' phase: stop claiming before core waits on in-flight jobs. The unregister is
-      // kept, never discarded: `stop()` hands it back, so start -> stop -> start holds ONE hook
-      // rather than one per start, each retaining the driver of a worker that is already gone.
+      // TWO hooks, for the two phases that answer two questions. `accept` stops claiming and
+      // returns, so every hook behind it — the HTTP server's "stop listening", the sync node's
+      // "stop upgrading" — runs while the budget is still whole; one hook doing both spent all of
+      // it in the phase whose whole purpose is to be quick. `close` waits out what this worker
+      // holds and closes the driver, bounded by the deadline the hook is handed.
+      //
+      // Both unregisters are kept, never discarded: `stop()` hands them back, so
+      // start -> stop -> start holds one pair rather than one per start, each retaining the
+      // driver of a worker that is already gone.
       if (options.drainOnShutdown !== false) {
-        releaseShutdownHook = onShutdown(`jobs.worker.${workerId}`, () => stop('SIGTERM'), {
-          phase: 'accept',
-        });
+        releaseShutdownHooks = [
+          onShutdown(`jobs.worker.${workerId}.accept`, stopAccepting, { phase: 'accept' }),
+          onShutdown(`jobs.worker.${workerId}`, (reason) => stop('SIGTERM', reason.deadlineAt), {
+            phase: 'close',
+          }),
+        ];
       }
       schedule();
     },