@ultimat3/jobs 10.0.0 → 11.0.0

package/CLAUDE.md

@@ -194,12 +194,38 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 - The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`) and `task.enqueue()`'s
   is the job's plain key. Deliberate: the first stops two schedulers double-firing a tick, the
   second is a manual run with no occurrence to scope to.
-- **One shutdown hook per worker, and `stop()` is what hands it back.** `start()` keeps the
-  unregister `onShutdown` returns; the teardown releases it in a `finally`, so a close that threw
-  still gives it up. Discarding it was a hook per `start()` — the `start()` guard reads a
-  standstill, so start -> stop -> start stacked a second registration retaining a stopped
-  worker's driver, and the next process-wide drain ran all of them. `start()` refuses while
-  draining for the same reason: a claim loop back on a driver the drain is about to close.
+- **TWO shutdown hooks per worker — one per PHASE — and `stop()` is what hands both back**
+  (`As of 2026-08-23`). `accept` is `stopAccepting()`: flip the state, clear the poll timer, return.
+  `close` is the teardown: wait out the rounds and the in-flight jobs, close the driver.
+
+  It was ONE hook, at `accept`, doing all of it — so a single 10-minute job spent the entire drain
+  deadline inside the phase whose whole purpose is to be over immediately, and every hook behind it
+  was invoked with **0 ms** left: `@ultimat3/http`'s "stop listening" and `listenSyncNode`'s
+  `stopAccepting` are both `accept` hooks, so the pod went on serving requests and upgrading
+  websockets for the whole of the drain the load balancer had already been told about. Reproduced —
+  4 hooks started, none finished.
+
+  `start()` keeps both unregisters; the teardown releases them in a `finally`, so a close that threw
+  still gives them up. Discarding them was a hook per `start()` — the `start()` guard reads a
+  standstill, so start -> stop -> start stacked a second registration retaining a stopped worker's
+  driver, and the next process-wide drain ran all of them. `start()` refuses while draining for the
+  same reason: a claim loop back on a driver the drain is about to close.
+- **A claimed job is counted with core's `beginWork()`, so the DRAIN does the waiting**
+  (`As of 2026-08-23`). The wait for in-flight jobs belongs to the phase between `accept` and
+  `inflight`, which exists for exactly this and is where `@ultimat3/http` already puts a request —
+  not to a hook, where one role's work starves every other role's teardown. `/readyz`'s `inflight`
+  becomes truthful on a worker node as a consequence.
+- **The teardown's wait is BOUNDED on the SIGTERM path and unbounded on a manual `stop()`**
+  (`As of 2026-08-23`). The `close` hook is handed `ShutdownReason.deadlineAt` and passes it to
+  `settleAllBy` (`drain-wait.ts`); a manual `stop()` passes nothing and waits as long as its
+  jobs take, because a caller that asked has no budget to spend. Nothing in JS can kill a body that
+  ignores `ctx.signal`, so the unbounded version was a teardown that never returned: the driver was
+  never closed, `state` never left `'draining'`, and the memoised `stopping` promise every later
+  `stop()` joins never settled — `x dev`'s role rollback awaits that promise. Abandoning the wait
+  costs a lapsed lease and a job the queue delivers again, which is what at-least-once already
+  promises; `jobs.worker.drain-abandoned` names it, with the `configureLifecycle({ deadlineMs })`
+  raise as its fix. **A worker always REACHES `'stopped'`**, which is what makes `stop()`'s
+  `state === 'stopped'` early return an answer rather than a wedge.
 - **One teardown, joined.** `stop()` shares the in-flight teardown promise, so a SIGTERM landing
   on a manual stop waits out the same in-flight jobs instead of closing the driver underneath
   it. The promise is cleared as it settles, so a worker that started again tears down again
@@ -223,6 +249,13 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   a climbing `queue_depth` as the only symptoms. The `catch` releases the lease and rethrows; the
   claimed row goes back to the queue by its visibility timeout, as it does for any round that dies.
 
+  `fleetSlots.release(jobId)` is AWAITED in the settle's `.finally`, never `void`ed: the slot is a
+  DELETE in `x_job_leases`, so a fire-and-forget one was still on the wire when the teardown's
+  `allSettled` returned and `driver.close()` took the connection out from under it — the row then
+  held its slot for a full TTL, and a `concurrency: 1` job was unclaimable by the pod replacing this
+  one for a whole visibility window after every deploy. `release` swallows its own failures, so
+  awaiting it cannot turn a finished job into a rejected one.
+
   `LeaseStore.renew` answering `false` means the row is another holder's — two runs live under a cap
   of one — and it was discarded by `.catch(noop)`. It now stops the timer, logs
   `jobs.worker.slot-lost` and aborts the run through `X_JOB_SLOT_LOST`. Its own code, not
@@ -316,10 +349,14 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   waits that round out BEFORE `leader.release()` — a lock handed back mid-dispatch promotes a
   standby onto an occurrence this node is still enqueueing for, which is the double-fire leader
   election exists to prevent — and the round re-reads the drain state before each task, so "stop
-  dispatching" means this round too. Same hook rules as the worker: one `onShutdown` at the
-  `accept` phase while it runs, handed back in the teardown's `finally` so a `release()` that
-  threw still gives it up, `isLeader` cleared there too because a lock this process could not
-  hand back is never treated as still held.
+  dispatching" means this round too. Same hook rules as the worker, `As of 2026-08-23`: **TWO**
+  `onShutdown` registrations — `accept` stops dispatching and returns, `close` waits the round out
+  and releases the lease under the deadline it is handed (`settleAllBy`) — both handed back in the
+  teardown's `finally` so a `release()` that threw still gives them up, `isLeader` cleared there
+  too because a lock this process could not hand back is never treated as still held. **An
+  ABANDONED round does not release**: it is still enqueueing, so handing the lock over is that same
+  double-fire delivered by the shutdown, and a lease row expiring on its own is the safe end —
+  `jobs.scheduler.drain-abandoned` is the line that says so.
 - **`backfill()` is a FACTORY over `job()`, never a ninth primitive.** Same rule `llm()` follows
   in `@ultimat3/ai`: a new capability arrives as a factory over an existing primitive, so a
   backfill inherits `.enqueue()`, the retry policy, the cancellation, the dead-letter path and
@@ -363,6 +400,14 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   the CHANGELOG, `CLAUDE.md` and `x g backfill`'s generated source all say the handler must be
   idempotent. Never invert it: checkpointing first would report a page as swept that nobody wrote,
   and a lost page is unrecoverable where a repeated one is the handler's problem.
+- **A REPLAYED batch writes no ledger row** (`As of 2026-08-23`). `ledger.progress` sits outside
+  `step.run` — it has to, the ledger is not transactional with the steps — so a resumed pass
+  re-issued one `x_backfills` UPDATE per already-completed batch before it read a single new row:
+  4,800 statements on a 5M-row sweep killed at batch 4,800, on every attempt, inside the visibility
+  lease the heartbeat is renewing. The flag is set INSIDE the step body, because that is the only
+  thing that can tell a replay from a run — `step.run` answers the same shape either way. Nothing is
+  lost: the value is ABSOLUTE, so the first batch that does run reports every replayed one behind
+  it, and `finish` writes the total regardless.
 - **`x_backfills` is what has already been SWEPT, and the step checkpoints are where a pass
   resumes. Never the other way round.** The ledger row (`backfill-ledger.ts`) is a report an
   operator reads and the record that a completed name is done; the checkpoints are written in step
@@ -528,7 +573,11 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   branch. `jobs.lease.lost` at error plus `recordLeaseLost(queue)` is the one signal meaning the
   queue re-delivered a job this process was still running, so a false one is a page for a
   non-event, and the window widens exactly when the pool is slow. Re-read the flag AFTER every
-  await and inside the reporter, the way `settleWithin`'s `decided` does in core.
+  await and inside the reporter, the way `settleWithin`'s `decided` does in core. **The interval is
+  `unref`ed**, like all three of `sync-node.ts`'s: it is armed from inside a job run, so a drain
+  that ABANDONS the worker's hook leaves the run — and this timer — with nobody left to call
+  `stop()`, and a refed interval is then the one thing holding a drained process open until the
+  kubelet's SIGKILL.
 - **`stepTimeout` and `eventPoll` are DECLARED on the job, and `execute.ts` is the only place they
   are forwarded** (`As of 2026-08`). `StepRunnerOptions` carried both, `withStepTimeout`
   implemented the ceiling and `steps.test.ts` exercised it by building a runner BY HAND — while the
@@ -617,11 +666,18 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 - **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which REFUSES rather than answering
   a silent no-op.** An operator cancelling a 40M-row sweep has to know whether they stopped it or
   missed it, so a finished job is `X_JOB_NOT_CANCELLABLE` and a driver with no `cancel` is too.
-- **The scheduler asks `leader.acquire()` EVERY round, not only while it thinks it is not the
-  leader.** A lease-backed election expires, so `acquire()` is its renewal and a cached
-  `isLeader = true` would keep dispatching past a lease another node already took. `soleLeader`
-  answers true every time and `createPgLeader` holds its grant behind an internal flag, so the
-  extra call is a no-op for both — the flag also stops Postgres refcounting a second advisory
+- **The scheduler asks `leader.acquire()` EVERY round AND before EVERY task in it, not only while
+  it thinks it is not the leader.** A lease-backed election expires, so `acquire()` is its renewal
+  and a cached `isLeader = true` would keep dispatching past a lease another node already took.
+  Per-task as well as per-round `As of 2026-08-23`: a round walks its tasks serially with an enqueue
+  per job, so a 30s lease and a slow queue leave the tail of the walk running under a lease node B
+  already holds — and **the occurrence key does not absorb that**, because `SQL_ENQUEUE`'s conflict
+  target is the PARTIAL index over the live states, so a second dispatch landing after that
+  occurrence's job completed or dead-lettered inserts a new row and the handler runs twice. Argued
+  from the index definition, not reproduced — `stillLeading()` is the one place the question is
+  asked and `break` is the answer, the same shape the drain's `dispatching()` check already had.
+  `soleLeader` answers true every time and `createPgLeader` holds its grant behind an internal flag,
+  so the extra calls are a no-op for both — the flag also stops Postgres refcounting a second advisory
   grant that `release()`'s single unlock would never hand back.
 - **`createPgLeader` is correct only on a DEDICATED connection and boot has a pool.** A
   session-level `pg_try_advisory_lock` is released when its connection returns to the pool, so
@@ -707,6 +763,7 @@ picture from the other side.
 | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
 | `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
 | `worker.ts` | `worker` role, claim loop, drain |
+| `drain-wait.ts` | the drain's wait, shared by both roles: everything a teardown holds, settled — or abandoned at the budget the `close` hook was handed |
 | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
 | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |
 | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back. The claim loop asks "may I start this one?"; this answers it across the fleet |

package/README.md

@@ -475,6 +475,24 @@ debugging a stuck queue can read and run the exact statement.
 | `worker` | `createWorker({ driver, context, queues, concurrency })` | per-queue pools, lease heartbeat, SIGTERM drain: stop claiming → finish in-flight → close |
 | `scheduler` | `createScheduler({ driver, leader, state })` | one dispatch round at a time, catch-up policy, SIGTERM drain: stop dispatching → finish the round → release the lock |
 
+Both roles register the same **pair** of hooks and bound the `close` half the same way. The
+scheduler's abandoned case differs in one respect: a round still enqueueing keeps the lease rather
+than handing it back, because promoting a standby onto an occurrence this node is mid-dispatch for
+is the double-fire leader election exists to prevent. The lease row expires on its own.
+
+The worker's drain is **two shutdown hooks**, one per phase: `accept` stops claiming and returns
+immediately, `close` waits out the in-flight jobs and closes the driver. A running job is counted
+with core's `beginWork()`, so the wait for it happens in the drain's own in-flight phase rather than
+inside a hook — one hook doing all of it spends the whole `configureLifecycle({ deadlineMs })`
+budget in `accept`, where every other role's "stop taking work" is still queued behind it.
+
+**The `close` hook's wait is bounded by that same budget**, because nothing can kill a handler that
+ignores `ctx.signal`: at the deadline the worker logs `jobs.worker.drain-abandoned`, closes the
+driver and reaches `stopped`, and the job's lease lapses so the queue delivers it again — which is
+what at-least-once already promises. Raise the budget past your slowest job rather than relying on
+the wait: `configureLifecycle({ deadlineMs: 600_000 })`, with a `terminationGracePeriodSeconds` at
+least as large. A manual `await worker.stop()` has no budget and waits as long as its jobs take.
+
 `driver` and `context` are the two required keys on `WorkerOptions`; everything else defaults.
 `context: () => Ctx` supplies the ambient `Ctx` a job run executes under — the app wires its ALS
 and its tenant there, and a worker with no way to build one would run every handler as nobody.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "10.0.0",
+  "version": "11.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "10.0.0",
-    "@ultimat3/entity": "10.0.0",
-    "@ultimat3/schema": "10.0.0",
-    "@ultimat3/time": "10.0.0"
+    "@ultimat3/core": "11.0.0",
+    "@ultimat3/entity": "11.0.0",
+    "@ultimat3/schema": "11.0.0",
+    "@ultimat3/time": "11.0.0"
   }
 }

package/src/backfill-pass.ts

@@ -223,6 +223,12 @@ export async function backfillPass<Row>(
       try {
         for (let index = 0; ; index += 1) {
           const stepName = `${STEP_PREFIX}${index}`;
+          /**
+           * Whether this batch's BODY ran, or whether the runner served its checkpoint from
+           * storage. Only the body can be asked: `step.run` answers the same shape either way,
+           * which is the whole point of a replay — so the flag is set inside it.
+           */
+          let swept = false;
           const checkpoint = asCheckpoint(
             await step.run(stepName, async (signal): Promise<Checkpoint> => {
               // INSIDE the body, which is the whole of it: a completed step is served from storage
@@ -237,6 +243,7 @@ export async function backfillPass<Row>(
               const next = await iteration.pull.next();
               if (next.done === true) return { cursor: null, rows: 0 };
               await definition.handle({ rows: next.value, ctx, signal, index });
+              swept = true;
               return { cursor: iteration.batches.cursor, rows: next.value.length };
             }),
             stepName,
@@ -248,8 +255,14 @@ export async function backfillPass<Row>(
           // whose last write is `finish` either way.
           if (checkpoint.rows > 0) {
             batches += 1;
-            // Absolute, so a replayed batch reports the position it reported the first time.
-            await ledger?.progress(runId, { rows, cursor });
+            // Only for a batch this attempt actually SWEPT. A replayed checkpoint runs no body and
+            // touches no row, so writing here re-issued one `x_backfills` UPDATE per completed
+            // batch before a resumed pass read a single new row — 4,800 statements on a 5M-row
+            // sweep killed at batch 4,800, on every attempt, inside the visibility lease the
+            // heartbeat is renewing. Nothing is lost by skipping them: the value is ABSOLUTE, so
+            // the first batch that does run reports every replayed one behind it, and `finish`
+            // writes the total whatever happens.
+            if (swept) await ledger?.progress(runId, { rows, cursor });
           }
           if (cursor === null) break;
         }

package/src/drain-wait.ts

@@ -0,0 +1,50 @@
+// The wait a role's teardown makes, under the drain's own budget — the worker's in-flight jobs,
+// the scheduler's dispatch round. One shape for both, because the race closes over nothing either
+// of them holds: the same seam `lifecycle-deadline.ts` takes in core, and for the same reason —
+// an unbounded wait inside a shutdown hook is a process the kubelet ends with SIGKILL.
+
+import { systemClock } from '@ultimat3/core';
+
+/**
+ * Everything in `pending`, settled — or abandoned once `deadlineAt` (real monotonic ms, the clock
+ * `ShutdownReason.deadlineAt` is measured on) has passed. Answers `true` when everything settled.
+ *
+ * `deadlineAt` is `undefined` for a MANUAL `stop()`, which waits as long as its work takes: a
+ * caller that asked a role to stop has no budget to spend, and closing the queue under a live job
+ * — or handing the lease back under a live dispatch — is exactly what draining exists to prevent. The bound belongs to the SIGTERM path, where the
+ * budget is real and a handler that ignores `ctx.signal` would otherwise hold the teardown — and
+ * with it the memoized `stopping` promise every later `stop()` joins — open forever.
+ *
+ * `allSettled`, so work that rejected is work that finished: each caller observes its own failures
+ * already, and a teardown that rethrew here would skip the close behind it.
+ */
+export async function settleAllBy(
+  pending: readonly Promise<unknown>[],
+  deadlineAt: number | undefined,
+): Promise<boolean> {
+  if (pending.length === 0) return true;
+  const settled = Promise.allSettled(pending);
+  if (deadlineAt === undefined) {
+    await settled;
+    return true;
+  }
+  const remainingMs = Math.max(0, deadlineAt - systemClock.monotonic());
+  return await new Promise<boolean>((resolve) => {
+    let decided = false;
+    const timer = setTimeout(() => {
+      if (decided) return;
+      decided = true;
+      resolve(false);
+    }, remainingMs);
+    // Never the thing keeping a drained process alive — the rule `lifecycle-deadline.ts` states
+    // for its own timer. A spent budget still gives the already-settled case its turn, because a
+    // resolved promise settles on a microtask and this timer on a macrotask.
+    timer.unref?.();
+    void settled.then(() => {
+      if (decided) return;
+      decided = true;
+      clearTimeout(timer);
+      resolve(true);
+    });
+  });
+}

package/src/driver-memory.ts

@@ -83,6 +83,22 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
     jobs.set(id, { ...existing, ...patch, updatedAt: nowMs(clock) });
   };
 
+  /**
+   * `update`, plus the lease columns `SQL_ACK`/`SQL_NACK` set to `null`.
+   *
+   * A settlement RELEASES the claim, and a `Partial<JobRecord>` cannot say so: both fields are
+   * optional, so `visibleAt: undefined` would keep the key and `{ ...existing }` keeps the value.
+   * Left stamped, a `done` row still named the worker that finished it and carried that attempt's
+   * lease deadline — which `x jobs show` prints, and which is the very pair the claim scan's
+   * lease-expiry branch reads to decide a row was abandoned.
+   */
+  const settle = (id: string, patch: Partial<JobRecord>): void => {
+    const existing = jobs.get(id);
+    if (existing === undefined) return;
+    const { visibleAt: _visibleAt, claimedBy: _claimedBy, ...released } = existing;
+    jobs.set(id, { ...released, ...patch, updatedAt: nowMs(clock) });
+  };
+
   const introspect: JobIntrospection = {
     job(jobId) {
       return Promise.resolve(jobs.get(jobId));
@@ -215,7 +231,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
     // would otherwise overwrite a row it no longer owns.
     ack(jobId: string): Promise<void> {
       if (jobs.get(jobId)?.state !== 'running') return Promise.resolve();
-      update(jobId, { state: 'done' });
+      settle(jobId, { state: 'done' });
       return Promise.resolve();
     },
 
@@ -241,7 +257,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
         attempt: counts ? record.attempt : Math.max(0, record.attempt - 1),
         ...(nackOptions.error === undefined ? {} : { lastError: nackOptions.error }),
       };
-      update(jobId, patch);
+      settle(jobId, patch);
       return Promise.resolve();
     },
 

package/src/renewal-timer.ts

@@ -39,6 +39,13 @@ export function startRenewalTimer(
         logger.error('jobs.renewal.raised', { error: renderThrowable(error) });
       });
   }, intervalMs);
+  // Never the thing keeping a drained process alive. This interval is armed from inside a job run,
+  // so a drain that ABANDONS the worker's hook leaves the run — and this timer — with nobody left
+  // to call `stop()`: refed, it holds the event loop open past every phase of the shutdown, and
+  // the kubelet's SIGKILL becomes the exit. `sync-node.ts` unrefs all three of its timers and
+  // `lifecycle-deadline.ts` its own for the same reason. A renewal is bookkeeping for work that is
+  // already over by then; nothing is lost by letting the process go.
+  timer.unref?.();
   return {
     stopped: () => stopped,
     stop(): void {

package/src/scheduler.ts

@@ -6,16 +6,25 @@
 // advisory lock is held by the SESSION, not by this process — it outlives every transaction and is
 // released only by an explicit unlock, the pool's reset on release, or the connection dying, and
 // the next round may run on a different connection. So a node can neither renew it nor prove it
-// still holds one, and leadership passes to a second node while the first is still dispatching. Two schedulers
-// double-enqueue every task; the idempotency key would absorb it, but leader election means
-// the queue never sees the duplicate at all. One ROUND at a time is the same rule inside one
-// process: the loop re-arms on the round it just finished, and any other caller joins that
-// round rather than opening a second one over the same `lastFiredAt`.
+// still holds one, and leadership passes to a second node while the first is still dispatching.
+//
+// **The occurrence key is not a second line of defence, and the lease is therefore re-asserted
+// before EVERY task, not once per round.** `SQL_ENQUEUE`'s conflict target is the PARTIAL index
+// over the live states (`ready`, `delayed`, `running`, `suspended`), so a duplicate enqueue is
+// absorbed only while the first job is still one of those: a second dispatcher landing after that
+// occurrence's job completed, dead-lettered or was retried past its row inserts a NEW row, and the
+// handler runs twice. A round walks its tasks serially with an enqueue per job, so a 30s lease and
+// a slow queue leave the tail of the walk running under a lease another node already took. Argued
+// from the index definition, not reproduced — say it that way, as `outbox.ts` does for the same
+// mechanism. One ROUND at a time is the same rule inside one process: the loop re-arms on the
+// round it just finished, and any other caller joins that round rather than opening a second one
+// over the same `lastFiredAt`.
 
 import type { Clock } from '@ultimat3/core';
 import { isUltimateError, logger, onShutdown, renderThrowable } from '@ultimat3/core';
 import { instant, nextCronOccurrence } from '@ultimat3/time';
 import { nowMs } from './clock';
+import { settleAllBy } from './drain-wait';
 import type { JobDriver } from './driver';
 import type { TaskHandle, TaskJobResult } from './task';
 import { registeredTasks } from './task';
@@ -114,8 +123,11 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
   let state: 'idle' | 'running' | 'draining' | 'stopped' = 'idle';
   /** The dispatch round in flight — what a second caller joins and what `stop()` waits out. */
   let round: Promise<readonly DispatchedOccurrence[]> | undefined;
-  /** The `onShutdown` registration this scheduler holds while it runs. Handed back by `stop()`. */
-  let releaseShutdownHook: (() => void) | undefined;
+  /**
+   * The two `onShutdown` registrations this scheduler holds while it runs, both handed back by
+   * `stop()`. Two, for the phases they answer — the worker's rule, and `listenSyncNode`'s.
+   */
+  let releaseShutdownHooks: (() => void)[] = [];
   /** The teardown in flight, so a SIGTERM landing on a manual stop joins it. */
   let stopping: Promise<void> | undefined;
 
@@ -176,24 +188,31 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
   /** The drain's one question: may this scheduler still dispatch an occurrence? */
   const dispatching = (): boolean => state !== 'draining' && state !== 'stopped';
 
+  /**
+   * Asked EVERY round AND before every task in it, never only while `isLeader` is false. A
+   * lease-backed election (the one a pooled executor can use — `createPgLeaseLeader`) expires on a
+   * wall clock, so `acquire()` is also its renewal and a node that cached `isLeader = true` keeps
+   * dispatching past a lease another node has already taken. `soleLeader` answers true every time
+   * and `createPgLeader` holds its grant behind an internal flag, so the extra calls are a no-op
+   * for both.
+   */
+  const stillLeading = async (): Promise<boolean> => {
+    if (await leader.acquire()) {
+      isLeader = true;
+      return true;
+    }
+    // Demoted, or never elected. Nothing to release — a lease we no longer hold is not ours to
+    // hand back, and `teardown` reads this same flag before it calls `release()`.
+    if (isLeader) logger.warn('jobs.scheduler.leadership-lost', { at: nowMs(options.clock) });
+    isLeader = false;
+    return false;
+  };
+
   const runRound = async (): Promise<readonly DispatchedOccurrence[]> => {
     // Never take the lock a drain is on its way to releasing: a round that acquired it here
     // would still be enqueueing after `stop()` handed the occurrence to the next node.
     if (!dispatching()) return [];
-    // Asked EVERY round, not only while `isLeader` is false. A lease-backed election (the one a
-    // pooled executor can use — `createPgLeaseLeader`) expires, so `acquire()` is also its
-    // renewal, and a node that cached `isLeader = true` would keep dispatching past a lease
-    // another node has already taken. `soleLeader` answers true every time, and `createPgLeader`
-    // holds its grant internally, so this is a no-op for both.
-    const held = await leader.acquire();
-    if (!held) {
-      // Demoted, or never elected. Nothing to release — a lease we no longer hold is not ours to
-      // hand back, and `teardown` reads this same flag before it calls `release()`.
-      if (isLeader) logger.warn('jobs.scheduler.leadership-lost', { at: nowMs(options.clock) });
-      isLeader = false;
-      return [];
-    }
-    isLeader = true;
+    if (!(await stillLeading())) return [];
 
     const at = nowMs(options.clock);
     const tasks = options.tasks ?? registeredTasks();
@@ -204,6 +223,10 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // at the next round. A task not reached simply fires next time — its `lastFiredAt` is
       // untouched — while the occurrence this round already began is the one `stop()` waits for.
       if (!dispatching()) break;
+      // The lease, on the same rule and for the same reason the drain state is re-read: it expires
+      // on a wall clock in the middle of this walk, not between rounds. See the file header for
+      // what a second dispatcher costs — the occurrence key does NOT absorb it in general.
+      if (!(await stillLeading())) break;
       const last = await schedulerState.lastFiredAt(handle.name);
       if (last === undefined) {
         // First sight of this task: arm it, never fire retroactively for all of history.
@@ -275,10 +298,20 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
     }, tickIntervalMs);
   };
 
-  const teardown = async (reason: string): Promise<void> => {
+  /**
+   * The whole of the `accept` phase: stop dispatching, and nothing else. Synchronous on purpose —
+   * the hook behind this one is somebody else's "stop taking work", and a phase that waits is a
+   * phase that spends the budget those hooks were going to need.
+   */
+  const stopDispatching = (): void => {
+    if (state === 'stopped') return;
     state = 'draining';
     if (timer !== undefined) clearTimeout(timer);
     timer = undefined;
+  };
+
+  const teardown = async (reason: string, deadlineAt?: number): Promise<void> => {
+    stopDispatching();
     logger.info('jobs.scheduler.draining', { reason, dispatching: round !== undefined });
     try {
       // The round this stop races runs to the end first. Releasing the lease under a
@@ -286,8 +319,24 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // then own the same occurrence — the exact double-fire leader election exists to prevent.
       // Settled, not awaited: a round that failed is its own caller's to see, and the lease still
       // has to go back.
-      await Promise.allSettled([round]);
-      if (isLeader) await leader.release();
+      //
+      // Bounded on the SIGTERM path, `undefined` on a manual stop: a round parked in `enqueue` on
+      // a queue that is not answering cannot be cancelled from here, and an unbounded wait is a
+      // teardown that never ends — hooks never handed back, `state` never past 'draining', and the
+      // memoised `stopping` every later `stop()` joins never settling.
+      const dispatched = await settleAllBy(round === undefined ? [] : [round], deadlineAt);
+      // A round we ABANDONED is a round still enqueueing, so the lock is deliberately NOT handed
+      // back: promoting a standby onto an occurrence this process is mid-dispatch for is the
+      // double-fire above, delivered by the shutdown. A lease row expires on its own, which is
+      // what an expiry is for — and `isLeader` is cleared below either way.
+      if (!dispatched) {
+        logger.warn('jobs.scheduler.drain-abandoned', {
+          reason,
+          fix: 'raise the drain budget past a dispatch round — configureLifecycle({ deadlineMs: 60_000 }) — and set terminationGracePeriodSeconds to at least as many seconds',
+        });
+      } else if (isLeader) {
+        await leader.release();
+      }
     } finally {
       // Whatever the release did, this scheduler is done. `isLeader` false because a lock this
       // process no longer holds — or failed to hand back — must never be re-used as if it did,
@@ -295,17 +344,20 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // the next process-wide drain, and keeps this closure and its driver alive with it.
       isLeader = false;
       state = 'stopped';
-      releaseShutdownHook?.();
-      releaseShutdownHook = undefined;
+      for (const release of releaseShutdownHooks) release();
+      releaseShutdownHooks = [];
     }
   };
 
-  const stop = async (reason = 'stop'): Promise<void> => {
+  const stop = async (reason = 'stop', deadlineAt?: number): Promise<void> => {
+    // Answered immediately once this scheduler is done: the teardown always REACHES 'stopped', so
+    // a caller landing after an abandoned drain gets an answer rather than joining a promise that
+    // never settles.
     if (state === 'stopped') return;
     // One teardown, joined rather than repeated — the worker's rule, for the same reason: a
     // SIGTERM landing on a manual stop must wait out the same round, not release the lock a
     // second time behind it. Cleared as it settles, so a scheduler started again stops again.
-    stopping ??= teardown(reason).finally(() => {
+    stopping ??= teardown(reason, deadlineAt).finally(() => {
       stopping = undefined;
     });
     await stopping;
@@ -317,14 +369,19 @@ export function createScheduler(options: SchedulerOptions): Scheduler {
       // about to release, and stack a second shutdown hook on the one still running.
       if (state !== 'idle' && state !== 'stopped') return;
       state = 'running';
-      // 'accept' phase: stop dispatching before core waits on in-flight work — an occurrence
-      // enqueued during the drain is work nothing in this process is left to run. The
-      // unregister is kept, never discarded: `stop()` hands it back, so start -> stop -> start
-      // holds ONE hook rather than one per start.
+      // TWO hooks, for the two phases. `accept` stops dispatching and returns — an occurrence
+      // enqueued during the drain is work nothing in this process is left to run, and every hook
+      // behind this one still has the whole budget. `close` waits the round out and hands the
+      // lease back, bounded by the deadline it is given. Both unregisters are kept, never
+      // discarded: `stop()` hands them back, so start -> stop -> start holds one pair rather
+      // than one per start.
       if (options.drainOnShutdown !== false) {
-        releaseShutdownHook = onShutdown('jobs.scheduler', () => stop('SIGTERM'), {
-          phase: 'accept',
-        });
+        releaseShutdownHooks = [
+          onShutdown('jobs.scheduler.accept', stopDispatching, { phase: 'accept' }),
+          onShutdown('jobs.scheduler', (reason) => stop('SIGTERM', reason.deadlineAt), {
+            phase: 'close',
+          }),
+        ];
       }
       schedule();
       logger.info('jobs.scheduler.started', { tasks: (options.tasks ?? registeredTasks()).length });

package/src/worker.ts

@@ -5,6 +5,7 @@
 
 import type { Clock, Ctx } from '@ultimat3/core';
 import {
+  beginWork,
   logger,
   onShutdown,
   recordJob,
@@ -13,6 +14,7 @@ import {
   uuid,
 } from '@ultimat3/core';
 import { nowMs } from './clock';
+import { settleAllBy } from './drain-wait';
 import type { ClaimedJob, JobDriver, QueueStats } from './driver';
 import { DEFAULT_QUEUE, DEFAULT_VISIBILITY_TIMEOUT_MS } from './driver';
 import { ConcurrencyUnenforceableError } from './errors';
@@ -113,8 +115,12 @@ export function createWorker(options: WorkerOptions): Worker {
   const rounds = new Set<Promise<unknown>>();
   let state: WorkerStats['state'] = 'idle';
   let loop: ReturnType<typeof setTimeout> | undefined;
-  /** The `onShutdown` registration this worker holds while it runs. Handed back by `stop()`. */
-  let releaseShutdownHook: (() => void) | undefined;
+  /**
+   * The two `onShutdown` registrations this worker holds while it runs, both handed back by
+   * `stop()`. Two, because they answer different questions in different PHASES — the split
+   * `listenSyncNode` and `@ultimat3/http` already have.
+   */
+  let releaseShutdownHooks: (() => void)[] = [];
   /** The teardown in flight, so a second `stop()` joins it instead of running a second one. */
   let stopping: Promise<void> | undefined;
   let processed = 0;
@@ -262,6 +268,10 @@ export function createWorker(options: WorkerOptions): Worker {
           continue;
         }
 
+        // The claimed job is the process's in-flight work, counted where the DRAIN can see it:
+        // core's own in-flight wait sits between `accept` and `inflight` and exists for exactly
+        // this. Counted nowhere, the worker had to wait for its own jobs inside a hook.
+        const finishWork = beginWork();
         const running = runClaimed(job)
           .then((execution) => {
             if (execution.outcome === 'completed') processed += 1;
@@ -276,9 +286,20 @@ export function createWorker(options: WorkerOptions): Worker {
             if (label !== null) recordJob(queue, label);
             return execution;
           })
-          .finally(() => {
+          .finally(async () => {
             lease.release();
-            void fleetSlots.release(job.id);
+            // AWAITED, never `void`: the slot is a row in `x_job_leases`, so the DELETE was still
+            // on the wire when the teardown's `allSettled` returned and `driver.close()` took the
+            // connection out from under it — a `concurrency: 1` job unclaimable by the pod
+            // replacing this one for a whole visibility window after every deploy. `release`
+            // swallows its own failures, so awaiting it cannot reject a job that finished; the
+            // `finally` is for the one that could, because a lost `finishWork()` is an in-flight
+            // count that never returns to zero and a drain that waits out its whole budget.
+            try {
+              await fleetSlots.release(job.id);
+            } finally {
+              finishWork();
+            }
           });
 
         started.push(running);
@@ -348,18 +369,42 @@ export function createWorker(options: WorkerOptions): Worker {
     }, pollIntervalMs);
   };
 
-  const teardown = async (reason: string): Promise<void> => {
+  /**
+   * The whole of the `accept` phase: stop taking work, and nothing else. Synchronous on purpose —
+   * a phase whose job is to be over before the load balancer's next health check must not contain
+   * a wait, and the hook behind this one is somebody else's "stop listening".
+   */
+  const stopAccepting = (): void => {
+    if (state === 'stopped') return;
     state = 'draining';
     if (loop !== undefined) clearTimeout(loop);
     loop = undefined;
+  };
+
+  const teardown = async (reason: string, deadlineAt?: number): Promise<void> => {
+    stopAccepting();
     logger.info('jobs.worker.draining', { workerId, reason, inFlight: inFlight.size });
     try {
       // Stop claiming, finish what we hold, then close. Anything else re-runs work on deploy.
       // Rounds first: one that passed the guard before the flag flipped is still awaiting its
       // `claim()`, and the jobs it starts join `inFlight` after any snapshot taken here — so a
       // drain that waited on `inFlight` alone closed the driver under a job that had just begun.
-      await Promise.allSettled([...rounds]);
-      await Promise.allSettled([...inFlight]);
+      //
+      // Both waits share ONE deadline on the SIGTERM path (`undefined` on a manual stop, which
+      // waits as long as its jobs take). Nothing can kill a body that ignores `ctx.signal`, so an
+      // unbounded wait here is a teardown that never ends: driver never closed, state never past
+      // 'draining', and the memoized `stopping` every later `stop()` joins never settling.
+      // Abandoning costs a lapsed lease and a redelivered job — at-least-once, as promised.
+      const rounded = await settleAllBy([...rounds], deadlineAt);
+      const drained = (await settleAllBy([...inFlight], deadlineAt)) && rounded;
+      if (!drained) {
+        logger.warn('jobs.worker.drain-abandoned', {
+          workerId,
+          reason,
+          inFlight: inFlight.size,
+          fix: 'raise the drain budget past the slowest job — configureLifecycle({ deadlineMs: 600_000 }) — and set terminationGracePeriodSeconds to at least as many seconds',
+        });
+      }
       await options.driver.close?.();
     } finally {
       // Whatever the close did, this worker is done: a state left at 'draining' is a drain that
@@ -369,19 +414,22 @@ export function createWorker(options: WorkerOptions): Worker {
       // through a driver already closed — and keeps this closure, its driver and its in-flight
       // set alive with it.
       state = 'stopped';
-      releaseShutdownHook?.();
-      releaseShutdownHook = undefined;
+      for (const release of releaseShutdownHooks) release();
+      releaseShutdownHooks = [];
     }
   };
 
-  const stop = async (reason = 'stop'): Promise<void> => {
+  const stop = async (reason = 'stop', deadlineAt?: number): Promise<void> => {
+    // Answered immediately once this worker is done: the teardown always REACHES 'stopped' (its
+    // waits are bounded and the state is set in a `finally`), so a caller landing after an
+    // abandoned drain gets an answer rather than joining a promise that never settles.
     if (state === 'stopped') return;
     // One teardown, joined rather than repeated: a SIGTERM landing on a manual stop must wait out
     // the same in-flight work, not close the driver a second time underneath it. Cleared as it
     // settles, so a worker that started again tears down again instead of joining a promise that
     // settled a lifetime ago. A close that threw still stopped this worker — the failure is the
     // caller's to see on the promise it awaited, not a teardown to run twice.
-    stopping ??= teardown(reason).finally(() => {
+    stopping ??= teardown(reason, deadlineAt).finally(() => {
       stopping = undefined;
     });
     await stopping;
@@ -407,13 +455,22 @@ export function createWorker(options: WorkerOptions): Worker {
       }
       state = 'running';
       logger.info('jobs.worker.started', { workerId, queues });
-      // 'accept' phase: stop claiming before core waits on in-flight jobs. The unregister is
-      // kept, never discarded: `stop()` hands it back, so start -> stop -> start holds ONE hook
-      // rather than one per start, each retaining the driver of a worker that is already gone.
+      // TWO hooks, for the two phases that answer two questions. `accept` stops claiming and
+      // returns, so every hook behind it — the HTTP server's "stop listening", the sync node's
+      // "stop upgrading" — runs while the budget is still whole; one hook doing both spent all of
+      // it in the phase whose whole purpose is to be quick. `close` waits out what this worker
+      // holds and closes the driver, bounded by the deadline the hook is handed.
+      //
+      // Both unregisters are kept, never discarded: `stop()` hands them back, so
+      // start -> stop -> start holds one pair rather than one per start, each retaining the
+      // driver of a worker that is already gone.
       if (options.drainOnShutdown !== false) {
-        releaseShutdownHook = onShutdown(`jobs.worker.${workerId}`, () => stop('SIGTERM'), {
-          phase: 'accept',
-        });
+        releaseShutdownHooks = [
+          onShutdown(`jobs.worker.${workerId}.accept`, stopAccepting, { phase: 'accept' }),
+          onShutdown(`jobs.worker.${workerId}`, (reason) => stop('SIGTERM', reason.deadlineAt), {
+            phase: 'close',
+          }),
+        ];
       }
       schedule();
     },