@ultimat3/jobs 19.3.1 → 19.3.3

package/CLAUDE.md

@@ -293,6 +293,31 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   promises; `jobs.worker.drain-abandoned` names it, with the `configureLifecycle({ deadlineMs })`
   raise as its fix. **A worker always REACHES `'stopped'`**, which is what makes `stop()`'s
   `state === 'stopped'` early return an answer rather than a wedge.
+- **SIGTERM reaches the job: the `accept` hook aborts every held run's `ctx.signal`**
+  (`As of 2026-09-07`). The worker holds ONE `AbortController` (`drainSignal`), composed into every
+  run by `worker-run.ts` as the third source beside the caller's signal and the heartbeat's;
+  `stopAccepting(reason)` aborts it with a `JobDrainedError` — core's `X_DRAINING`, naming the
+  worker and the signal — and a manual `stop()` passes no reason and aborts nothing, for the same
+  line `settleAllBy` draws: a caller that asked wants its work finished. It has to be the `accept`
+  hook and not the teardown, because core runs `accept`, then waits out in-flight work (every
+  claimed job is `beginWork()`ed) under the same budget, then `close`: told in `close`, a body
+  would hear it after the in-flight wait had already spent the whole budget on it. Which is what
+  happened until this landed — the drain told nobody, so a body reading `ctx.signal` (the one
+  documented way to stop early) ran to the deadline and was abandoned there, indistinguishable
+  from one that ignores the signal; ai-maxxing measured it as every Ctrl-C paying the full 25s
+  and wrote a process-wide signal of its own to get around it. The controller is replaced with a
+  fresh one in the teardown's `finally`: aborted once stays aborted, and a restarted worker would
+  otherwise hand every job it claimed a signal born cancelled.
+  `executeJob` reads the CODE off the run signal's reason (`drainedBy`) and settles a drained
+  attempt as **`interrupted`** — `nack` with `countsAsAttempt: false`, no park, no dead letter,
+  `delayMs: 0`, the error still recorded on the row — read off the SIGNAL and not the thrown
+  error, because what a body stops WITH is not always the reason (a killed ssh child surfaces as
+  the app's own coded error) and an attempt burned per deploy is the "always twice" draining
+  exists to prevent, one layer down. Not a `retried`: `attempts: 1` would dead-letter a job the
+  process, not the job, cut short, and `jobs_total{outcome="failed"}` would spike on every
+  rollout. The first reason on a controller wins, so a timeout that fired before the drain still
+  reports as a timeout. `WorkerStats.interrupted` counts them; `jobs.worker.drain-signalled` and
+  `jobs.attempt.interrupted` are the two lines. `worker-drain-signal.test.ts` holds all of it.
 - **One teardown, joined.** `stop()` shares the in-flight teardown promise, so a SIGTERM landing
   on a manual stop waits out the same in-flight jobs instead of closing the driver underneath
   it. The promise is cleared as it settles, so a worker that started again tears down again
@@ -937,7 +962,8 @@ picture from the other side.
 | `execute.ts` | `executeJob` — one claimed job run and settled, and the run's deadline/cancel |
 | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
 | `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
-| `worker.ts` | `worker` role, claim loop, drain |
+| `worker.ts` | `worker` role, claim loop, drain — and the one `AbortController` SIGTERM reaches every held run through |
+| `worker-types.ts` | the worker's public contract: `WorkerOptions`, `WorkerStats`, `Worker` |
 | `drain-wait.ts` | the drain's wait, shared by both roles: everything a teardown holds, settled — or abandoned at the budget the `close` hook was handed |
 | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
 | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |

package/README.md

@@ -550,6 +550,24 @@ Nothing can kill a body that ignores the signal, so the durable state is fenced:
 cancel every step write is refused with `X_ABORTED`, and a run that finishes anyway is logged
 as `jobs.timeout.abandoned` — the one way to find a handler that never reads `ctx.signal`.
 
+**SIGTERM fires the same signal** (`As of 2026-09-07`). The moment the drain begins — its `accept`
+phase, before the in-flight wait starts spending the budget — the worker aborts `ctx.signal` on
+every job it holds with `X_DRAINING`, naming the worker and the signal. A body that unwinds is
+**interrupted**, not failed: the job goes straight back to the ready bucket with the attempt
+uncounted, so `attempts: 1` survives a deploy and the worker replacing this one claims it at
+once. Whatever the body stopped with — the reason back from `fetch`, `throwIfAborted`'s
+`X_ABORTED`, an app's own error for a child the shutdown killed — the verdict is read off the
+signal, not the error. A body that ignores the signal is waited on to the deadline and abandoned
+there, as before. A manual `stop()` aborts nothing: it waits for the work it holds.
+
+| The attempt ended by | `JobOutcome` | attempt counted | `jobs_total` |
+|---|---|---|---|
+| the body returning | `completed` | — | `ok` |
+| `step.sleep` / `step.waitForEvent` | `suspended` | no | not counted |
+| the body throwing, attempts left | `retried` | yes | `failed` |
+| the body throwing, none left or `terminal` | `dead-lettered` | yes | `dead` |
+| the worker's drain (`X_DRAINING` on `ctx.signal`) | `interrupted` | **no** | not counted |
+
 Three ceilings, declared on the job and nowhere else (`As of 2026-08` — `stepTimeout` and
 `eventPoll` had been implemented in the step runner since 1.0 with no declaration able to reach
 them, so no `job()` could ask for either):
@@ -603,8 +621,9 @@ things have to be true in a process:
 | the facade is installed | `setJobsFacade(createJobsFacade({ store, driver }, currentTx))` |
 | the relay is running | `createOutboxRelay({ store, driver }).start()` |
 
-`start()` registers the same two shutdown hooks `createWorker` does — `accept` stops polling,
-`close` waits out the pass in flight under the drain's deadline — and `stop()` hands both back.
+`start()` registers the same two shutdown hooks `createWorker` does — `accept` stops polling
+(the worker's also aborts every held job's `ctx.signal`), `close` waits out the pass in flight
+under the drain's deadline — and `stop()` hands both back.
 `drainOnShutdown: false` opts out, for a caller that drives its own teardown.
 
 with `store = createPgOutboxStore({ executor, txExecutor })`. `txExecutor` is what makes it

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "19.3.1",
+  "version": "19.3.3",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,10 +32,10 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "19.3.1",
-    "@ultimat3/db": "19.3.1",
-    "@ultimat3/entity": "19.3.1",
-    "@ultimat3/schema": "19.3.1",
-    "@ultimat3/time": "19.3.1"
+    "@ultimat3/core": "19.3.3",
+    "@ultimat3/db": "19.3.3",
+    "@ultimat3/entity": "19.3.3",
+    "@ultimat3/schema": "19.3.3",
+    "@ultimat3/time": "19.3.3"
   }
 }

package/src/drain-wait.ts

@@ -6,45 +6,95 @@
 import { systemClock } from '@ultimat3/core';
 
 /**
- * Everything in `pending`, settled — or abandoned once `deadlineAt` (real monotonic ms, the clock
- * `ShutdownReason.deadlineAt` is measured on) has passed. Answers `true` when everything settled.
+ * The deadline a teardown waits under, and it is bound LATE. `undefined` while the stop is manual;
+ * a real monotonic instant (the clock `ShutdownReason.deadlineAt` is measured on) the moment a
+ * shutdown lands — before the teardown starts, or in the middle of it. A wait already in progress
+ * adopts it, which is the case a plain number could not express: `worker.stop('deploy')` starts a
+ * teardown with no budget, SIGTERM arrives, core's `close` hook JOINS the memoised teardown — and
+ * the number that teardown was started with is the number it kept. Core abandoned the hook at the
+ * deadline and moved on; the worker sat on a body that ignores `ctx.signal` with its driver open
+ * and `stopping` never settling, exactly the wedge the bound was written to end.
  *
- * `deadlineAt` is `undefined` for a MANUAL `stop()`, which waits as long as its work takes: a
- * caller that asked a role to stop has no budget to spend, and closing the queue under a live job
- * — or handing the lease back under a live dispatch — is exactly what draining exists to prevent. The bound belongs to the SIGTERM path, where the
- * budget is real and a handler that ignores `ctx.signal` would otherwise hold the teardown — and
- * with it the memoized `stopping` promise every later `stop()` joins — open forever.
+ * The EARLIEST deadline wins: a second bind can only tighten, never extend, because the budget it
+ * comes from is one process-wide grace period and not a per-caller allowance.
+ */
+export interface DrainBudget {
+  /** The deadline in force — `undefined` until a shutdown binds one. */
+  readonly deadlineAt: number | undefined;
+  /** Bind a deadline, or tighten the one held. Every wait in progress hears it at once. */
+  bind(deadlineAt: number): void;
+  /**
+   * Hear the deadline: now, when one is already bound, and again each time it tightens. Answers
+   * the unsubscribe, which a settled wait calls so a budget outlives none of its waiters.
+   */
+  watch(listener: (deadlineAt: number) => void): () => void;
+}
+
+export function createDrainBudget(deadlineAt?: number): DrainBudget {
+  let bound = deadlineAt;
+  const listeners = new Set<(deadlineAt: number) => void>();
+  return {
+    get deadlineAt() {
+      return bound;
+    },
+    bind(at) {
+      if (bound !== undefined && at >= bound) return;
+      bound = at;
+      for (const listener of listeners) listener(at);
+    },
+    watch(listener) {
+      listeners.add(listener);
+      if (bound !== undefined) listener(bound);
+      return () => {
+        listeners.delete(listener);
+      };
+    },
+  };
+}
+
+/**
+ * Everything in `pending`, settled — or abandoned once the deadline (real monotonic ms) has
+ * passed. Answers `true` when everything settled.
+ *
+ * The deadline is a number for a role whose teardown cannot be joined mid-flight, and a
+ * `DrainBudget` for one that can — the worker, whose manual `stop()` a later SIGTERM joins.
+ * `undefined` is a MANUAL `stop()`, which waits as long as its work takes: a caller that asked a
+ * role to stop has no budget to spend, and closing the queue under a live job — or handing the
+ * lease back under a live dispatch — is exactly what draining exists to prevent. The bound belongs
+ * to the SIGTERM path, where the budget is real and a handler that ignores `ctx.signal` would
+ * otherwise hold the teardown — and with it the memoized `stopping` promise every later `stop()`
+ * joins — open forever.
  *
  * `allSettled`, so work that rejected is work that finished: each caller observes its own failures
  * already, and a teardown that rethrew here would skip the close behind it.
  */
 export async function settleAllBy(
   pending: readonly Promise<unknown>[],
-  deadlineAt: number | undefined,
+  deadline: DrainBudget | number | undefined,
 ): Promise<boolean> {
   if (pending.length === 0) return true;
   const settled = Promise.allSettled(pending);
-  if (deadlineAt === undefined) {
-    await settled;
-    return true;
-  }
-  const remainingMs = Math.max(0, deadlineAt - systemClock.monotonic());
+  const budget = typeof deadline === 'object' ? deadline : createDrainBudget(deadline);
   return await new Promise<boolean>((resolve) => {
     let decided = false;
-    const timer = setTimeout(() => {
-      if (decided) return;
-      decided = true;
-      resolve(false);
-    }, remainingMs);
-    // Never the thing keeping a drained process alive — the rule `lifecycle-deadline.ts` states
-    // for its own timer. A spent budget still gives the already-settled case its turn, because a
-    // resolved promise settles on a microtask and this timer on a macrotask.
-    timer.unref?.();
-    void settled.then(() => {
+    let timer: ReturnType<typeof setTimeout> | undefined;
+    const decide = (drained: boolean): void => {
       if (decided) return;
       decided = true;
-      clearTimeout(timer);
-      resolve(true);
+      unwatch();
+      if (timer !== undefined) clearTimeout(timer);
+      resolve(drained);
+    };
+    // Re-armed on every tightening, so a wait that began with no deadline ends at the one a later
+    // shutdown bound. Never the thing keeping a drained process alive — the rule
+    // `lifecycle-deadline.ts` states for its own timer. A spent budget still gives the
+    // already-settled case its turn, because a resolved promise settles on a microtask and this
+    // timer on a macrotask.
+    const unwatch = budget.watch((deadlineAt) => {
+      if (timer !== undefined) clearTimeout(timer);
+      timer = setTimeout(() => decide(false), Math.max(0, deadlineAt - systemClock.monotonic()));
+      timer.unref?.();
     });
+    void settled.then(() => decide(true));
   });
 }

package/src/errors.ts

@@ -39,13 +39,15 @@ export const JOB_OWNED_ERROR_CODES = [
 ] as const;
 
 /**
- * `X_NOT_IMPLEMENTED` and `X_ABORTED` are `@ultimat3/core`'s. `JobsNotImplementedError` and
- * `JobAbortedError` below throw them; jobs keeps no title for either, because the copy this file
- * used to hold was a second title that nothing would have failed on once core's changed.
+ * `X_NOT_IMPLEMENTED`, `X_ABORTED` and `X_DRAINING` are `@ultimat3/core`'s. `JobsNotImplementedError`,
+ * `JobAbortedError` and `JobDrainedError` below throw them; jobs keeps no title for any of the
+ * three, because the copy this file used to hold was a second title that nothing would have failed
+ * on once core's changed. Listed here all the same, so `JobErrorCode` can name every code a job
+ * can see — `X_DRAINING` was thrown for a day before it was, and the type said it could not be.
  */
-export const JOB_BORROWED_ERROR_CODES = ['X_NOT_IMPLEMENTED', 'X_ABORTED'] as const;
+export const JOB_BORROWED_ERROR_CODES = ['X_NOT_IMPLEMENTED', 'X_ABORTED', 'X_DRAINING'] as const;
 
-/** Every code jobs can throw: the ones it owns plus the one it borrows. */
+/** Every code jobs can throw: the ones it owns plus the ones it borrows. */
 export const JOB_ERROR_CODES = [...JOB_OWNED_ERROR_CODES, ...JOB_BORROWED_ERROR_CODES] as const;
 
 export type JobOwnedErrorCode = (typeof JOB_OWNED_ERROR_CODES)[number];
@@ -271,6 +273,27 @@ export class JobAbortedError extends UltimateError {
   }
 }
 
+/**
+ * The worker holding this attempt received SIGTERM. Handed to the run as `ctx.signal`'s reason
+ * the moment the drain's `accept` phase runs — before core's in-flight wait, not at the deadline
+ * that ends it — so a body reading the one cancellation seam learns the process is going away
+ * while there is still budget to unwind in.
+ *
+ * Core's `X_DRAINING` rather than a code of jobs' own, for `JobAbortedError`'s reason: the
+ * framework already means exactly one thing by "the process is draining", it is already
+ * classified `retryable`, and `executeJob` reads the CODE off the run signal's reason to tell a
+ * drained attempt from a timed-out or lease-lost one — the drained one is handed back uncounted.
+ */
+export class JobDrainedError extends UltimateError {
+  constructor(input: { workerId: string; signal: string }) {
+    super({
+      code: 'X_DRAINING',
+      cause: `worker "${input.workerId}" is draining (${input.signal}) — this attempt is cut short and the job handed back to the queue with the attempt uncounted`,
+      fix: 'nothing on the job: another worker claims it. To unwind inside the drain budget instead of being killed at it, pass ctx.signal to every outbound call — fetch(url, { signal: ctx.signal }) — and call throwIfAborted(ctx) between steps',
+    });
+  }
+}
+
 /** Retries exhausted. The job is in the dead-letter queue, not lost. */
 export class JobMaxAttemptsError extends UltimateError {
   constructor(input: { job: string; jobId: string; attempts: number; lastError: string }) {

package/src/execute.ts

@@ -28,7 +28,13 @@ import type { EventLookup, StepRecord } from './steps';
 import { createStepRunner, isStepSuspension } from './steps';
 import { jobRunActor } from './tenant';
 
-export type JobOutcome = 'completed' | 'suspended' | 'retried' | 'dead-lettered';
+/**
+ * How one attempt ended. `interrupted` is the worker's drain cutting the attempt short: the job is
+ * back in the ready bucket with the attempt UNCOUNTED, because the process ended it and not the
+ * job — filed as `retried`, a deploy would burn an attempt per job it held, and with
+ * `attempts: 1` dead-letter it.
+ */
+export type JobOutcome = 'completed' | 'suspended' | 'retried' | 'dead-lettered' | 'interrupted';
 
 /** Stands in for a caller with nothing to cancel, so the composition below has one shape. */
 const NEVER_ABORTED = new AbortController().signal;
@@ -188,6 +194,33 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
     }
 
     const message = renderThrowable(error);
+    if (drainedBy(signal)) {
+      // The worker's drain told this body to stop, and it did. Whatever it stopped WITH is the
+      // framework's doing — the reason itself back from `fetch`, `throwIfAborted`'s `X_ABORTED`,
+      // a fenced step write, or an app's own coded error for a child the shutdown killed — so
+      // the attempt is handed back rather than failed: `countsAsAttempt: false`, no park, no
+      // dead letter, claimable at once by the worker replacing this one. Read off the SIGNAL and
+      // not the error, because the body's error is not always the signal's reason, and an
+      // attempt burned per deploy is the "always twice" draining exists to prevent. The `error`
+      // is still recorded on the row: `x jobs show` should say why the last attempt ended.
+      await driver.nack(claimed.id, { delayMs: 0, error: message, countsAsAttempt: false });
+      logger.info('jobs.attempt.interrupted', {
+        job: handle.name,
+        jobId: claimed.id,
+        attempt: claimed.attempt,
+        error: message,
+      });
+      return settle({
+        outcome: 'interrupted',
+        jobId: claimed.id,
+        job: handle.name,
+        attempt: claimed.attempt,
+        durationMs: nowMs(options.clock) - startedAt,
+        error: message,
+        steps: [],
+        replayed: [],
+      });
+    }
     // The ERROR decides too, not only the attempt count. A `terminal` code — a rotated password,
     // a schema mismatch, a permission denial — fails identically on every remaining attempt, so
     // spending them is a queue slot, a provider bill and, at a site that locks an account after
@@ -317,6 +350,15 @@ function raceTimeout(
   });
 }
 
+/**
+ * The run was cancelled by the worker's drain: the code `JobDrainedError` carries, read off the
+ * signal. The FIRST reason wins on a controller, so a timeout that fired before the drain still
+ * reports as a timeout — the drain only claims an attempt it ended.
+ */
+function drainedBy(signal: AbortSignal): boolean {
+  return signal.aborted && isUltimateError(signal.reason) && signal.reason.code === 'X_DRAINING';
+}
+
 /** The body stopped because we cancelled it: our own reason back, or a fenced step write. */
 function isCancellation(error: unknown, reason: unknown): boolean {
   return error === reason || (isUltimateError(error) && error.code === 'X_ABORTED');

package/src/index.ts

@@ -159,6 +159,7 @@ export {
   JOB_ERROR_CODES,
   JOB_ERROR_TITLES,
   JobAbortedError,
+  JobDrainedError,
   JobDuplicateError,
   JobMaxAttemptsError,
   JobNameTakenError,

package/src/metrics.ts

@@ -14,6 +14,24 @@
 
 import type { Gauge } from '@ultimat3/core';
 import { gauge } from '@ultimat3/core';
+import type { JobOutcome } from './execute';
+
+/**
+ * `JobOutcome` -> the `jobs_total` label, and `null` for the outcomes that are not one. `suspended`
+ * is deliberately unmapped: parking a run is control flow, so counting it would make every
+ * `step.sleep` read as a finished job and make the failure ratio meaningless. `interrupted` for
+ * the same reason: a deploy cutting a job short is the process's doing, and a failure ratio that
+ * spikes on every rollout is a page nobody answers. Read by the worker's one `recordJob` site.
+ */
+export const JOB_OUTCOME_LABELS = Object.freeze<
+  Record<JobOutcome, 'ok' | 'failed' | 'dead' | null>
+>({
+  completed: 'ok',
+  suspended: null,
+  retried: 'failed',
+  'dead-lettered': 'dead',
+  interrupted: null,
+});
 
 /** Seconds and not milliseconds: every Prometheus duration is seconds, and the alert is `> 300`. */
 export const queueOldestReady: Gauge = gauge('queue_oldest_ready_seconds', {

package/src/worker-run.ts

@@ -27,6 +27,12 @@ export interface RunClaimedOptions {
   readonly heartbeatIntervalMs: number;
   readonly clock?: Clock;
   readonly events?: EventLookup;
+  /**
+   * The worker's drain, composed into every run it starts: aborted with a `JobDrainedError` when
+   * the process is going away, so the body hears it on `ctx.signal` — the one seam it already
+   * reads — before core's in-flight wait starts spending the budget on it.
+   */
+  readonly drain?: AbortSignal;
 }
 
 /** A name this deploy does not know, parked rather than failed — almost always a deploy skew. */
@@ -95,8 +101,10 @@ export async function runClaimedJob(options: RunClaimedOptions): Promise<JobExec
     // controller this worker owns rather than `AbortSignal.any`, for two reasons: it is handed BACK
     // when the run settles (an app whose `context()` carries a process-lifetime signal was
     // accumulating one composite per job), and the worker can abort it itself — which is the only
-    // way a fleet slot taken by somebody else reaches the body running under it.
-    runSignal = createRunSignal([base.signal, heartbeat.signal]);
+    // way a fleet slot taken by somebody else reaches the body running under it. The worker's
+    // drain is the third source: SIGTERM reaches the body through the same signal, carrying the
+    // `X_DRAINING` reason `executeJob` reads to hand the attempt back uncounted.
+    runSignal = createRunSignal([base.signal, heartbeat.signal, options.drain]);
     const signal = runSignal;
     const ctx: Ctx = { ...base, signal: signal.signal };
 

package/src/worker-types.ts

@@ -0,0 +1,56 @@
+// The `worker` role's public contract: what `createWorker` takes, what it hands back, and what
+// `stats()` reports. Apart from `worker.ts` because that file's job is the claim loop and the
+// drain, and a contract three files import should not sit under 500 lines of loop.
+
+import type { Clock, Ctx } from '@ultimat3/core';
+import type { JobDriver, QueueStats } from './driver';
+import type { JobExecution } from './execute';
+import type { Limiter } from './limits';
+import type { EventLookup } from './steps';
+
+export interface WorkerOptions {
+  readonly driver: JobDriver;
+  /** Queues this process serves. Default `['default']`. */
+  readonly queues?: readonly string[];
+  /** Slots per queue. A number applies to every queue. */
+  readonly concurrency?: number | Readonly<Record<string, number>>;
+  readonly limiter?: Limiter;
+  readonly clock?: Clock;
+  readonly events?: EventLookup;
+  /** Supplies the ambient Ctx for a job run; the app wires ALS + tenant here. */
+  readonly context: () => Ctx;
+  readonly visibilityTimeoutMs?: number;
+  readonly pollIntervalMs?: number;
+  readonly heartbeatIntervalMs?: number;
+  readonly workerId?: string;
+  /** Default true. Registers a SIGTERM drain via `onShutdown`. */
+  readonly drainOnShutdown?: boolean;
+}
+
+export interface WorkerStats {
+  readonly workerId: string;
+  readonly queues: readonly string[];
+  readonly state: 'idle' | 'running' | 'draining' | 'stopped';
+  readonly inFlight: number;
+  readonly processed: number;
+  readonly failed: number;
+  readonly suspended: number;
+  readonly deadLettered: number;
+  /** Attempts this worker's drain cut short and handed back uncounted — see `JobDrainedError`. */
+  readonly interrupted: number;
+  readonly queueDepth: readonly QueueStats[];
+}
+
+export interface Worker {
+  start(): void;
+  /** One claim+run round. Returns jobs processed. Tests drive this instead of the timer. */
+  tick(): Promise<readonly JobExecution[]>;
+  /**
+   * Stop claiming, wait for every job this worker holds, close the driver. Unbounded, and it
+   * aborts nothing: a caller that asked wants its work finished. SIGTERM takes the other path —
+   * the shutdown hooks `start()` registers abort every held run's `ctx.signal` and wait under
+   * the lifecycle's deadline.
+   */
+  stop(reason?: string): Promise<void>;
+  stats(): Promise<WorkerStats>;
+}

package/src/worker.ts

@@ -3,7 +3,7 @@
 // bug here — the visibility timeout re-delivers it — but a worker that exits mid-job on EVERY
 // deploy turns "at least once" into "always twice", so draining is on by default.
 
-import type { Clock, Ctx } from '@ultimat3/core';
+import type { ShutdownReason } from '@ultimat3/core';
 import {
   beginWork,
   logger,
@@ -14,19 +14,18 @@ import {
   uuid,
 } from '@ultimat3/core';
 import { nowMs } from './clock';
-import { settleAllBy } from './drain-wait';
-import type { ClaimedJob, JobDriver, QueueStats } from './driver';
+import { createDrainBudget, settleAllBy } from './drain-wait';
+import type { ClaimedJob } from './driver';
 import { DEFAULT_QUEUE } from './driver';
-import { ConcurrencyUnenforceableError } from './errors';
-import type { JobExecution, JobOutcome } from './execute';
+import { ConcurrencyUnenforceableError, JobDrainedError } from './errors';
+import type { JobExecution } from './execute';
 import { getJob, registeredJobs } from './job';
-import type { Limiter } from './limits';
 import { createLimiter } from './limits';
-import { recordQueueDeadJobs, recordQueueOldestReady } from './metrics';
-import type { EventLookup } from './steps';
+import { JOB_OUTCOME_LABELS, recordQueueDeadJobs, recordQueueOldestReady } from './metrics';
 import { createFleetSlots } from './worker-fleet-slots';
 import { resolveWorkerTimings } from './worker-options';
 import { runClaimedJob } from './worker-run';
+import type { Worker, WorkerOptions, WorkerStats } from './worker-types';
 
 /**
  * How often the claim loop republishes `queue_depth`. Its own interval, not `pollIntervalMs`:
@@ -36,56 +35,7 @@ import { runClaimedJob } from './worker-run';
  */
 const QUEUE_DEPTH_INTERVAL_MS = 15_000;
 
-/**
- * `JobOutcome` -> the `jobs_total` label, and `null` for the outcome that is not one. `suspended`
- * is deliberately unmapped: parking a run is control flow, so counting it would make every
- * `step.sleep` read as a finished job and make the failure ratio meaningless.
- */
-const JOB_OUTCOME_LABELS = Object.freeze<Record<JobOutcome, 'ok' | 'failed' | 'dead' | null>>({
-  completed: 'ok',
-  suspended: null,
-  retried: 'failed',
-  'dead-lettered': 'dead',
-});
-
-export interface WorkerOptions {
-  readonly driver: JobDriver;
-  /** Queues this process serves. Default `['default']`. */
-  readonly queues?: readonly string[];
-  /** Slots per queue. A number applies to every queue. */
-  readonly concurrency?: number | Readonly<Record<string, number>>;
-  readonly limiter?: Limiter;
-  readonly clock?: Clock;
-  readonly events?: EventLookup;
-  /** Supplies the ambient Ctx for a job run; the app wires ALS + tenant here. */
-  readonly context: () => Ctx;
-  readonly visibilityTimeoutMs?: number;
-  readonly pollIntervalMs?: number;
-  readonly heartbeatIntervalMs?: number;
-  readonly workerId?: string;
-  /** Default true. Registers a SIGTERM drain via `onShutdown`. */
-  readonly drainOnShutdown?: boolean;
-}
-
-export interface WorkerStats {
-  readonly workerId: string;
-  readonly queues: readonly string[];
-  readonly state: 'idle' | 'running' | 'draining' | 'stopped';
-  readonly inFlight: number;
-  readonly processed: number;
-  readonly failed: number;
-  readonly suspended: number;
-  readonly deadLettered: number;
-  readonly queueDepth: readonly QueueStats[];
-}
-
-export interface Worker {
-  start(): void;
-  /** One claim+run round. Returns jobs processed. Tests drive this instead of the timer. */
-  tick(): Promise<readonly JobExecution[]>;
-  stop(reason?: string): Promise<void>;
-  stats(): Promise<WorkerStats>;
-}
+export type { Worker, WorkerOptions, WorkerStats } from './worker-types';
 
 export function createWorker(options: WorkerOptions): Worker {
   const workerId = options.workerId ?? `worker-${uuid()}`;
@@ -111,6 +61,23 @@ export function createWorker(options: WorkerOptions): Worker {
    * `inFlight` waited on a set the round it was racing had not finished filling.
    */
   const rounds = new Set<Promise<unknown>>();
+  /**
+   * The drain, as every run this worker starts hears it: composed into each run's `ctx.signal`
+   * (`worker-run.ts`), aborted by the `accept` hook with a `JobDrainedError`. ONE controller and
+   * not one per run, because the fact it carries — "this process is going away" — is one fact.
+   * Replaced with a fresh one when a teardown ends: a controller aborted once stays aborted, and
+   * a restarted worker would otherwise hand every job it claimed a signal born cancelled.
+   */
+  let drainSignal = new AbortController();
+  /**
+   * The deadline the teardown waits under. `undefined` for a manual `stop()`, bound the moment a
+   * shutdown lands — and bound LATE when that shutdown lands on a teardown already in flight: the
+   * `close` hook joins the memoised `stopping` rather than starting a second, and until 2026-09-07
+   * the teardown it joined kept the `undefined` it was started with. Core abandoned the hook at
+   * the deadline; the worker sat on a body ignoring `ctx.signal` with its driver open. Fresh per
+   * teardown, for the controller's reason: a restarted worker's manual stop is unbounded again.
+   */
+  let budget = createDrainBudget();
   let state: WorkerStats['state'] = 'idle';
   let loop: ReturnType<typeof setTimeout> | undefined;
   /**
@@ -125,6 +92,7 @@ export function createWorker(options: WorkerOptions): Worker {
   let failed = 0;
   let suspended = 0;
   let deadLettered = 0;
+  let interrupted = 0;
   let depthPublishedAt = Number.NEGATIVE_INFINITY;
 
   /**
@@ -167,6 +135,7 @@ export function createWorker(options: WorkerOptions): Worker {
       workerId,
       visibilityTimeoutMs,
       heartbeatIntervalMs,
+      drain: drainSignal.signal,
       ...(options.clock === undefined ? {} : { clock: options.clock }),
       ...(options.events === undefined ? {} : { events: options.events }),
     });
@@ -275,6 +244,7 @@ export function createWorker(options: WorkerOptions): Worker {
             if (execution.outcome === 'completed') processed += 1;
             else if (execution.outcome === 'suspended') suspended += 1;
             else if (execution.outcome === 'retried') failed += 1;
+            else if (execution.outcome === 'interrupted') interrupted += 1;
             else deadLettered += 1;
             // The other half of this package's metrics contract: `queue_depth` says how much work
             // is waiting, `jobs_total` says whether any of it is succeeding. Depth alone cannot
@@ -368,19 +338,39 @@ export function createWorker(options: WorkerOptions): Worker {
   };
 
   /**
-   * The whole of the `accept` phase: stop taking work, and nothing else. Synchronous on purpose —
-   * a phase whose job is to be over before the load balancer's next health check must not contain
-   * a wait, and the hook behind this one is somebody else's "stop listening".
+   * The whole of the `accept` phase: stop taking work, tell the work already held, and nothing
+   * else. Synchronous on purpose — a phase whose job is to be over before the load balancer's next
+   * health check must not contain a wait, and the hook behind this one is somebody else's "stop
+   * listening". An abort is synchronous and costs nothing, which is why it belongs HERE and not in
+   * the teardown: core runs `accept`, then waits out in-flight work (every claimed job is
+   * `beginWork()`ed) under the same budget, then `close`. Told in `close`, a body that reads
+   * `ctx.signal` would hear it after the in-flight wait had already spent the whole budget on it —
+   * which is what happened until 2026-09-07: a job that would have stopped in a second was waited
+   * on for the full deadline and abandoned there, exactly like one that ignores the signal.
+   *
+   * Only a SHUTDOWN aborts, and only a shutdown binds the budget. A manual `stop()` passes
+   * nothing: a caller that asked has no budget to spend and wants its work finished, the same
+   * line `settleAllBy` draws for the wait. The bind comes BEFORE the abort's once-guard and on
+   * every call, because the second shutdown to reach a worker is the one that finds the abort
+   * already fired and the teardown already waiting — with no deadline, if the first was manual.
    */
-  const stopAccepting = (): void => {
+  const stopAccepting = (shutdown?: ShutdownReason): void => {
     if (state === 'stopped') return;
     state = 'draining';
     if (loop !== undefined) clearTimeout(loop);
     loop = undefined;
+    if (shutdown === undefined) return;
+    budget.bind(shutdown.deadlineAt);
+    if (drainSignal.signal.aborted) return;
+    logger.info('jobs.worker.drain-signalled', {
+      workerId,
+      signal: shutdown.signal,
+      inFlight: inFlight.size,
+    });
+    drainSignal.abort(new JobDrainedError({ workerId, signal: shutdown.signal }));
   };
 
-  const teardown = async (reason: string, deadlineAt?: number): Promise<void> => {
-    stopAccepting();
+  const teardown = async (reason: string): Promise<void> => {
     logger.info('jobs.worker.draining', { workerId, reason, inFlight: inFlight.size });
     try {
       // Stop claiming, finish what we hold, then close. Anything else re-runs work on deploy.
@@ -388,13 +378,14 @@ export function createWorker(options: WorkerOptions): Worker {
       // `claim()`, and the jobs it starts join `inFlight` after any snapshot taken here — so a
       // drain that waited on `inFlight` alone closed the driver under a job that had just begun.
       //
-      // Both waits share ONE deadline on the SIGTERM path (`undefined` on a manual stop, which
-      // waits as long as its jobs take). Nothing can kill a body that ignores `ctx.signal`, so an
-      // unbounded wait here is a teardown that never ends: driver never closed, state never past
-      // 'draining', and the memoized `stopping` every later `stop()` joins never settling.
-      // Abandoning costs a lapsed lease and a redelivered job — at-least-once, as promised.
-      const rounded = await settleAllBy([...rounds], deadlineAt);
-      const drained = (await settleAllBy([...inFlight], deadlineAt)) && rounded;
+      // Both waits share ONE budget: unbound on a manual stop, which waits as long as its jobs
+      // take, and bound by the SIGTERM — whether it arrived before this teardown or lands in the
+      // middle of it. Nothing can kill a body that ignores `ctx.signal`, so an unbounded wait
+      // here is a teardown that never ends: driver never closed, state never past 'draining', and
+      // the memoized `stopping` every later `stop()` joins never settling. Abandoning costs a
+      // lapsed lease and a redelivered job — at-least-once, as promised.
+      const rounded = await settleAllBy([...rounds], budget);
+      const drained = (await settleAllBy([...inFlight], budget)) && rounded;
       if (!drained) {
         logger.warn('jobs.worker.drain-abandoned', {
           workerId,
@@ -414,20 +405,28 @@ export function createWorker(options: WorkerOptions): Worker {
       state = 'stopped';
       for (const release of releaseShutdownHooks) release();
       releaseShutdownHooks = [];
+      // A run this drain abandoned still follows the old controller through its own composition;
+      // the next start's jobs must not. Fresh here, in the one place a teardown always reaches —
+      // and the budget with it, or the next manual stop would inherit a deadline already spent.
+      drainSignal = new AbortController();
+      budget = createDrainBudget();
     }
   };
 
-  const stop = async (reason = 'stop', deadlineAt?: number): Promise<void> => {
+  const stop = async (reason = 'stop', shutdown?: ShutdownReason): Promise<void> => {
     // Answered immediately once this worker is done: the teardown always REACHES 'stopped' (its
     // waits are bounded and the state is set in a `finally`), so a caller landing after an
     // abandoned drain gets an answer rather than joining a promise that never settles.
     if (state === 'stopped') return;
-    // One teardown, joined rather than repeated: a SIGTERM landing on a manual stop must wait out
-    // the same in-flight work, not close the driver a second time underneath it. Cleared as it
-    // settles, so a worker that started again tears down again instead of joining a promise that
-    // settled a lifetime ago. A close that threw still stopped this worker — the failure is the
-    // caller's to see on the promise it awaited, not a teardown to run twice.
-    stopping ??= teardown(reason, deadlineAt).finally(() => {
+    // Before the join, every time: a SIGTERM landing on a manual stop still aborts every held
+    // run's `ctx.signal` and binds the teardown already waiting to the shutdown's deadline.
+    stopAccepting(shutdown);
+    // One teardown, joined rather than repeated: that SIGTERM must wait out the same in-flight
+    // work, not close the driver a second time underneath it. Cleared as it settles, so a worker
+    // that started again tears down again instead of joining a promise that settled a lifetime
+    // ago. A close that threw still stopped this worker — the failure is the caller's to see on
+    // the promise it awaited, not a teardown to run twice.
+    stopping ??= teardown(reason).finally(() => {
       stopping = undefined;
     });
     await stopping;
@@ -453,19 +452,22 @@ export function createWorker(options: WorkerOptions): Worker {
       }
       state = 'running';
       logger.info('jobs.worker.started', { workerId, queues });
-      // TWO hooks, for the two phases that answer two questions. `accept` stops claiming and
-      // returns, so every hook behind it — the HTTP server's "stop listening", the sync node's
-      // "stop upgrading" — runs while the budget is still whole; one hook doing both spent all of
-      // it in the phase whose whole purpose is to be quick. `close` waits out what this worker
-      // holds and closes the driver, bounded by the deadline the hook is handed.
+      // TWO hooks, for the two phases that answer two questions. `accept` stops claiming, aborts
+      // every held run's `ctx.signal` and returns, so every hook behind it — the HTTP server's
+      // "stop listening", the sync node's "stop upgrading" — runs while the budget is still whole;
+      // one hook doing both spent all of it in the phase whose whole purpose is to be quick.
+      // `close` waits out what this worker holds and closes the driver, bounded by the deadline
+      // the hook is handed.
       //
       // Both unregisters are kept, never discarded: `stop()` hands them back, so
       // start -> stop -> start holds one pair rather than one per start, each retaining the
       // driver of a worker that is already gone.
       if (options.drainOnShutdown !== false) {
         releaseShutdownHooks = [
-          onShutdown(`jobs.worker.${workerId}.accept`, stopAccepting, { phase: 'accept' }),
-          onShutdown(`jobs.worker.${workerId}`, (reason) => stop('SIGTERM', reason.deadlineAt), {
+          onShutdown(`jobs.worker.${workerId}.accept`, (reason) => stopAccepting(reason), {
+            phase: 'accept',
+          }),
+          onShutdown(`jobs.worker.${workerId}`, (reason) => stop(reason.signal, reason), {
             phase: 'close',
           }),
         ];
@@ -484,6 +486,7 @@ export function createWorker(options: WorkerOptions): Worker {
         failed,
         suspended,
         deadLettered,
+        interrupted,
         queueDepth: [...(await options.driver.stats())],
       };
     },