@ultimat3/jobs 23.0.0 → 25.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (70) hide show
  1. package/CLAUDE.md +72 -75
  2. package/README.md +83 -38
  3. package/package.json +7 -7
  4. package/src/backfill-errors.ts +3 -2
  5. package/src/backfill-ledger.ts +25 -3
  6. package/src/backfill-pass.ts +22 -5
  7. package/src/backfill-rate.ts +2 -2
  8. package/src/backfill.ts +2 -2
  9. package/src/claim-exhausted.ts +76 -0
  10. package/src/counters.ts +1 -1
  11. package/src/driver-memory-operator.ts +34 -12
  12. package/src/driver-memory.ts +80 -38
  13. package/src/driver-pg-jobs-sql.ts +27 -19
  14. package/src/driver-pg-operator-sql.ts +32 -12
  15. package/src/driver-pg-operator.ts +19 -3
  16. package/src/driver-pg-rows.ts +3 -3
  17. package/src/driver-pg-settle-sql.ts +2 -3
  18. package/src/driver-pg-sql.ts +99 -30
  19. package/src/driver-pg.ts +103 -98
  20. package/src/driver.ts +46 -16
  21. package/src/errors-concurrency.ts +1 -1
  22. package/src/errors-operator.ts +12 -4
  23. package/src/errors-requeue.ts +21 -3
  24. package/src/errors.ts +9 -17
  25. package/src/events-pg.ts +11 -21
  26. package/src/events.ts +26 -15
  27. package/src/execute.ts +47 -5
  28. package/src/export.ts +2 -2
  29. package/src/heartbeat.ts +7 -2
  30. package/src/index.ts +45 -99
  31. package/src/inspect-operator.ts +4 -4
  32. package/src/inspect.ts +5 -4
  33. package/src/introspection.ts +23 -11
  34. package/src/job.ts +6 -7
  35. package/src/leases.ts +1 -1
  36. package/src/limits.ts +46 -15
  37. package/src/outbox-lease.ts +1 -1
  38. package/src/outbox-pg.ts +6 -7
  39. package/src/outbox-relay.ts +6 -6
  40. package/src/outbox.ts +47 -19
  41. package/src/purge.ts +1 -1
  42. package/src/queue-wake.ts +1 -2
  43. package/src/renewal-timer.ts +1 -1
  44. package/src/retry-classification.ts +52 -15
  45. package/src/scheduler-leader.ts +2 -2
  46. package/src/scheduler-pg.ts +19 -16
  47. package/src/scheduler-state.ts +10 -3
  48. package/src/scheduler.ts +14 -15
  49. package/src/steps-memory.ts +2 -2
  50. package/src/steps-suspension.ts +46 -0
  51. package/src/steps-timeout.ts +33 -0
  52. package/src/steps.ts +90 -88
  53. package/src/task.ts +1 -1
  54. package/src/webhook-attempt.ts +206 -0
  55. package/src/webhook-ledger.ts +15 -4
  56. package/src/webhook.ts +64 -118
  57. package/src/worker-admit.ts +14 -6
  58. package/src/worker-drain-cutoff.ts +110 -0
  59. package/src/worker-fleet-slots.ts +82 -45
  60. package/src/worker-held.ts +100 -0
  61. package/src/worker-key-busy.ts +38 -10
  62. package/src/worker-loop.ts +35 -11
  63. package/src/worker-options.ts +19 -15
  64. package/src/worker-registry.ts +16 -1
  65. package/src/worker-run.ts +38 -8
  66. package/src/worker-tally.ts +53 -0
  67. package/src/worker-types.ts +7 -4
  68. package/src/worker.ts +124 -73
  69. package/src/driver-nats.ts +0 -86
  70. package/src/driver-redis.ts +0 -89
package/CLAUDE.md CHANGED
@@ -8,7 +8,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
8
8
  `expectedQueryLoop` ONLY: `steps.ts` declares the per-step write one-per-step to the N+1
9
9
  detector there, and no client is ever taken from it. Never `http`, `render`, `ui`.
10
10
  - Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
11
- - External deps: none. Postgres access goes through the injected `PgExecutor`.
11
+ - External deps: none. Postgres access goes through the injected `PgExecutor` (`@ultimat3/core`'s,
12
+ re-exported from `index.ts`).
12
13
 
13
14
  ## Rules — registration and declarations
14
15
 
@@ -29,7 +30,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
29
30
  - **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — do not re-litigate.**
30
31
  `tenant` is REQUIRED (`(input) => input.orgId` or `'none'`; `X_JOB_TENANT_REQUIRED` backstop).
31
32
  `executeJob` puts the derived org on the ctx's actor (`jobRunActor`, org only) and runs the body in
32
- `runWithContext`; `'none'` STRIPS the org (fail-closed). A boot-supplied service actor was rejected.
33
+ `runWithContext`; `'none'` STRIPS the org (fail-closed).
33
34
  `tenantFor` is a METHOD on `JobHandle` (variance). The queue row's `tenantId` stays the ENQUEUER's
34
35
  (the limiter's bucket). `tenancy-cross-surface.test.ts` asserts HTTP and job verdicts are EQUAL.
35
36
  - **`enqueuedBy` is ATTRIBUTION, never authority** — do not re-litigate. A job acting FOR a user takes
@@ -43,39 +44,37 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
43
44
 
44
45
  ## Rules — the driver and SQL
45
46
 
46
- - **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_steps`,
47
- `x_backfills`, `x_outbox`, `x_scheduler_state`, `x_scheduler_leader`, `x_job_leases`,
48
- `x_job_events`, `x_job_pauses`, `x_job_workers`, `x_job_counters`). A NEW table is also a name in
47
+ - **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`,
48
+ `x_outbox`, `x_scheduler_*`). A NEW table is also a name in
49
49
  `packages/cli/src/framework-schema.ts`. A shipped table grows by `alter table ... add column if
50
- not exists`. Its comments carry NO apostrophes and NO semicolons (`dev-queue.ts` splits on `;`; `driver-pg-sql.test.ts` checks
50
+ not exists`. Its comments carry NO apostrophes and NO semicolons (it splits on `;`; `driver-pg-sql.test.ts` checks
51
51
  quote parity).
52
52
  - **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
53
- `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's `claim`/`list`/`deadLetters` are `async` so a
54
- refusal rejects on both.
53
+ `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's reads are `async` so a refusal rejects on both.
55
54
  - **`ack`, `nack`, `heartbeat` and `recordProgress` are FENCED on `state = 'running'` AND the
56
55
  CLAIM** — `{ workerId, claim }`, `claimOf(claimed)`; `x_jobs.claims` is moved by the claim and
57
- never reset. A worker id alone let a worker's own earlier body settle its re-claim. `heartbeat`
58
- answers a boolean, read as `held === false` (never `!held`).
59
- - **A driver's semantics are pinned in ONE test beside the pg statement** (`driver-parity.test.ts`):
60
- list order, the `attempt` floor, `SQL_LEASE_RENEW` fencing on `expires_at > now()` and `holder`,
61
- `SQL_STATS` one bucket per row.
56
+ never reset. `heartbeat` answers a boolean, read as `held === false` (never `!held`).
57
+ - **A driver's semantics are pinned beside the pg statement** (`driver-parity*.test.ts`);
58
+ `driver-*-fixture.ts` scenarios run on memory AND Postgres.
59
+ - **The claim BURIES a lease that lapsed on the final attempt** (`attempt >= max_attempts`): `dead`
60
+ in `SQL_CLAIM` — `failed` for a name in `ClaimOptions.dropExhausted` (`retry.deadLetter: false`)
61
+ — counted, never returned; the worker's round hears `onExhausted` and calls `announceExhausted`.
62
+ - **`cancel` is fenced on `LIVE_STATES`.** **`requeue`** refuses (`X_JOB_NOT_FOUND`, not requeueable,
63
+ duplicate), then ONE statement. **`enqueue` inserts twice at most.** **`BulkResult.remaining` is
64
+ what a second call would move.**
62
65
  - **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
63
66
  (discovered, comments stripped) for `select *`/`returning *`.
64
67
  - Drivers implement the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`. New
65
68
  capabilities go behind the interface. `inspect.ts` returns plain JSON for CLI, `/_x` and MCP.
66
- - Step results are persisted BEFORE the step returns. All time is epoch ms from an injected `Clock`
67
- (`nowMs()` in `clock.ts`).
68
- - **`createPgLeader` is correct only on a DEDICATED connection**; boot uses `createPgLeaseLeader`.
69
- - `driver-redis.ts` / `driver-nats.ts` are honest `X_NOT_IMPLEMENTED` stubs.
69
+ - Step results are persisted BEFORE the step returns. All time is epoch ms (`nowMs()`, `clock.ts`).
70
+ - **`postgresLeader` is correct only on a DEDICATED connection**; boot uses `postgresLeaseLeader`.
70
71
 
71
72
  ## Rules — numbers and limits
72
73
 
73
74
  - **Every numeric knob is refused when not FINITE** — core's `finiteOption()` (a bound) and
74
- `finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `createWorker` reads them.
75
- **`bun run finite-bounds` is a floor, never proof**: `createLimiter`'s five numbers (`perTenant`,
76
- `perQueue`, `global`, `ratePerTenant.limit` / `.windowMs`) are screened with min 0 (zero is a HARD
77
- STOP). **A row count is `finiteCount` (min 0)** for driver parity (`list`, `deadLetters`, the
78
- backfill ledger's `list`, `assertClaimBounds`).
75
+ `finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `jobWorker` reads them.
76
+ **`bun run finite-bounds` is a floor, never proof**: `concurrencyLimiter`'s five numbers are screened
77
+ with min 0 (zero is a HARD STOP). **A row count is `finiteCount` (min 0)**; `retry.attempts` is one with min 1.
79
78
  - **`WorkerOptions.concurrency` is read by OWN key** — a queue named `constructor`
80
79
  (`worker-slots.test.ts`).
81
80
  - **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
@@ -99,24 +98,21 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
99
98
  - The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
100
99
  job's plain key.
101
100
  - **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
102
- the poll timer, abort held runs); `close` is the teardown. `stop()` hands both back in a `finally`;
101
+ the poll timer, arm the drain cut-off — it aborts nothing); `close` is the teardown. `stop()` hands both back in a `finally`;
103
102
  `start()` refuses while draining.
104
- - **A claimed job is counted with core's `beginWork()`**, so the drain's in-flight phase does the
105
- waiting. **The teardown's wait is bounded on SIGTERM** (`settleAllBy` in `drain-wait.ts`,
106
- `jobs.worker.drain-abandoned`) and unbounded on a manual `stop()`. A worker always reaches `'stopped'`;
107
- `state` is set in the `finally`. **One teardown, joined.**
108
- - **SIGTERM reaches the job**: the worker's one `AbortController` (`drainSignal`) is composed into every
109
- run (`worker-run.ts`); `stopAccepting(reason)` aborts it with `JobDrainedError` (`X_DRAINING`). A
110
- drained attempt settles as **`interrupted`** (`nack`, `countsAsAttempt: false`, no park, no dead
111
- letter), read off the SIGNAL. `WorkerStats.interrupted`; `worker-drain-signal.test.ts`.
103
+ - **A claimed job is counted with core's `beginWork()`**: the drain's in-flight phase waits. The
104
+ teardown's wait is bounded on SIGTERM (`drain-wait.ts`), unbounded on `stop()`; one teardown, joined.
105
+ - **A deploy lets a running job FINISH**: SIGTERM only stops claiming. `drainSignal` aborts
106
+ (`X_DRAINING`) at `deadlineAt − margin` (`worker-drain-cutoff.ts`), then held claims go back
107
+ uncounted (`worker-held.ts`). Settles **`interrupted`**; `docs/history/jobs.md`, "Drain, 2026-10".
108
+ - **`X_RATE_LIMITED` defers uncounted** (`rateLimitDeferralMs`).
112
109
  - **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
113
110
  synchronously with its guard.
114
- - **A fleet slot is taken INSIDE a `try`, released AWAITED, and a renewal answering `false` CANCELS the
115
- run** (`X_JOB_SLOT_LOST`, read as `renewed !== false`).
111
+ - **A fleet slot is taken INSIDE a `try`, released AWAITED, and HELD, not owned**: `false`, or a TTL
112
+ with no renewal landing, CANCELS the run (`X_JOB_SLOT_LOST`); held per job id as a LIST.
116
113
  - **The run's signal is a controller this worker owns** (`run-signal.ts`), never `AbortSignal.any`.
117
- - **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal is `jobs.heartbeat.failed`; a
118
- whole window without one landing is `jobs.lease.lost` + `recordLeaseLost(queue)`, measured on this
119
- process's clock, asked both sides of the call.
114
+ - **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal warns; a whole window without
115
+ one landing is `jobs.lease.lost`, on this process's clock, asked both sides of the call.
120
116
  - **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
121
117
  interval is `unref`ed; every renewal is armed through `WorkerOptions.schedule`.
122
118
  - **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
@@ -125,21 +121,26 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
125
121
  - **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
126
122
  - **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
127
123
  the attempt is cancelled in `executeJob`'s `finally`.
128
- - **A cancelled runner writes NOTHING** — `put()` in `steps.ts` is the one fence (`X_ABORTED`);
129
- `jobs.timeout.abandoned` names an uncooperative body. `executeJob` reads `ctx.signal` defensively.
124
+ - **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`) — save a fenced
125
+ `X_DRAINING` on a live attempt: a step done in the drain is recorded once. The STORE fences
126
+ the claim (`StepFence`, `SQL_STEP_PUT` as `SQL_ACK`: `X_JOB_LEASE_LOST`). `executeJob` passes it.
130
127
  - **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
131
128
  - **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
132
129
  out before `leader.release()`. Same two-hook rule; an ABANDONED round does not release
133
130
  (`jobs.scheduler.drain-abandoned`). `stillLeading()` is asked every round and before every task,
134
131
  and answers from memory inside `renewEveryMs`.
135
- - **`run-once` fires ONE catch-up** (marks `at`); `skip` fires the true latest missed occurrence
136
- (`latestOccurrenceBy`, bisection).
132
+ - **`run-once` fires ONE catch-up**: the watermark rides the fire (`ScheduledFire.watermarkMs`,
133
+ `SQL_SCHEDULER_FIRE` `$4`), never a `markFired` behind it; `skip` fires the true latest missed
134
+ occurrence (`latestOccurrenceBy`, bisection).
135
+ - **A lease for a run that never STARTED is `abandon()`ed, never `release()`d** — the rate stamp
136
+ goes with it (`worker-admit.ts`). **A failed pass re-arms at the floor** (`worker-loop.ts`
137
+ catch); `timers` is the test seam.
137
138
  - **Every timer body catches before it finalises** (`void work().catch(log).finally(...)`).
138
139
  - **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
139
140
  `nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
140
141
  stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
141
- - **A run's acquisitions are handed back even when the wiring throws** (`worker-run.ts`: `context()`
142
- above the heartbeat; `createRunSignal` and `fleetSlots.startRenewal` inside the `try`).
142
+ - **A wiring throw hands back the run's holdings, and its ROW until `executeJob` starts**
143
+ (`worker-run.ts`), as a failed shed does its job (`worker-admit.ts`).
143
144
  - **`step.run` hydrates the run's steps from ONE `store.list(runId)`**; `sleep` and `waitForEvent`
144
145
  record replays through `trace()`. **`claimName` reads a Set**; `usedNames()` keeps
145
146
  `MAX_TRACE_NAMES` (200).
@@ -153,9 +154,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
153
154
  suite (memory + real pg).
154
155
  - **A settle answers whether it landed**; a miss is `jobs.settle.unowned`, not a throw.
155
156
  - **The counter moves in the SETTLING statement** (`driver-pg-settle-sql.ts`): one-minute bucket per
156
- job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing), and
157
- `ack({ counted: false })` — `x jobs drain` — adds nothing. The scheduler leader folds tiers
158
- (`rollupCounters`), at most once a bucket.
157
+ job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing).
158
+ The scheduler leader folds tiers (`rollupCounters`), at most once a bucket.
159
159
  - **`list()` is KEYSET** (`after: jobCursor(lastRow)`, newest first by `(created_at, id)`); the pg
160
160
  seek reads the cursor row's own `created_at` — the cursor's ms is rounded. A page past
161
161
  `MAX_JOB_PAGE` or a foreign cursor is `X_JOB_PAGE_INVALID`, never a bare invariant.
@@ -166,10 +166,9 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
166
166
  - **An occurrence fires in ONE statement** (`SchedulerState.fire`, `SQL_SCHEDULER_FIRE`): the
167
167
  watermark is the fence, jobs insert only `where exists (select 1 from moved)`.
168
168
  - **`progress()` is throttled in `progress.ts`** and flushed before every settle. **`onSettled`
169
- (`settled.ts`) is the ONE ending hook** — `completed` (with `result`), `dead-lettered`,
170
- `dropped`, `refused` — AFTER a settle that LANDED, in the body's tenant scope,
171
- `ON_SETTLED_ATTEMPTS` tries, never rethrown (`X_JOB_ON_SETTLED_FAILED`), at most once across a
172
- crash. Never for a cancel, a retry or a suspension; no second hook beside it.
169
+ (`settled.ts`) is the ONE ending hook** — `completed`, `dead-lettered`, `dropped`, `refused` —
170
+ AFTER a settle that LANDED, in the body's tenant scope, `ON_SETTLED_ATTEMPTS` tries, never
171
+ rethrown, at most once across a crash. Never for a cancel, a retry or a suspension.
173
172
  - **Failed for good + `deadLetter: false` is DROPPED**: `nack({ fail })`, outcome `dropped`.
174
173
  - **Worker registry rows expire** (`worker-registry.ts`, TTL = visibility timeout); `stop()` forgets.
175
174
 
@@ -196,7 +195,9 @@ Long form: `docs/history/jobs.md`, "the cross-process wake".
196
195
  - **`idle-cost.test.ts` / `queue-wake.test.ts` count statements per idle minute** — scheduler 6,
197
196
  worker and relay 30 with no wake, 12 with. Raising one is a regression.
198
197
  - **The memory driver stores a payload's JSON form**, as pg binds it (`driver-settle-parity.test.ts`).
199
- - **The event bus has ONE clock**: `createPgEventBus` takes no `clock`; `bus.now()` is "asked at".
198
+ - **The event bus has ONE clock**: `postgresEventBus` takes no `clock`; `EventLookup.now()` is
199
+ REQUIRED, stamps a NEW wait, and calls the TIMEOUT (asked once the runner thinks time is up).
200
+ `purgeExpired()` is awaited and counted; `eventsPurgeTarget(bus)` is its `PurgeTarget`.
200
201
 
201
202
  ## Rules — the outbox
202
203
 
@@ -225,8 +226,8 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
225
226
  - **`purge()`** is the one caller of every `purgeExpired()`: one table per `step.run`, one clock
226
227
  reading per pass, duplicate names refused.
227
228
  - **`exportRows()`**: one object per PAGE, named by page index; NO cross-tenant escape.
228
- - **`webhook()`** delivers ONE event to ONE endpoint; no steps; every attempt recorded before the
229
- throw. **The wire format is core's**, re-exported.
229
+ - **`webhook()`**: ONE event to ONE endpoint, no steps; each attempt but a CANCELLED one recorded
230
+ before the throw; `ledger.isDisabled` gates the socket. **The wire format is core's.**
230
231
  - A `-fixture.ts` file does not ship; `backfill-pass-fixture.ts` raises a plain `Error` subclass on
231
232
  purpose (it stands in for app code).
232
233
 
@@ -238,7 +239,7 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
238
239
  | `tenant.ts` | what tenant a run acts under: `JobTenant`, the declaration's backstop, the actor `executeJob` installs |
239
240
  | `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input |
240
241
  | `backfill-pass.ts` | one pass: the batched iteration, its cursor checkpoints and its ledger row |
241
- | `backfill-scope.ts` | which sweeps run across tenants — `tenant: 'none'` only, on the pass's own actor, for the pass's own life |
242
+ | `backfill-scope.ts` | which sweeps run across tenants (`tenant: 'none'` only) |
242
243
  | `backfill-ledger.ts` | `x_backfills` — the contract, `BACKFILL_STATUSES`, the checksum, the verdict, the memory ledger |
243
244
  | `backfill-registry.ts` | what was DECLARED: the `origin` stamp, `isBackfill`, `registeredBackfills` |
244
245
  | `backfill-gate.ts` | may this sweep run here and now — environment, `requires`, already-applied |
@@ -249,38 +250,39 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
249
250
  | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements; an `action-job` projection is `X_ACTION_JOB_UNBRIDGED` |
250
251
  | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
251
252
  | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
253
+ | `steps-timeout.ts` / `steps-suspension.ts` | a step's ceiling; `StepSuspension` and the timed-out marker |
252
254
  | `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
253
255
  | `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
254
- | `outbox-pg.ts` | `createPgOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
256
+ | `outbox-pg.ts` | `postgresOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
255
257
  | `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
256
- | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one (drops a key with its last slot), `jobLeaseKey` |
258
+ | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
257
259
  | `concurrency.ts` | `job.concurrency` as declared, resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
258
- | `errors-concurrency.ts` | every concurrency refusal's class — declaration, key, `X_JOB_KEY_BUSY`, the boot refusal. Codes stay declared in `errors.ts` |
260
+ | `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `errors.ts` |
259
261
  | `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
260
262
  | `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
261
- | `scheduler-pg.ts` | `pgSchedulerState` (the durable watermark, the atomic fire) + `createPgLeaseLeader` |
262
- | `events-pg.ts` | `createPgEventBus` — `step.waitForEvent` across processes |
263
+ | `scheduler-pg.ts` | `postgresSchedulerState` (the durable watermark, the atomic fire) + `postgresLeaseLeader` |
264
+ | `events-pg.ts` | `postgresEventBus` — `step.waitForEvent` across processes |
263
265
  | `driver.ts` | `JobDriver` contract + wire records |
264
- | `driver-pg.ts` | default driver, and `createPgLeader` — the advisory-lock election a scheduler does NOT use |
266
+ | `driver-pg.ts` | default driver, and `postgresLeader` |
265
267
  | `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the ONE install point. Its comments carry no `;` and no `'` |
266
268
  | `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and `JOB_ROW_COLUMNS`; re-exported from `driver-pg-sql.ts` |
267
269
  | `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
268
- | `driver-memory.ts` | `x dev` / tests |
269
- | `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
270
+ | `driver-memory.ts` / `claim-exhausted.ts` | `x dev` / tests; a buried row's log line and `onSettled` |
270
271
  | `retry.ts` | the dead-letter decision, and this package's option names over core's `backoffDelay` — no curve of its own |
271
- | `retry-classification.ts` | the OTHER half of that decision: what the thrown error says, and the stop reason the row and the log carry |
272
+ | `retry-classification.ts` | the other half: what the thrown error says, and the stop reason |
272
273
  | `execute.ts` / `run-deadline.ts` | `executeJob` — one claimed job run and settled; the run's deadline: cancel, then fail |
273
274
  | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
274
- | `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
275
- | `worker.ts` | `worker` role, claim loop, drain — and the one `AbortController` SIGTERM reaches every held run through |
275
+ | `renewal-timer.ts` | the renewal interval and its `stopped()` latch |
276
+ | `worker.ts` | `worker` role, claim round, drain |
277
+ | `worker-drain-cutoff.ts` / `worker-held.ts` / `worker-tally.ts` | drain cut-off · held claims · counters |
276
278
  | `worker-admit.ts` | may this claimed job start: the limiter, the fleet slot, and the shed / refusal |
277
279
  | `worker-registry.ts` | one worker's registry row: announce, heartbeat, forget |
278
280
  | `worker-types.ts` | the worker's public contract: `WorkerOptions`, `WorkerStats`, `Worker` |
279
- | `drain-wait.ts` | the drain's wait, shared by both roles: settled, or abandoned at the `close` hook's budget |
280
- | `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span — handed back in one `finally` |
281
- | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |
281
+ | `drain-wait.ts` | the drain's wait, shared by both roles |
282
+ | `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span |
283
+ | `run-signal.ts` | the signal ONE run is cancelled by |
282
284
  | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back — and the boot refusal |
283
- | `purge.ts` | `purge()` — a factory over `job()`: the retention sweep, its structural target seam and the hourly cron a host schedules it on |
285
+ | `purge.ts` | `purge()` — a factory over `job()`: the retention sweep and its target seam |
284
286
  | `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
285
287
  | `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
286
288
  | `limits.ts` | per-tenant / per-queue / global concurrency + rate |
@@ -298,16 +300,11 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
298
300
  | `queue-wake.ts` / `driver-pg-wake-sql.ts` | the `LISTEN` session and its proof; which statement notifies |
299
301
  | `worker-loop.ts` | the claim loop's timer: wake, `again`, the kicks |
300
302
  | `driver-pg-outbox-sql.ts` | the `x_outbox` statements |
301
- | `errors-operator.ts` | `X_JOB_NOT_REMOVABLE`, `X_JOB_NOT_PROMOTABLE`, `X_JOB_ON_SETTLED_FAILED` classes |
303
+ | `errors-operator.ts` / `errors-requeue.ts` | the operator verbs' refusals; `requeue`'s |
302
304
 
303
305
  Which `-fixture.ts` harness each suite shares, and what each `.job.` suite proves:
304
306
  `docs/history/jobs.md`, "Moved 2026-10-01 — test harnesses".
305
307
 
306
- ## Commands
307
-
308
- ```
309
- bun test packages/jobs
310
- bun run --filter @ultimat3/jobs typecheck
311
- ```
308
+ Commands: `bun test packages/jobs` · `bun run --filter @ultimat3/jobs typecheck`.
312
309
 
313
310
  Why each rule above is shaped the way it is: [`docs/history/jobs.md`](../../docs/history/jobs.md).
package/README.md CHANGED
@@ -192,11 +192,17 @@ input. The tenant is a fact about the work, so the job declares it.
192
192
  expires, so on a bus shared by one `bun test` process an answer one test published resumes the
193
193
  next test's wait — call it between tests.
194
194
 
195
- The event bus has ONE clock. `createPgEventBus({ executor })` stamps `publishedAt` and the expiry
195
+ The event bus has ONE clock. `postgresEventBus({ executor })` stamps `publishedAt` and the expiry
196
196
  with the database's `now()` — it takes no `clock` — and `bus.now()` answers that same clock. A
197
197
  consumer that wants "published after I asked" (`@ultimat3/scraping`'s `eventPrompt`) reads its
198
198
  "asked at" from `bus.now()`, never from its own process: a skew between two pods must not decide
199
- which answers count.
199
+ which answers count. `step.waitForEvent` stamps a new wait from the same clock
200
+ (`EventLookup.now()`), so a worker running ahead of the database still matches an event published
201
+ after it asked.
202
+ `step.waitForEvent` stamps a new wait from that clock (`EventLookup.now()`, required) and asks it
203
+ again before declaring a timeout, so a worker running ahead of the database still matches an event
204
+ published after it asked and does not give up early. `eventsPurgeTarget(bus)` is the bus as a
205
+ `purge()` target: `purgeExpired()` is awaited and answers the rows it removed.
200
206
 
201
207
  Step names are the replay key, so they must be deterministic and unique in a run — a
202
208
  duplicate is `X_STEP_DUPLICATE`, not a silent overwrite. Suspension is control flow
@@ -405,6 +411,11 @@ export const hourly = task({
405
411
  | at least once is safe here | a replayed delete removes rows that are already gone, and a row a purge deleted answers exactly as one that was never there |
406
412
  | two targets under one name | `X_INVARIANT`, before the first delete — `step.run` would raise `X_STEP_DUPLICATE` after one table was already empty |
407
413
 
414
+ The stored event bus is a target too: `eventsPurgeTarget(bus)` answers
415
+ `{ name: EVENTS_PURGE_TARGET, purgeExpired }` over `bus.purgeExpired()`, which is awaited and
416
+ answers the rows it removed — `x_job_events` only grew while nothing called it. Add it to
417
+ `targets`: `targets: () => [eventsPurgeTarget(eventBus())]`.
418
+
408
419
  `@ultimat3/cli`'s boot declares both halves over the three tables it owns, so an app gets the sweep
409
420
  without writing any of the above. It needs a `worker` to run it and a `scheduler` to fire it: a
410
421
  deployment with neither has no background work at all, and this is one more thing it does not do.
@@ -526,9 +537,11 @@ x-ultimate-webhook-signature: t=1700000000,v1=<hex hmac-sha256>
526
537
 
527
538
  The receiving half is `verifyWebhookSignature(request, { secret })` in `@ultimat3/http`. The format
528
539
  itself — the canonical string, the mac and the header names — is **one module in `@ultimat3/core`**
529
- (`webhook-signature.ts`), re-exported by both packages and re-declared by neither: this package
530
- signs, `http` verifies, and neither may import the other, so the one copy lives at the tier both
531
- can reach. Same argument `timing-safe-equal.ts` makes for itself.
540
+ (`webhook-signature.ts`), imported from `@ultimat3/core` by both packages and re-declared by
541
+ neither: this package signs, `http` verifies, and neither may import the other, so the one copy
542
+ lives at the tier both can reach. A receiver or a test imports `WEBHOOK_SIGNATURE_HEADER`,
543
+ `webhookSignature` and the rest from `@ultimat3/core` — 25.0.0 dropped this package's
544
+ re-exports (`X_HELPER_COPY`). Same argument `timing-safe-equal.ts` makes for itself.
532
545
 
533
546
  | Rule | Why |
534
547
  |---|---|
@@ -539,10 +552,21 @@ can reach. Same argument `timing-safe-equal.ts` makes for itself.
539
552
  | `redirect: 'manual'` | following a 3xx would re-POST a body signed for one host to whatever the receiver named |
540
553
  | the endpoint is never checkpointed | a `step.run` output lands in `x_job_steps`, and the endpoint carries the secret |
541
554
  | every attempt is recorded **before** the throw | a failure the ledger cannot see is a failure the consecutive count cannot see, which is an endpoint that never gets disabled |
542
- | a `Retry-After` the receiver names is honoured | `X_WEBHOOK_DELIVERY_THROTTLED` carries `meta.retryAfterSeconds`, which the nack waits out (clamped by `retry.maxDelay`) rather than guessing a curve against an answer it already has |
555
+ | a `Retry-After` the receiver names is honoured | `X_WEBHOOK_DELIVERY_THROTTLED` carries `meta.retryAfterSeconds`, which the nack waits out — read by core's `retryAfterSecondsOf` (delta-seconds, or an HTTP-date against the response's own `Date`; `0` or a past date is unstated), clamped by `retry.maxDelay`, plus core's `jitterStatedDelay` spread so deliveries handed one instant do not wake together |
556
+ | `WebhookLedger.isDisabled(endpointId)` is asked before every socket, beside the row's own `disabled` | `disableAfter` writes only the ledger: a verdict nobody asks for again is an endpoint that keeps receiving POSTs. Answer it from the column your `disable` sets |
557
+ | the deadline and a lost lease abort `fetch`; a failure that abort caused writes no ledger row | the nack behind them hands the job to another worker at once, so a request left open is a duplicate delivery, and an abandoned attempt's failure would count toward `disableAfter` |
558
+ | the DRAIN never aborts a POST already on the wire; it stops one not yet sent | torn down, the job goes to the next pod, which re-POSTs the same event on every deploy. Finished, it is recorded like any other outcome — a real failure during the drain included |
543
559
  | re-enabling is always yours | an endpoint the framework un-disabled on its own is a retry loop with no end |
544
560
  | `WebhookLedger` is a seam, not a table | retention is seven years for one business and thirty days for the next, so shipping a schema would ship one of those answers |
545
561
 
562
+ **Set `timeout` below the drain budget.** A webhook has no timeout unless you declare one, and a
563
+ POST on the wire when SIGTERM lands runs until the receiver answers. With a `timeout`, a drained
564
+ request is still torn down at the attempt's deadline — where `executeJob` hands the job on — and
565
+ writes no ledger row. Without one, the only bound is `configureLifecycle({ deadlineMs })`, 25 s by
566
+ default: past it the worker logs `jobs.worker.drain-abandoned` and exits, and the row comes back
567
+ only when its lease lapses, as a COUNTED attempt the receiver may already have processed.
568
+ `timeout: '10s'` against the default budget ends every drained delivery inside it.
569
+
546
570
  ## The deadline cancels
547
571
 
548
572
  A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that
@@ -659,15 +683,15 @@ things have to be true in a process:
659
683
  | Step | Call |
660
684
  |---|---|
661
685
  | the table exists | ships in `SQL_JOBS_TABLE` — applying the queue DDL is enough |
662
- | the facade is installed | `setJobsFacade(createJobsFacade({ store, driver }, currentTx))` |
663
- | the relay is running | `createOutboxRelay({ store, driver }).start()` |
686
+ | the facade is installed | `setJobsFacade(outboxJobsFacade({ store, driver }, currentTx))` |
687
+ | the relay is running | `outboxRelay({ store, driver }).start()` |
664
688
 
665
- `start()` registers the same two shutdown hooks `createWorker` does — `accept` stops polling
689
+ `start()` registers the same two shutdown hooks `jobWorker` does — `accept` stops polling
666
690
  (the worker's also aborts every held job's `ctx.signal`), `close` waits out the pass in flight
667
691
  under the drain's deadline — and `stop()` hands both back.
668
692
  `drainOnShutdown: false` opts out, for a caller that drives its own teardown.
669
693
 
670
- with `store = createPgOutboxStore({ executor, txExecutor })`. `txExecutor` is what makes it
694
+ with `store = postgresOutboxStore({ executor, txExecutor })`. `txExecutor` is what makes it
671
695
  transactional: `stage()` runs on the CALLER'S connection, never the pool. With nothing installed,
672
696
  `jobsFacade()` answers a fallback whose `currentTx` is `() => undefined` and every enqueue
673
697
  publishes straight to the driver — deliberate, so a script and a test enqueue with no wiring, but
@@ -702,7 +726,7 @@ What the lease buys, precisely:
702
726
  | a relay that died mid-batch stranding its rows forever | anything a **non-idempotent** handler does on its second run |
703
727
  | — | a lapsed visibility lease re-delivering a job that is still running |
704
728
 
705
- The memory store (`createMemoryOutboxStore`, `x dev` and tests) **drops** a published row —
729
+ The memory store (`memoryOutboxStore`, `x dev` and tests) **drops** a published row —
706
730
  `retained()` is the relay's backlog, not a running total; the pg store keeps `published_at` as
707
731
  the audit trail this map is not. A relay pass that throws is logged as `jobs.outbox.tick-failed`
708
732
  and the loop re-arms: an unobserved rejection would end the process with rows still staged.
@@ -719,17 +743,35 @@ One interface: `enqueue`, `claim` (visibility timeout), `ack(jobId, { workerId,
719
743
  `backfills` and `leases`. Both settles are fenced on the CLAIM — `claimOf(claimed)` is the
720
744
  `{ workerId, claim }` a `ClaimedJob` carries — and answer whether they landed. Zero job-code change
721
745
  between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driver` config line**:
722
- `JobsConfig.driver` has no reader and boot always builds `createPgDriver`.
746
+ `JobsConfig.driver` has no reader and boot always builds `postgresJobDriver`.
723
747
 
724
748
  | Driver | Status | Backing | Use |
725
749
  |---|---|---|---|
726
750
  | `pg` | **default** | `SELECT ... FOR UPDATE SKIP LOCKED`, a partial unique index on `(name, coalesce(tenant_id, ''), idempotency_key)` over live rows (`x_jobs_name_tenant_idempotency_live_idx`), lease-based leader, `x_job_leases` | zero-infra start, most apps |
727
751
  | `memory` | complete | in-process maps | `x dev`, tests |
728
- | `redis` | interface-complete, `X_NOT_IMPLEMENTED` | Streams + consumer groups | planned |
729
- | `nats` | interface-complete, `X_NOT_IMPLEMENTED` | JetStream work queue | planned |
730
752
 
731
- The pg SQL is exported verbatim (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …) so an agent
732
- debugging a stuck queue can read and run the exact statement.
753
+ There is **no NATS or Redis jobs driver**: `createNatsDriver` and `createRedisDriver`, all-throw
754
+ stubs, were deleted in 25.0.0. A real `redisJobDriver()` would arrive as a minor.
755
+
756
+ The pg SQL lives verbatim in `src/driver-pg-*sql.ts` (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …)
757
+ so an agent debugging a stuck queue can read and run the exact statement. The barrel exports
758
+ only the statement another package runs: `SQL_JOBS_TABLE` (the install) — 25.0.0 took the rest
759
+ off, `SQL_CLAIM` and `SQL_OUTBOX_RELEASE` included (their array-parameter proofs are this package's
760
+ `driver-pg-array.live.test.ts`).
761
+
762
+ **A lease that lapses on a row's final attempt is buried by the claim.** `claim` re-takes a
763
+ `running` row whose lease lapsed — the worker died — but one already at `attempt >= maxAttempts`
764
+ is settled `dead` in the claim statement itself (`SQL_CLAIM`'s second arm; `isFinalAttempt` on the
765
+ memory driver), with `lastError` set to `LEASE_LAPSED_FINAL_ATTEMPT`, counted `dead`, and never
766
+ handed out: a job that kills its worker reaches the dead-letter queue after `retry.attempts`
767
+ claims instead of taking a worker per visibility timeout, forever. The pass is told through
768
+ `ClaimOptions.onExhausted(rows)` — at most once per pass, once per row — and `announceExhausted`
769
+ (`claim-exhausted.ts`) is what a claim round does with them: `jobs.claim.exhausted` at `error`, and
770
+ the job's `onSettled`, and the worker's claim round does exactly that. The row carries
771
+ `maxAttempts` and not the policy, so the CALLER names the jobs declaring `retry.deadLetter: false`
772
+ (`ClaimOptions.dropExhausted` — the worker reads them off its registry): those rows are buried
773
+ `failed` and announced `dropped`, every other one `dead` / `dead-lettered`. A caller that names
774
+ none buries every such row `dead`.
733
775
 
734
776
  A driver you write reads a nack's target state through `nackState(options)` — `dead`
735
777
  (`deadLetter`), `failed` (`fail`, terminal and out of the dead-letter queue), `suspended` (`park`)
@@ -740,8 +782,8 @@ or `ready` — so a new branch cannot land in one driver and not another. Its le
740
782
 
741
783
  | Role | Entry | Behaviour |
742
784
  |---|---|---|
743
- | `worker` | `createWorker({ driver, context, queues, concurrency })` | per-queue pools, lease heartbeat, SIGTERM drain: stop claiming → finish in-flight → close |
744
- | `scheduler` | `createScheduler({ driver, leader, state })` | one dispatch round at a time, catch-up policy, SIGTERM drain: stop dispatching → finish the round → release the lock |
785
+ | `worker` | `jobWorker({ driver, context, queues, concurrency })` | per-queue pools, lease heartbeat, SIGTERM drain: stop claiming → finish in-flight → close. `concurrency` is a number for every queue or a table per queue; a queue it does not name gets core's `JOBS_CONCURRENCY_DEFAULT` (8 — this package's own default was 5 before 25.0.0) |
786
+ | `scheduler` | `jobScheduler({ driver, leader, state })` | one dispatch round at a time, catch-up policy, SIGTERM drain: stop dispatching → finish the round → release the lock |
745
787
 
746
788
  Both roles register the same **pair** of hooks and bound the `close` half the same way. The
747
789
  scheduler's abandoned case differs in one respect: a round still enqueueing keeps the lease rather
@@ -773,24 +815,24 @@ update runs two leaders.
773
815
 
774
816
  ```ts
775
817
  import {
776
- createPgLeaseLeader,
777
- createScheduler,
818
+ postgresLeaseLeader,
819
+ jobScheduler,
778
820
  type JobDriver,
779
821
  type PgExecutor,
780
- pgSchedulerState,
822
+ postgresSchedulerState,
781
823
  } from '@ultimat3/jobs';
782
824
 
783
825
  declare const driver: JobDriver;
784
826
  declare const executor: PgExecutor; // see "A `PgExecutor`" below
785
827
 
786
- createScheduler({
828
+ jobScheduler({
787
829
  driver,
788
- state: pgSchedulerState(executor),
789
- leader: createPgLeaseLeader({ executor }),
830
+ state: postgresSchedulerState(executor),
831
+ leader: postgresLeaseLeader({ executor }),
790
832
  });
791
833
  ```
792
834
 
793
- `createPgLeaseLeader` and not `createPgLeader`: `pg_try_advisory_lock` is scoped to a Postgres
835
+ `postgresLeaseLeader` and not `postgresLeader`: `pg_try_advisory_lock` is scoped to a Postgres
794
836
  *session*, and the executor this package is handed is a **pool** — the lock is released the moment
795
837
  that connection goes back to it, so every node reads itself as leader. The lease is a row with an
796
838
  expiry and needs no connection affinity. `acquire()` is also the renewal, called every round, which
@@ -830,10 +872,10 @@ hourly task fire 24 catch-ups a second apart.
830
872
  client, which already speaks `(text, values)` and decodes `jsonb`:
831
873
 
832
874
  ```ts
833
- import { createPostgresClient } from '@ultimat3/db';
875
+ import { postgresClient } from '@ultimat3/db';
834
876
  import type { PgExecutor } from '@ultimat3/jobs';
835
877
 
836
- const client = createPostgresClient({ url: 'postgres://localhost:5432/app_test' });
878
+ const client = postgresClient({ url: 'postgres://localhost:5432/app_test' });
837
879
  export const executor: PgExecutor = {
838
880
  query: <R>(text: string, values: readonly unknown[]) => client.query<R>({ text, values }),
839
881
  };
@@ -861,11 +903,11 @@ A queue with nothing to do costs almost nothing, and a job starts when it is com
861
903
  `startQueueWake({ listener, executor })` holds ONE `LISTEN` session per worker pod and turns "another process committed" into the signal the idle loops already hear. The boot starts it for the `worker` role.
862
904
 
863
905
  ```ts
864
- import { createPostgresClient } from '@ultimat3/db';
906
+ import { postgresClient } from '@ultimat3/db';
865
907
  import type { PgExecutor } from '@ultimat3/jobs';
866
908
  import { startQueueWake } from '@ultimat3/jobs';
867
909
 
868
- const client = createPostgresClient({ url: 'postgres://localhost:5432/app_test' });
910
+ const client = postgresClient({ url: 'postgres://localhost:5432/app_test' });
869
911
  const executor: PgExecutor = {
870
912
  query: <R>(text: string, values: readonly unknown[]) => client.query<R>({ text, values }),
871
913
  };
@@ -934,7 +976,7 @@ exactly as it had before.
934
976
  | Thrown | What the queue does |
935
977
  |---|---|
936
978
  | a `terminal` code (`X_SCRAPE_AUTH_FAILED`, a validation fault, a permission denial) | dead-lettered on the attempt it happened, `attempt` recorded, remaining attempts unspent — a rotated password retried five times is five more wrong passwords at a site that locks the account after three |
937
- | a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay` — instead of the backoff. Still an attempt, still under the ceiling |
979
+ | a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay`, plus core's `jitterStatedDelay` spread unless `jitter: false` — instead of the backoff. Still an attempt, still under the ceiling |
938
980
  | a `retryable` code (`X_TIMEOUT`, `X_DRAINING`) | the backoff schedule above, unchanged |
939
981
  | an **unclassified** code, or anything that is not an `UltimateError` | the backoff schedule above, unchanged. Most codes are unclassified and `retryFor` answers `terminal` for all of them, so reading that would have stopped every transient retry in every app |
940
982
 
@@ -965,7 +1007,7 @@ contractual rate.
965
1007
  `job.concurrency` is the one that is fleet-wide, and it is enforced by a row every replica can
966
1008
  see — one per held slot, keyed `job:<name>`, renewed by the same heartbeat that renews the
967
1009
  visibility lease and reclaimed by TTL when a worker is SIGKILLed. A driver with no lease store
968
- cannot hold the cap, so `createWorker().start()` **refuses to boot**
1010
+ cannot hold the cap, so `jobWorker().start()` **refuses to boot**
969
1011
  (`X_JOB_CONCURRENCY_UNENFORCEABLE`) rather than let a documented guarantee do nothing.
970
1012
 
971
1013
  Over any cap the claim is handed straight back without burning an attempt — one org's 50k-row
@@ -1003,7 +1045,7 @@ export const syncAccount = job({
1003
1045
  |---|---|---|
1004
1046
  | a cap of `0`, negative, fractional, `NaN` or `Infinity` — `concurrency: 0` and `limit: 0` alike; a `key` that is not a function; a `whenBusy` outside the two | where the job is declared | `X_JOB_DECLARATION_INVALID` |
1005
1047
  | `key(input)` answering `''`, a non-string, or more than 200 characters | at that **enqueue** — the facade and the scheduler both ask | `X_JOB_DECLARATION_INVALID` |
1006
- | a driver with no lease store, for any `concurrency` | `createWorker().start()` | `X_JOB_CONCURRENCY_UNENFORCEABLE` |
1048
+ | a driver with no lease store, for any `concurrency` | `jobWorker().start()` | `X_JOB_CONCURRENCY_UNENFORCEABLE` |
1007
1049
  | a `LeaseStore` with no `holders(key)` | the build — the member is required | `TS2741` |
1008
1050
 
1009
1051
  What `'fail'` decides, exactly:
@@ -1071,7 +1113,9 @@ the pg and the memory driver alike. `x jobs`, `/_x` and a dashboard read nothing
1071
1113
  |---|---|---|
1072
1114
  | `list(filter)` | one page, newest first; `filter.after = jobCursor(lastRow)` for the next, `filter.before = jobCursor(firstRow)` for the previous (the rows nearest the cursor, still newest first; never both). Filters: `queue`, `name`, `state`, `tenantId`, `idPrefix`, `createdFrom`, `createdTo` | `MAX_JOB_PAGE` = 200 rows; `DEFAULT_JOB_PAGE` = 100. A page shorter than its limit is the last. A larger `limit`, or a cursor no page produced, is `X_JOB_PAGE_INVALID` |
1073
1115
  | `remove(id)` | delete one job and its steps. A running one is `X_JOB_NOT_REMOVABLE` | — |
1074
- | `requeueMany({ state, queue?, name?, tenantId? })` | re-queue finished rows; a row whose key a live job holds stays | `MAX_BULK_ROWS` = 1,000 per call; answers `{ affected, remaining }` |
1116
+ | `requeue(id, { fromStep? })` | re-queue ONE finished job (`REQUEUEABLE_STATES`). A live one is `X_JOB_NOT_REQUEUEABLE`, a key a live job holds `X_JOB_DUPLICATE`, an id nobody queued `X_JOB_NOT_FOUND` — on both drivers. `fromStep` drops that step and every later one in the statement that moves the row | — |
1117
+ | `cancel(id, reason?)` | stop a LIVE job (`LIVE_STATES`: ready, delayed, running, suspended). A done, failed, dead or cancelled row is left exactly as it ended and answers `undefined` — `cancelJob` turns that into `X_JOB_NOT_CANCELLABLE` | — |
1118
+ | `requeueMany({ state, queue?, name?, tenantId? })` | re-queue finished rows; a row whose key a live job holds stays, and of several finished rows sharing one key the oldest is requeued | `MAX_BULK_ROWS` = 1,000 per call, spent only on rows it can move; answers `{ affected, remaining }`, where `remaining` is what a second call could still move — a held row is not counted, so "call until zero" ends |
1075
1119
  | `removeMany({ state, queue?, name?, tenantId? })` | delete rows in any state but `running` | the same |
1076
1120
  | `promoteMany({ state, queue?, name?, tenantId? })` | `promote` over a set: every row still waiting on its `runAt` is due now. `state` is `delayed` or `ready` (`PROMOTABLE_STATES`) | the same; `remaining` is the rows still waiting |
1077
1121
  | `promote(id)` | a job waiting on its `runAt` — delayed, or backing off — is due now | — |
@@ -1089,7 +1133,6 @@ the pg and the memory driver alike. `x jobs`, `/_x` and a dashboard read nothing
1089
1133
  | the counter moves in the statement that settles the row | a count that can disagree with the rows is worse than none — and no second round trip per job |
1090
1134
  | a shed, a suspension and a drained attempt add nothing | they are handed back uncounted; counted, every `step.sleep` would read as a failure |
1091
1135
  | `ack` / `nack` / `heartbeat` / `recordProgress` are fenced on the CLAIM | `{ workerId, claim }`: a body whose lease lapsed must not settle, renew or report on the run that replaced it — claimed by another worker, or by the SAME one, whose id did not change. A miss answers `false` and logs `jobs.settle.unowned` |
1092
- | `ack(id, { …, counted: false })` adds nothing to the counters | `x jobs drain` settles the row it moved: `done` here, and not a completed run |
1093
1136
  | a pause is a row the claim reads | never a column on every queued row: pausing a million-row queue takes effect within one poll |
1094
1137
  | an occurrence fires in one statement | the watermark and its jobs move together (`SchedulerState.fire`), so a crash between them cannot fire it twice |
1095
1138
 
@@ -1133,6 +1176,7 @@ export const syncAccountWatched = job({
1133
1176
  |---|---|---|---|
1134
1177
  | the body returned and the ack landed | `completed` | `input`, `result` — what `run` returned, typed from it | `done` |
1135
1178
  | failed for good, parked in the dead-letter queue | `dead-lettered` | `input` (or `undefined`: the stored payload no longer parses), `error`, `code` | `dead` |
1179
+ | its lease lapsed on the final attempt and the claim buried it — every worker that took it died | `dead-lettered` | `input`, `error` (`LEASE_LAPSED_FINAL_ATTEMPT`), `code: undefined` — nothing was thrown | `dead` |
1136
1180
  | failed for good on `retry.deadLetter: false` | `dropped` | the same | `failed` |
1137
1181
  | `whenBusy: 'fail'` over a busy key — the body never ran | `refused` | the same, `code: 'X_JOB_KEY_BUSY'` | `failed` |
1138
1182
 
@@ -1198,15 +1242,16 @@ FOR a user takes that user's id in its input and re-authorises it in the body.
1198
1242
  | `X_ABORTED` | a cancelled attempt tried to write a step — core's code, not a second name for it |
1199
1243
  | `X_JOB_LEASE_LOST` | the job was cancelled, or its lease lapsed and the queue re-delivered it, while this worker was still running it |
1200
1244
  | `X_JOB_SLOT_LOST` | the fleet `concurrency` slot this run held was taken by another worker — a different row on a different clock from the lease above |
1201
- | `X_JOB_NOT_CANCELLABLE` | `cancelJob` reached a job that already finished, or a driver with no `cancel` |
1245
+ | `X_JOB_NOT_CANCELLABLE` | `cancelJob` reached a job that is not live — done, failed, dead or already cancelled — an id no queue holds, or a driver with no `cancel` |
1246
+ | `X_JOB_NOT_FOUND` | `requeue` was asked for an id the queue does not hold — one answer on both drivers. Fix: `x jobs ls --json` |
1202
1247
  | `X_JOB_CONCURRENCY_UNENFORCEABLE` | a registered job declares `concurrency`, plain or keyed, and the driver has no lease store |
1203
1248
  | `X_JOB_KEY_BUSY` | a run claimed under `whenBusy: 'fail'` while its key already held `limit` runs. Classified `terminal`. Fix: `x jobs ls --name <job> --state running --json` |
1204
1249
  | `X_JOB_DECLARATION_INVALID` | `job()` is missing a required field, declares a `concurrency` no worker can honour (plain or keyed), or its `concurrency.key` answered an empty, non-string or over-long key at enqueue |
1205
1250
  | `X_JOB_NOT_REMOVABLE` | `remove` / `removeMany` reached a running job. Fix: `x jobs cancel <id> --json` |
1206
1251
  | `X_JOB_NOT_PROMOTABLE` | `promote` reached a job not waiting on its run time |
1207
- | `X_JOB_PAGE_INVALID` | `list()` was asked for more than `MAX_JOB_PAGE` rows, or handed a cursor no page produced. Fix: `list({ limit: 200, after: jobCursor(lastRow) })` |
1252
+ | `X_JOB_PAGE_INVALID` | `list()` was asked for more than `MAX_JOB_PAGE` rows, or handed a cursor no page produced. The cause names which cursor (`after` / `before`). Fix: `x jobs ls --limit 200 --json` |
1208
1253
  | `X_JOB_ON_SETTLED_FAILED` | a declared `onSettled` threw on every one of its tries; logged and reported, never thrown |
1209
- | `X_NOT_IMPLEMENTED` | redis / nats driver |
1254
+ | `X_NOT_IMPLEMENTED` | introspection or the operator surface on a driver with no `introspect` (a hand-rolled one) |
1210
1255
 
1211
1256
  ### Error classes
1212
1257
 
@@ -1240,6 +1285,7 @@ a job boundary the class is gone and the `code` is what survives — match on th
1240
1285
  | `JobMaxAttemptsError` | `X_JOB_MAX_ATTEMPTS` | `src/errors.ts` |
1241
1286
  | `JobNameTakenError` | `X_JOB_DUPLICATE` | `src/errors.ts` |
1242
1287
  | `JobNotCancellableError` | `X_JOB_NOT_CANCELLABLE` | `src/errors.ts` |
1288
+ | `JobNotFoundError` | `X_JOB_NOT_FOUND` | `src/errors-requeue.ts` |
1243
1289
  | `JobNotPromotableError` | `X_JOB_NOT_PROMOTABLE` | `src/errors-operator.ts` |
1244
1290
  | `JobNotRemovableError` | `X_JOB_NOT_REMOVABLE` | `src/errors-operator.ts` |
1245
1291
  | `JobNotRequeueableError` | `X_JOB_NOT_REQUEUEABLE` | `src/errors-requeue.ts` |
@@ -1247,7 +1293,6 @@ a job boundary the class is gone and the `code` is what survives — match on th
1247
1293
  | `JobPageInvalidError` | `X_JOB_PAGE_INVALID` | `src/errors-operator.ts` |
1248
1294
  | `JobRowStatusUnknownError` | `X_JOB_ROW_STATUS_UNKNOWN` | `src/errors.ts` |
1249
1295
  | `JobSlotLostError` | `X_JOB_SLOT_LOST` | `src/errors.ts` |
1250
- | `JobsNotImplementedError` | `X_NOT_IMPLEMENTED` | `src/errors.ts` |
1251
1296
  | `JobTenantRequiredError` | `X_JOB_TENANT_REQUIRED` | `src/errors.ts` |
1252
1297
  | `JobTimeoutError` | `X_JOB_TIMEOUT` | `src/errors.ts` |
1253
1298
  | `LeaseLostError` | `X_JOB_LEASE_LOST` | `src/errors.ts` |