@ultimat3/jobs 23.0.0 → 25.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +72 -75
- package/README.md +83 -38
- package/package.json +7 -7
- package/src/backfill-errors.ts +3 -2
- package/src/backfill-ledger.ts +25 -3
- package/src/backfill-pass.ts +22 -5
- package/src/backfill-rate.ts +2 -2
- package/src/backfill.ts +2 -2
- package/src/claim-exhausted.ts +76 -0
- package/src/counters.ts +1 -1
- package/src/driver-memory-operator.ts +34 -12
- package/src/driver-memory.ts +80 -38
- package/src/driver-pg-jobs-sql.ts +27 -19
- package/src/driver-pg-operator-sql.ts +32 -12
- package/src/driver-pg-operator.ts +19 -3
- package/src/driver-pg-rows.ts +3 -3
- package/src/driver-pg-settle-sql.ts +2 -3
- package/src/driver-pg-sql.ts +99 -30
- package/src/driver-pg.ts +103 -98
- package/src/driver.ts +46 -16
- package/src/errors-concurrency.ts +1 -1
- package/src/errors-operator.ts +12 -4
- package/src/errors-requeue.ts +21 -3
- package/src/errors.ts +9 -17
- package/src/events-pg.ts +11 -21
- package/src/events.ts +26 -15
- package/src/execute.ts +47 -5
- package/src/export.ts +2 -2
- package/src/heartbeat.ts +7 -2
- package/src/index.ts +45 -99
- package/src/inspect-operator.ts +4 -4
- package/src/inspect.ts +5 -4
- package/src/introspection.ts +23 -11
- package/src/job.ts +6 -7
- package/src/leases.ts +1 -1
- package/src/limits.ts +46 -15
- package/src/outbox-lease.ts +1 -1
- package/src/outbox-pg.ts +6 -7
- package/src/outbox-relay.ts +6 -6
- package/src/outbox.ts +47 -19
- package/src/purge.ts +1 -1
- package/src/queue-wake.ts +1 -2
- package/src/renewal-timer.ts +1 -1
- package/src/retry-classification.ts +52 -15
- package/src/scheduler-leader.ts +2 -2
- package/src/scheduler-pg.ts +19 -16
- package/src/scheduler-state.ts +10 -3
- package/src/scheduler.ts +14 -15
- package/src/steps-memory.ts +2 -2
- package/src/steps-suspension.ts +46 -0
- package/src/steps-timeout.ts +33 -0
- package/src/steps.ts +90 -88
- package/src/task.ts +1 -1
- package/src/webhook-attempt.ts +206 -0
- package/src/webhook-ledger.ts +15 -4
- package/src/webhook.ts +64 -118
- package/src/worker-admit.ts +14 -6
- package/src/worker-drain-cutoff.ts +110 -0
- package/src/worker-fleet-slots.ts +82 -45
- package/src/worker-held.ts +100 -0
- package/src/worker-key-busy.ts +38 -10
- package/src/worker-loop.ts +35 -11
- package/src/worker-options.ts +19 -15
- package/src/worker-registry.ts +16 -1
- package/src/worker-run.ts +38 -8
- package/src/worker-tally.ts +53 -0
- package/src/worker-types.ts +7 -4
- package/src/worker.ts +124 -73
- package/src/driver-nats.ts +0 -86
- package/src/driver-redis.ts +0 -89
package/CLAUDE.md
CHANGED
|
@@ -8,7 +8,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
8
8
|
`expectedQueryLoop` ONLY: `steps.ts` declares the per-step write one-per-step to the N+1
|
|
9
9
|
detector there, and no client is ever taken from it. Never `http`, `render`, `ui`.
|
|
10
10
|
- Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
|
|
11
|
-
- External deps: none. Postgres access goes through the injected `PgExecutor
|
|
11
|
+
- External deps: none. Postgres access goes through the injected `PgExecutor` (`@ultimat3/core`'s,
|
|
12
|
+
re-exported from `index.ts`).
|
|
12
13
|
|
|
13
14
|
## Rules — registration and declarations
|
|
14
15
|
|
|
@@ -29,7 +30,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
29
30
|
- **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — do not re-litigate.**
|
|
30
31
|
`tenant` is REQUIRED (`(input) => input.orgId` or `'none'`; `X_JOB_TENANT_REQUIRED` backstop).
|
|
31
32
|
`executeJob` puts the derived org on the ctx's actor (`jobRunActor`, org only) and runs the body in
|
|
32
|
-
`runWithContext`; `'none'` STRIPS the org (fail-closed).
|
|
33
|
+
`runWithContext`; `'none'` STRIPS the org (fail-closed).
|
|
33
34
|
`tenantFor` is a METHOD on `JobHandle` (variance). The queue row's `tenantId` stays the ENQUEUER's
|
|
34
35
|
(the limiter's bucket). `tenancy-cross-surface.test.ts` asserts HTTP and job verdicts are EQUAL.
|
|
35
36
|
- **`enqueuedBy` is ATTRIBUTION, never authority** — do not re-litigate. A job acting FOR a user takes
|
|
@@ -43,39 +44,37 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
43
44
|
|
|
44
45
|
## Rules — the driver and SQL
|
|
45
46
|
|
|
46
|
-
- **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `
|
|
47
|
-
`
|
|
48
|
-
`x_job_events`, `x_job_pauses`, `x_job_workers`, `x_job_counters`). A NEW table is also a name in
|
|
47
|
+
- **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`,
|
|
48
|
+
`x_outbox`, `x_scheduler_*`). A NEW table is also a name in
|
|
49
49
|
`packages/cli/src/framework-schema.ts`. A shipped table grows by `alter table ... add column if
|
|
50
|
-
not exists`. Its comments carry NO apostrophes and NO semicolons (
|
|
50
|
+
not exists`. Its comments carry NO apostrophes and NO semicolons (it splits on `;`; `driver-pg-sql.test.ts` checks
|
|
51
51
|
quote parity).
|
|
52
52
|
- **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
|
|
53
|
-
`X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's
|
|
54
|
-
refusal rejects on both.
|
|
53
|
+
`X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's reads are `async` so a refusal rejects on both.
|
|
55
54
|
- **`ack`, `nack`, `heartbeat` and `recordProgress` are FENCED on `state = 'running'` AND the
|
|
56
55
|
CLAIM** — `{ workerId, claim }`, `claimOf(claimed)`; `x_jobs.claims` is moved by the claim and
|
|
57
|
-
never reset.
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
`
|
|
56
|
+
never reset. `heartbeat` answers a boolean, read as `held === false` (never `!held`).
|
|
57
|
+
- **A driver's semantics are pinned beside the pg statement** (`driver-parity*.test.ts`);
|
|
58
|
+
`driver-*-fixture.ts` scenarios run on memory AND Postgres.
|
|
59
|
+
- **The claim BURIES a lease that lapsed on the final attempt** (`attempt >= max_attempts`): `dead`
|
|
60
|
+
in `SQL_CLAIM` — `failed` for a name in `ClaimOptions.dropExhausted` (`retry.deadLetter: false`)
|
|
61
|
+
— counted, never returned; the worker's round hears `onExhausted` and calls `announceExhausted`.
|
|
62
|
+
- **`cancel` is fenced on `LIVE_STATES`.** **`requeue`** refuses (`X_JOB_NOT_FOUND`, not requeueable,
|
|
63
|
+
duplicate), then ONE statement. **`enqueue` inserts twice at most.** **`BulkResult.remaining` is
|
|
64
|
+
what a second call would move.**
|
|
62
65
|
- **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
|
|
63
66
|
(discovered, comments stripped) for `select *`/`returning *`.
|
|
64
67
|
- Drivers implement the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`. New
|
|
65
68
|
capabilities go behind the interface. `inspect.ts` returns plain JSON for CLI, `/_x` and MCP.
|
|
66
|
-
- Step results are persisted BEFORE the step returns. All time is epoch ms
|
|
67
|
-
|
|
68
|
-
- **`createPgLeader` is correct only on a DEDICATED connection**; boot uses `createPgLeaseLeader`.
|
|
69
|
-
- `driver-redis.ts` / `driver-nats.ts` are honest `X_NOT_IMPLEMENTED` stubs.
|
|
69
|
+
- Step results are persisted BEFORE the step returns. All time is epoch ms (`nowMs()`, `clock.ts`).
|
|
70
|
+
- **`postgresLeader` is correct only on a DEDICATED connection**; boot uses `postgresLeaseLeader`.
|
|
70
71
|
|
|
71
72
|
## Rules — numbers and limits
|
|
72
73
|
|
|
73
74
|
- **Every numeric knob is refused when not FINITE** — core's `finiteOption()` (a bound) and
|
|
74
|
-
`finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `
|
|
75
|
-
**`bun run finite-bounds` is a floor, never proof**: `
|
|
76
|
-
|
|
77
|
-
STOP). **A row count is `finiteCount` (min 0)** for driver parity (`list`, `deadLetters`, the
|
|
78
|
-
backfill ledger's `list`, `assertClaimBounds`).
|
|
75
|
+
`finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `jobWorker` reads them.
|
|
76
|
+
**`bun run finite-bounds` is a floor, never proof**: `concurrencyLimiter`'s five numbers are screened
|
|
77
|
+
with min 0 (zero is a HARD STOP). **A row count is `finiteCount` (min 0)**; `retry.attempts` is one with min 1.
|
|
79
78
|
- **`WorkerOptions.concurrency` is read by OWN key** — a queue named `constructor`
|
|
80
79
|
(`worker-slots.test.ts`).
|
|
81
80
|
- **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
|
|
@@ -99,24 +98,21 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
99
98
|
- The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
|
|
100
99
|
job's plain key.
|
|
101
100
|
- **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
|
|
102
|
-
the poll timer,
|
|
101
|
+
the poll timer, arm the drain cut-off — it aborts nothing); `close` is the teardown. `stop()` hands both back in a `finally`;
|
|
103
102
|
`start()` refuses while draining.
|
|
104
|
-
- **A claimed job is counted with core's `beginWork()
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
`
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
drained attempt settles as **`interrupted`** (`nack`, `countsAsAttempt: false`, no park, no dead
|
|
111
|
-
letter), read off the SIGNAL. `WorkerStats.interrupted`; `worker-drain-signal.test.ts`.
|
|
103
|
+
- **A claimed job is counted with core's `beginWork()`**: the drain's in-flight phase waits. The
|
|
104
|
+
teardown's wait is bounded on SIGTERM (`drain-wait.ts`), unbounded on `stop()`; one teardown, joined.
|
|
105
|
+
- **A deploy lets a running job FINISH**: SIGTERM only stops claiming. `drainSignal` aborts
|
|
106
|
+
(`X_DRAINING`) at `deadlineAt − margin` (`worker-drain-cutoff.ts`), then held claims go back
|
|
107
|
+
uncounted (`worker-held.ts`). Settles **`interrupted`**; `docs/history/jobs.md`, "Drain, 2026-10".
|
|
108
|
+
- **`X_RATE_LIMITED` defers uncounted** (`rateLimitDeferralMs`).
|
|
112
109
|
- **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
|
|
113
110
|
synchronously with its guard.
|
|
114
|
-
- **A fleet slot is taken INSIDE a `try`, released AWAITED, and
|
|
115
|
-
run
|
|
111
|
+
- **A fleet slot is taken INSIDE a `try`, released AWAITED, and HELD, not owned**: `false`, or a TTL
|
|
112
|
+
with no renewal landing, CANCELS the run (`X_JOB_SLOT_LOST`); held per job id as a LIST.
|
|
116
113
|
- **The run's signal is a controller this worker owns** (`run-signal.ts`), never `AbortSignal.any`.
|
|
117
|
-
- **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal
|
|
118
|
-
|
|
119
|
-
process's clock, asked both sides of the call.
|
|
114
|
+
- **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal warns; a whole window without
|
|
115
|
+
one landing is `jobs.lease.lost`, on this process's clock, asked both sides of the call.
|
|
120
116
|
- **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
|
|
121
117
|
interval is `unref`ed; every renewal is armed through `WorkerOptions.schedule`.
|
|
122
118
|
- **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
|
|
@@ -125,21 +121,26 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
125
121
|
- **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
|
|
126
122
|
- **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
|
|
127
123
|
the attempt is cancelled in `executeJob`'s `finally`.
|
|
128
|
-
- **A cancelled runner writes NOTHING** — `put()` in `steps.ts`
|
|
129
|
-
`
|
|
124
|
+
- **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`) — save a fenced
|
|
125
|
+
`X_DRAINING` on a live attempt: a step done in the drain is recorded once. The STORE fences
|
|
126
|
+
the claim (`StepFence`, `SQL_STEP_PUT` as `SQL_ACK`: `X_JOB_LEASE_LOST`). `executeJob` passes it.
|
|
130
127
|
- **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
|
|
131
128
|
- **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
|
|
132
129
|
out before `leader.release()`. Same two-hook rule; an ABANDONED round does not release
|
|
133
130
|
(`jobs.scheduler.drain-abandoned`). `stillLeading()` is asked every round and before every task,
|
|
134
131
|
and answers from memory inside `renewEveryMs`.
|
|
135
|
-
- **`run-once` fires ONE catch-up
|
|
136
|
-
|
|
132
|
+
- **`run-once` fires ONE catch-up**: the watermark rides the fire (`ScheduledFire.watermarkMs`,
|
|
133
|
+
`SQL_SCHEDULER_FIRE` `$4`), never a `markFired` behind it; `skip` fires the true latest missed
|
|
134
|
+
occurrence (`latestOccurrenceBy`, bisection).
|
|
135
|
+
- **A lease for a run that never STARTED is `abandon()`ed, never `release()`d** — the rate stamp
|
|
136
|
+
goes with it (`worker-admit.ts`). **A failed pass re-arms at the floor** (`worker-loop.ts`
|
|
137
|
+
catch); `timers` is the test seam.
|
|
137
138
|
- **Every timer body catches before it finalises** (`void work().catch(log).finally(...)`).
|
|
138
139
|
- **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
|
|
139
140
|
`nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
|
|
140
141
|
stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
|
|
141
|
-
- **A
|
|
142
|
-
|
|
142
|
+
- **A wiring throw hands back the run's holdings, and its ROW until `executeJob` starts**
|
|
143
|
+
(`worker-run.ts`), as a failed shed does its job (`worker-admit.ts`).
|
|
143
144
|
- **`step.run` hydrates the run's steps from ONE `store.list(runId)`**; `sleep` and `waitForEvent`
|
|
144
145
|
record replays through `trace()`. **`claimName` reads a Set**; `usedNames()` keeps
|
|
145
146
|
`MAX_TRACE_NAMES` (200).
|
|
@@ -153,9 +154,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
153
154
|
suite (memory + real pg).
|
|
154
155
|
- **A settle answers whether it landed**; a miss is `jobs.settle.unowned`, not a throw.
|
|
155
156
|
- **The counter moves in the SETTLING statement** (`driver-pg-settle-sql.ts`): one-minute bucket per
|
|
156
|
-
job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing)
|
|
157
|
-
|
|
158
|
-
(`rollupCounters`), at most once a bucket.
|
|
157
|
+
job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing).
|
|
158
|
+
The scheduler leader folds tiers (`rollupCounters`), at most once a bucket.
|
|
159
159
|
- **`list()` is KEYSET** (`after: jobCursor(lastRow)`, newest first by `(created_at, id)`); the pg
|
|
160
160
|
seek reads the cursor row's own `created_at` — the cursor's ms is rounded. A page past
|
|
161
161
|
`MAX_JOB_PAGE` or a foreign cursor is `X_JOB_PAGE_INVALID`, never a bare invariant.
|
|
@@ -166,10 +166,9 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
166
166
|
- **An occurrence fires in ONE statement** (`SchedulerState.fire`, `SQL_SCHEDULER_FIRE`): the
|
|
167
167
|
watermark is the fence, jobs insert only `where exists (select 1 from moved)`.
|
|
168
168
|
- **`progress()` is throttled in `progress.ts`** and flushed before every settle. **`onSettled`
|
|
169
|
-
(`settled.ts`) is the ONE ending hook** — `completed`
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
crash. Never for a cancel, a retry or a suspension; no second hook beside it.
|
|
169
|
+
(`settled.ts`) is the ONE ending hook** — `completed`, `dead-lettered`, `dropped`, `refused` —
|
|
170
|
+
AFTER a settle that LANDED, in the body's tenant scope, `ON_SETTLED_ATTEMPTS` tries, never
|
|
171
|
+
rethrown, at most once across a crash. Never for a cancel, a retry or a suspension.
|
|
173
172
|
- **Failed for good + `deadLetter: false` is DROPPED**: `nack({ fail })`, outcome `dropped`.
|
|
174
173
|
- **Worker registry rows expire** (`worker-registry.ts`, TTL = visibility timeout); `stop()` forgets.
|
|
175
174
|
|
|
@@ -196,7 +195,9 @@ Long form: `docs/history/jobs.md`, "the cross-process wake".
|
|
|
196
195
|
- **`idle-cost.test.ts` / `queue-wake.test.ts` count statements per idle minute** — scheduler 6,
|
|
197
196
|
worker and relay 30 with no wake, 12 with. Raising one is a regression.
|
|
198
197
|
- **The memory driver stores a payload's JSON form**, as pg binds it (`driver-settle-parity.test.ts`).
|
|
199
|
-
- **The event bus has ONE clock**: `
|
|
198
|
+
- **The event bus has ONE clock**: `postgresEventBus` takes no `clock`; `EventLookup.now()` is
|
|
199
|
+
REQUIRED, stamps a NEW wait, and calls the TIMEOUT (asked once the runner thinks time is up).
|
|
200
|
+
`purgeExpired()` is awaited and counted; `eventsPurgeTarget(bus)` is its `PurgeTarget`.
|
|
200
201
|
|
|
201
202
|
## Rules — the outbox
|
|
202
203
|
|
|
@@ -225,8 +226,8 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
225
226
|
- **`purge()`** is the one caller of every `purgeExpired()`: one table per `step.run`, one clock
|
|
226
227
|
reading per pass, duplicate names refused.
|
|
227
228
|
- **`exportRows()`**: one object per PAGE, named by page index; NO cross-tenant escape.
|
|
228
|
-
- **`webhook()
|
|
229
|
-
throw. **The wire format is core's
|
|
229
|
+
- **`webhook()`**: ONE event to ONE endpoint, no steps; each attempt but a CANCELLED one recorded
|
|
230
|
+
before the throw; `ledger.isDisabled` gates the socket. **The wire format is core's.**
|
|
230
231
|
- A `-fixture.ts` file does not ship; `backfill-pass-fixture.ts` raises a plain `Error` subclass on
|
|
231
232
|
purpose (it stands in for app code).
|
|
232
233
|
|
|
@@ -238,7 +239,7 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
238
239
|
| `tenant.ts` | what tenant a run acts under: `JobTenant`, the declaration's backstop, the actor `executeJob` installs |
|
|
239
240
|
| `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input |
|
|
240
241
|
| `backfill-pass.ts` | one pass: the batched iteration, its cursor checkpoints and its ledger row |
|
|
241
|
-
| `backfill-scope.ts` | which sweeps run across tenants
|
|
242
|
+
| `backfill-scope.ts` | which sweeps run across tenants (`tenant: 'none'` only) |
|
|
242
243
|
| `backfill-ledger.ts` | `x_backfills` — the contract, `BACKFILL_STATUSES`, the checksum, the verdict, the memory ledger |
|
|
243
244
|
| `backfill-registry.ts` | what was DECLARED: the `origin` stamp, `isBackfill`, `registeredBackfills` |
|
|
244
245
|
| `backfill-gate.ts` | may this sweep run here and now — environment, `requires`, already-applied |
|
|
@@ -249,38 +250,39 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
249
250
|
| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements; an `action-job` projection is `X_ACTION_JOB_UNBRIDGED` |
|
|
250
251
|
| `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
|
|
251
252
|
| `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
|
|
253
|
+
| `steps-timeout.ts` / `steps-suspension.ts` | a step's ceiling; `StepSuspension` and the timed-out marker |
|
|
252
254
|
| `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
|
|
253
255
|
| `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
|
|
254
|
-
| `outbox-pg.ts` | `
|
|
256
|
+
| `outbox-pg.ts` | `postgresOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
|
|
255
257
|
| `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
|
|
256
|
-
| `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one
|
|
258
|
+
| `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
|
|
257
259
|
| `concurrency.ts` | `job.concurrency` as declared, resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
|
|
258
|
-
| `errors-concurrency.ts` | every concurrency refusal's class
|
|
260
|
+
| `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `errors.ts` |
|
|
259
261
|
| `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
|
|
260
262
|
| `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
|
|
261
|
-
| `scheduler-pg.ts` | `
|
|
262
|
-
| `events-pg.ts` | `
|
|
263
|
+
| `scheduler-pg.ts` | `postgresSchedulerState` (the durable watermark, the atomic fire) + `postgresLeaseLeader` |
|
|
264
|
+
| `events-pg.ts` | `postgresEventBus` — `step.waitForEvent` across processes |
|
|
263
265
|
| `driver.ts` | `JobDriver` contract + wire records |
|
|
264
|
-
| `driver-pg.ts` | default driver, and `
|
|
266
|
+
| `driver-pg.ts` | default driver, and `postgresLeader` |
|
|
265
267
|
| `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the ONE install point. Its comments carry no `;` and no `'` |
|
|
266
268
|
| `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and `JOB_ROW_COLUMNS`; re-exported from `driver-pg-sql.ts` |
|
|
267
269
|
| `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
|
|
268
|
-
| `driver-memory.ts` | `x dev` / tests |
|
|
269
|
-
| `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
|
|
270
|
+
| `driver-memory.ts` / `claim-exhausted.ts` | `x dev` / tests; a buried row's log line and `onSettled` |
|
|
270
271
|
| `retry.ts` | the dead-letter decision, and this package's option names over core's `backoffDelay` — no curve of its own |
|
|
271
|
-
| `retry-classification.ts` | the
|
|
272
|
+
| `retry-classification.ts` | the other half: what the thrown error says, and the stop reason |
|
|
272
273
|
| `execute.ts` / `run-deadline.ts` | `executeJob` — one claimed job run and settled; the run's deadline: cancel, then fail |
|
|
273
274
|
| `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
|
|
274
|
-
| `renewal-timer.ts` | the
|
|
275
|
-
| `worker.ts` | `worker` role, claim
|
|
275
|
+
| `renewal-timer.ts` | the renewal interval and its `stopped()` latch |
|
|
276
|
+
| `worker.ts` | `worker` role, claim round, drain |
|
|
277
|
+
| `worker-drain-cutoff.ts` / `worker-held.ts` / `worker-tally.ts` | drain cut-off · held claims · counters |
|
|
276
278
|
| `worker-admit.ts` | may this claimed job start: the limiter, the fleet slot, and the shed / refusal |
|
|
277
279
|
| `worker-registry.ts` | one worker's registry row: announce, heartbeat, forget |
|
|
278
280
|
| `worker-types.ts` | the worker's public contract: `WorkerOptions`, `WorkerStats`, `Worker` |
|
|
279
|
-
| `drain-wait.ts` | the drain's wait, shared by both roles
|
|
280
|
-
| `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span
|
|
281
|
-
| `run-signal.ts` | the signal ONE run is cancelled by
|
|
281
|
+
| `drain-wait.ts` | the drain's wait, shared by both roles |
|
|
282
|
+
| `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span |
|
|
283
|
+
| `run-signal.ts` | the signal ONE run is cancelled by |
|
|
282
284
|
| `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back — and the boot refusal |
|
|
283
|
-
| `purge.ts` | `purge()` — a factory over `job()`: the retention sweep
|
|
285
|
+
| `purge.ts` | `purge()` — a factory over `job()`: the retention sweep and its target seam |
|
|
284
286
|
| `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
|
|
285
287
|
| `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
|
|
286
288
|
| `limits.ts` | per-tenant / per-queue / global concurrency + rate |
|
|
@@ -298,16 +300,11 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
298
300
|
| `queue-wake.ts` / `driver-pg-wake-sql.ts` | the `LISTEN` session and its proof; which statement notifies |
|
|
299
301
|
| `worker-loop.ts` | the claim loop's timer: wake, `again`, the kicks |
|
|
300
302
|
| `driver-pg-outbox-sql.ts` | the `x_outbox` statements |
|
|
301
|
-
| `errors-operator.ts`
|
|
303
|
+
| `errors-operator.ts` / `errors-requeue.ts` | the operator verbs' refusals; `requeue`'s |
|
|
302
304
|
|
|
303
305
|
Which `-fixture.ts` harness each suite shares, and what each `.job.` suite proves:
|
|
304
306
|
`docs/history/jobs.md`, "Moved 2026-10-01 — test harnesses".
|
|
305
307
|
|
|
306
|
-
|
|
307
|
-
|
|
308
|
-
```
|
|
309
|
-
bun test packages/jobs
|
|
310
|
-
bun run --filter @ultimat3/jobs typecheck
|
|
311
|
-
```
|
|
308
|
+
Commands: `bun test packages/jobs` · `bun run --filter @ultimat3/jobs typecheck`.
|
|
312
309
|
|
|
313
310
|
Why each rule above is shaped the way it is: [`docs/history/jobs.md`](../../docs/history/jobs.md).
|
package/README.md
CHANGED
|
@@ -192,11 +192,17 @@ input. The tenant is a fact about the work, so the job declares it.
|
|
|
192
192
|
expires, so on a bus shared by one `bun test` process an answer one test published resumes the
|
|
193
193
|
next test's wait — call it between tests.
|
|
194
194
|
|
|
195
|
-
The event bus has ONE clock. `
|
|
195
|
+
The event bus has ONE clock. `postgresEventBus({ executor })` stamps `publishedAt` and the expiry
|
|
196
196
|
with the database's `now()` — it takes no `clock` — and `bus.now()` answers that same clock. A
|
|
197
197
|
consumer that wants "published after I asked" (`@ultimat3/scraping`'s `eventPrompt`) reads its
|
|
198
198
|
"asked at" from `bus.now()`, never from its own process: a skew between two pods must not decide
|
|
199
|
-
which answers count.
|
|
199
|
+
which answers count. `step.waitForEvent` stamps a new wait from the same clock
|
|
200
|
+
(`EventLookup.now()`), so a worker running ahead of the database still matches an event published
|
|
201
|
+
after it asked.
|
|
202
|
+
`step.waitForEvent` stamps a new wait from that clock (`EventLookup.now()`, required) and asks it
|
|
203
|
+
again before declaring a timeout, so a worker running ahead of the database still matches an event
|
|
204
|
+
published after it asked and does not give up early. `eventsPurgeTarget(bus)` is the bus as a
|
|
205
|
+
`purge()` target: `purgeExpired()` is awaited and answers the rows it removed.
|
|
200
206
|
|
|
201
207
|
Step names are the replay key, so they must be deterministic and unique in a run — a
|
|
202
208
|
duplicate is `X_STEP_DUPLICATE`, not a silent overwrite. Suspension is control flow
|
|
@@ -405,6 +411,11 @@ export const hourly = task({
|
|
|
405
411
|
| at least once is safe here | a replayed delete removes rows that are already gone, and a row a purge deleted answers exactly as one that was never there |
|
|
406
412
|
| two targets under one name | `X_INVARIANT`, before the first delete — `step.run` would raise `X_STEP_DUPLICATE` after one table was already empty |
|
|
407
413
|
|
|
414
|
+
The stored event bus is a target too: `eventsPurgeTarget(bus)` answers
|
|
415
|
+
`{ name: EVENTS_PURGE_TARGET, purgeExpired }` over `bus.purgeExpired()`, which is awaited and
|
|
416
|
+
answers the rows it removed — `x_job_events` only grew while nothing called it. Add it to
|
|
417
|
+
`targets`: `targets: () => [eventsPurgeTarget(eventBus())]`.
|
|
418
|
+
|
|
408
419
|
`@ultimat3/cli`'s boot declares both halves over the three tables it owns, so an app gets the sweep
|
|
409
420
|
without writing any of the above. It needs a `worker` to run it and a `scheduler` to fire it: a
|
|
410
421
|
deployment with neither has no background work at all, and this is one more thing it does not do.
|
|
@@ -526,9 +537,11 @@ x-ultimate-webhook-signature: t=1700000000,v1=<hex hmac-sha256>
|
|
|
526
537
|
|
|
527
538
|
The receiving half is `verifyWebhookSignature(request, { secret })` in `@ultimat3/http`. The format
|
|
528
539
|
itself — the canonical string, the mac and the header names — is **one module in `@ultimat3/core`**
|
|
529
|
-
(`webhook-signature.ts`),
|
|
530
|
-
signs, `http` verifies, and neither may import the other, so the one copy
|
|
531
|
-
can reach.
|
|
540
|
+
(`webhook-signature.ts`), imported from `@ultimat3/core` by both packages and re-declared by
|
|
541
|
+
neither: this package signs, `http` verifies, and neither may import the other, so the one copy
|
|
542
|
+
lives at the tier both can reach. A receiver or a test imports `WEBHOOK_SIGNATURE_HEADER`,
|
|
543
|
+
`webhookSignature` and the rest from `@ultimat3/core` — 25.0.0 dropped this package's
|
|
544
|
+
re-exports (`X_HELPER_COPY`). Same argument `timing-safe-equal.ts` makes for itself.
|
|
532
545
|
|
|
533
546
|
| Rule | Why |
|
|
534
547
|
|---|---|
|
|
@@ -539,10 +552,21 @@ can reach. Same argument `timing-safe-equal.ts` makes for itself.
|
|
|
539
552
|
| `redirect: 'manual'` | following a 3xx would re-POST a body signed for one host to whatever the receiver named |
|
|
540
553
|
| the endpoint is never checkpointed | a `step.run` output lands in `x_job_steps`, and the endpoint carries the secret |
|
|
541
554
|
| every attempt is recorded **before** the throw | a failure the ledger cannot see is a failure the consecutive count cannot see, which is an endpoint that never gets disabled |
|
|
542
|
-
| a `Retry-After` the receiver names is honoured | `X_WEBHOOK_DELIVERY_THROTTLED` carries `meta.retryAfterSeconds`, which the nack waits out (clamped by `retry.maxDelay`
|
|
555
|
+
| a `Retry-After` the receiver names is honoured | `X_WEBHOOK_DELIVERY_THROTTLED` carries `meta.retryAfterSeconds`, which the nack waits out — read by core's `retryAfterSecondsOf` (delta-seconds, or an HTTP-date against the response's own `Date`; `0` or a past date is unstated), clamped by `retry.maxDelay`, plus core's `jitterStatedDelay` spread so deliveries handed one instant do not wake together |
|
|
556
|
+
| `WebhookLedger.isDisabled(endpointId)` is asked before every socket, beside the row's own `disabled` | `disableAfter` writes only the ledger: a verdict nobody asks for again is an endpoint that keeps receiving POSTs. Answer it from the column your `disable` sets |
|
|
557
|
+
| the deadline and a lost lease abort `fetch`; a failure that abort caused writes no ledger row | the nack behind them hands the job to another worker at once, so a request left open is a duplicate delivery, and an abandoned attempt's failure would count toward `disableAfter` |
|
|
558
|
+
| the DRAIN never aborts a POST already on the wire; it stops one not yet sent | torn down, the job goes to the next pod, which re-POSTs the same event on every deploy. Finished, it is recorded like any other outcome — a real failure during the drain included |
|
|
543
559
|
| re-enabling is always yours | an endpoint the framework un-disabled on its own is a retry loop with no end |
|
|
544
560
|
| `WebhookLedger` is a seam, not a table | retention is seven years for one business and thirty days for the next, so shipping a schema would ship one of those answers |
|
|
545
561
|
|
|
562
|
+
**Set `timeout` below the drain budget.** A webhook has no timeout unless you declare one, and a
|
|
563
|
+
POST on the wire when SIGTERM lands runs until the receiver answers. With a `timeout`, a drained
|
|
564
|
+
request is still torn down at the attempt's deadline — where `executeJob` hands the job on — and
|
|
565
|
+
writes no ledger row. Without one, the only bound is `configureLifecycle({ deadlineMs })`, 25 s by
|
|
566
|
+
default: past it the worker logs `jobs.worker.drain-abandoned` and exits, and the row comes back
|
|
567
|
+
only when its lease lapses, as a COUNTED attempt the receiver may already have processed.
|
|
568
|
+
`timeout: '10s'` against the default budget ends every drained delivery inside it.
|
|
569
|
+
|
|
546
570
|
## The deadline cancels
|
|
547
571
|
|
|
548
572
|
A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that
|
|
@@ -659,15 +683,15 @@ things have to be true in a process:
|
|
|
659
683
|
| Step | Call |
|
|
660
684
|
|---|---|
|
|
661
685
|
| the table exists | ships in `SQL_JOBS_TABLE` — applying the queue DDL is enough |
|
|
662
|
-
| the facade is installed | `setJobsFacade(
|
|
663
|
-
| the relay is running | `
|
|
686
|
+
| the facade is installed | `setJobsFacade(outboxJobsFacade({ store, driver }, currentTx))` |
|
|
687
|
+
| the relay is running | `outboxRelay({ store, driver }).start()` |
|
|
664
688
|
|
|
665
|
-
`start()` registers the same two shutdown hooks `
|
|
689
|
+
`start()` registers the same two shutdown hooks `jobWorker` does — `accept` stops polling
|
|
666
690
|
(the worker's also aborts every held job's `ctx.signal`), `close` waits out the pass in flight
|
|
667
691
|
under the drain's deadline — and `stop()` hands both back.
|
|
668
692
|
`drainOnShutdown: false` opts out, for a caller that drives its own teardown.
|
|
669
693
|
|
|
670
|
-
with `store =
|
|
694
|
+
with `store = postgresOutboxStore({ executor, txExecutor })`. `txExecutor` is what makes it
|
|
671
695
|
transactional: `stage()` runs on the CALLER'S connection, never the pool. With nothing installed,
|
|
672
696
|
`jobsFacade()` answers a fallback whose `currentTx` is `() => undefined` and every enqueue
|
|
673
697
|
publishes straight to the driver — deliberate, so a script and a test enqueue with no wiring, but
|
|
@@ -702,7 +726,7 @@ What the lease buys, precisely:
|
|
|
702
726
|
| a relay that died mid-batch stranding its rows forever | anything a **non-idempotent** handler does on its second run |
|
|
703
727
|
| — | a lapsed visibility lease re-delivering a job that is still running |
|
|
704
728
|
|
|
705
|
-
The memory store (`
|
|
729
|
+
The memory store (`memoryOutboxStore`, `x dev` and tests) **drops** a published row —
|
|
706
730
|
`retained()` is the relay's backlog, not a running total; the pg store keeps `published_at` as
|
|
707
731
|
the audit trail this map is not. A relay pass that throws is logged as `jobs.outbox.tick-failed`
|
|
708
732
|
and the loop re-arms: an unobserved rejection would end the process with rows still staged.
|
|
@@ -719,17 +743,35 @@ One interface: `enqueue`, `claim` (visibility timeout), `ack(jobId, { workerId,
|
|
|
719
743
|
`backfills` and `leases`. Both settles are fenced on the CLAIM — `claimOf(claimed)` is the
|
|
720
744
|
`{ workerId, claim }` a `ClaimedJob` carries — and answer whether they landed. Zero job-code change
|
|
721
745
|
between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driver` config line**:
|
|
722
|
-
`JobsConfig.driver` has no reader and boot always builds `
|
|
746
|
+
`JobsConfig.driver` has no reader and boot always builds `postgresJobDriver`.
|
|
723
747
|
|
|
724
748
|
| Driver | Status | Backing | Use |
|
|
725
749
|
|---|---|---|---|
|
|
726
750
|
| `pg` | **default** | `SELECT ... FOR UPDATE SKIP LOCKED`, a partial unique index on `(name, coalesce(tenant_id, ''), idempotency_key)` over live rows (`x_jobs_name_tenant_idempotency_live_idx`), lease-based leader, `x_job_leases` | zero-infra start, most apps |
|
|
727
751
|
| `memory` | complete | in-process maps | `x dev`, tests |
|
|
728
|
-
| `redis` | interface-complete, `X_NOT_IMPLEMENTED` | Streams + consumer groups | planned |
|
|
729
|
-
| `nats` | interface-complete, `X_NOT_IMPLEMENTED` | JetStream work queue | planned |
|
|
730
752
|
|
|
731
|
-
|
|
732
|
-
|
|
753
|
+
There is **no NATS or Redis jobs driver**: `createNatsDriver` and `createRedisDriver`, all-throw
|
|
754
|
+
stubs, were deleted in 25.0.0. A real `redisJobDriver()` would arrive as a minor.
|
|
755
|
+
|
|
756
|
+
The pg SQL lives verbatim in `src/driver-pg-*sql.ts` (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …)
|
|
757
|
+
so an agent debugging a stuck queue can read and run the exact statement. The barrel exports
|
|
758
|
+
only the statement another package runs: `SQL_JOBS_TABLE` (the install) — 25.0.0 took the rest
|
|
759
|
+
off, `SQL_CLAIM` and `SQL_OUTBOX_RELEASE` included (their array-parameter proofs are this package's
|
|
760
|
+
`driver-pg-array.live.test.ts`).
|
|
761
|
+
|
|
762
|
+
**A lease that lapses on a row's final attempt is buried by the claim.** `claim` re-takes a
|
|
763
|
+
`running` row whose lease lapsed — the worker died — but one already at `attempt >= maxAttempts`
|
|
764
|
+
is settled `dead` in the claim statement itself (`SQL_CLAIM`'s second arm; `isFinalAttempt` on the
|
|
765
|
+
memory driver), with `lastError` set to `LEASE_LAPSED_FINAL_ATTEMPT`, counted `dead`, and never
|
|
766
|
+
handed out: a job that kills its worker reaches the dead-letter queue after `retry.attempts`
|
|
767
|
+
claims instead of taking a worker per visibility timeout, forever. The pass is told through
|
|
768
|
+
`ClaimOptions.onExhausted(rows)` — at most once per pass, once per row — and `announceExhausted`
|
|
769
|
+
(`claim-exhausted.ts`) is what a claim round does with them: `jobs.claim.exhausted` at `error`, and
|
|
770
|
+
the job's `onSettled`, and the worker's claim round does exactly that. The row carries
|
|
771
|
+
`maxAttempts` and not the policy, so the CALLER names the jobs declaring `retry.deadLetter: false`
|
|
772
|
+
(`ClaimOptions.dropExhausted` — the worker reads them off its registry): those rows are buried
|
|
773
|
+
`failed` and announced `dropped`, every other one `dead` / `dead-lettered`. A caller that names
|
|
774
|
+
none buries every such row `dead`.
|
|
733
775
|
|
|
734
776
|
A driver you write reads a nack's target state through `nackState(options)` — `dead`
|
|
735
777
|
(`deadLetter`), `failed` (`fail`, terminal and out of the dead-letter queue), `suspended` (`park`)
|
|
@@ -740,8 +782,8 @@ or `ready` — so a new branch cannot land in one driver and not another. Its le
|
|
|
740
782
|
|
|
741
783
|
| Role | Entry | Behaviour |
|
|
742
784
|
|---|---|---|
|
|
743
|
-
| `worker` | `
|
|
744
|
-
| `scheduler` | `
|
|
785
|
+
| `worker` | `jobWorker({ driver, context, queues, concurrency })` | per-queue pools, lease heartbeat, SIGTERM drain: stop claiming → finish in-flight → close. `concurrency` is a number for every queue or a table per queue; a queue it does not name gets core's `JOBS_CONCURRENCY_DEFAULT` (8 — this package's own default was 5 before 25.0.0) |
|
|
786
|
+
| `scheduler` | `jobScheduler({ driver, leader, state })` | one dispatch round at a time, catch-up policy, SIGTERM drain: stop dispatching → finish the round → release the lock |
|
|
745
787
|
|
|
746
788
|
Both roles register the same **pair** of hooks and bound the `close` half the same way. The
|
|
747
789
|
scheduler's abandoned case differs in one respect: a round still enqueueing keeps the lease rather
|
|
@@ -773,24 +815,24 @@ update runs two leaders.
|
|
|
773
815
|
|
|
774
816
|
```ts
|
|
775
817
|
import {
|
|
776
|
-
|
|
777
|
-
|
|
818
|
+
postgresLeaseLeader,
|
|
819
|
+
jobScheduler,
|
|
778
820
|
type JobDriver,
|
|
779
821
|
type PgExecutor,
|
|
780
|
-
|
|
822
|
+
postgresSchedulerState,
|
|
781
823
|
} from '@ultimat3/jobs';
|
|
782
824
|
|
|
783
825
|
declare const driver: JobDriver;
|
|
784
826
|
declare const executor: PgExecutor; // see "A `PgExecutor`" below
|
|
785
827
|
|
|
786
|
-
|
|
828
|
+
jobScheduler({
|
|
787
829
|
driver,
|
|
788
|
-
state:
|
|
789
|
-
leader:
|
|
830
|
+
state: postgresSchedulerState(executor),
|
|
831
|
+
leader: postgresLeaseLeader({ executor }),
|
|
790
832
|
});
|
|
791
833
|
```
|
|
792
834
|
|
|
793
|
-
`
|
|
835
|
+
`postgresLeaseLeader` and not `postgresLeader`: `pg_try_advisory_lock` is scoped to a Postgres
|
|
794
836
|
*session*, and the executor this package is handed is a **pool** — the lock is released the moment
|
|
795
837
|
that connection goes back to it, so every node reads itself as leader. The lease is a row with an
|
|
796
838
|
expiry and needs no connection affinity. `acquire()` is also the renewal, called every round, which
|
|
@@ -830,10 +872,10 @@ hourly task fire 24 catch-ups a second apart.
|
|
|
830
872
|
client, which already speaks `(text, values)` and decodes `jsonb`:
|
|
831
873
|
|
|
832
874
|
```ts
|
|
833
|
-
import {
|
|
875
|
+
import { postgresClient } from '@ultimat3/db';
|
|
834
876
|
import type { PgExecutor } from '@ultimat3/jobs';
|
|
835
877
|
|
|
836
|
-
const client =
|
|
878
|
+
const client = postgresClient({ url: 'postgres://localhost:5432/app_test' });
|
|
837
879
|
export const executor: PgExecutor = {
|
|
838
880
|
query: <R>(text: string, values: readonly unknown[]) => client.query<R>({ text, values }),
|
|
839
881
|
};
|
|
@@ -861,11 +903,11 @@ A queue with nothing to do costs almost nothing, and a job starts when it is com
|
|
|
861
903
|
`startQueueWake({ listener, executor })` holds ONE `LISTEN` session per worker pod and turns "another process committed" into the signal the idle loops already hear. The boot starts it for the `worker` role.
|
|
862
904
|
|
|
863
905
|
```ts
|
|
864
|
-
import {
|
|
906
|
+
import { postgresClient } from '@ultimat3/db';
|
|
865
907
|
import type { PgExecutor } from '@ultimat3/jobs';
|
|
866
908
|
import { startQueueWake } from '@ultimat3/jobs';
|
|
867
909
|
|
|
868
|
-
const client =
|
|
910
|
+
const client = postgresClient({ url: 'postgres://localhost:5432/app_test' });
|
|
869
911
|
const executor: PgExecutor = {
|
|
870
912
|
query: <R>(text: string, values: readonly unknown[]) => client.query<R>({ text, values }),
|
|
871
913
|
};
|
|
@@ -934,7 +976,7 @@ exactly as it had before.
|
|
|
934
976
|
| Thrown | What the queue does |
|
|
935
977
|
|---|---|
|
|
936
978
|
| a `terminal` code (`X_SCRAPE_AUTH_FAILED`, a validation fault, a permission denial) | dead-lettered on the attempt it happened, `attempt` recorded, remaining attempts unspent — a rotated password retried five times is five more wrong passwords at a site that locks the account after three |
|
|
937
|
-
| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay` — instead of the backoff. Still an attempt, still under the ceiling |
|
|
979
|
+
| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay`, plus core's `jitterStatedDelay` spread unless `jitter: false` — instead of the backoff. Still an attempt, still under the ceiling |
|
|
938
980
|
| a `retryable` code (`X_TIMEOUT`, `X_DRAINING`) | the backoff schedule above, unchanged |
|
|
939
981
|
| an **unclassified** code, or anything that is not an `UltimateError` | the backoff schedule above, unchanged. Most codes are unclassified and `retryFor` answers `terminal` for all of them, so reading that would have stopped every transient retry in every app |
|
|
940
982
|
|
|
@@ -965,7 +1007,7 @@ contractual rate.
|
|
|
965
1007
|
`job.concurrency` is the one that is fleet-wide, and it is enforced by a row every replica can
|
|
966
1008
|
see — one per held slot, keyed `job:<name>`, renewed by the same heartbeat that renews the
|
|
967
1009
|
visibility lease and reclaimed by TTL when a worker is SIGKILLed. A driver with no lease store
|
|
968
|
-
cannot hold the cap, so `
|
|
1010
|
+
cannot hold the cap, so `jobWorker().start()` **refuses to boot**
|
|
969
1011
|
(`X_JOB_CONCURRENCY_UNENFORCEABLE`) rather than let a documented guarantee do nothing.
|
|
970
1012
|
|
|
971
1013
|
Over any cap the claim is handed straight back without burning an attempt — one org's 50k-row
|
|
@@ -1003,7 +1045,7 @@ export const syncAccount = job({
|
|
|
1003
1045
|
|---|---|---|
|
|
1004
1046
|
| a cap of `0`, negative, fractional, `NaN` or `Infinity` — `concurrency: 0` and `limit: 0` alike; a `key` that is not a function; a `whenBusy` outside the two | where the job is declared | `X_JOB_DECLARATION_INVALID` |
|
|
1005
1047
|
| `key(input)` answering `''`, a non-string, or more than 200 characters | at that **enqueue** — the facade and the scheduler both ask | `X_JOB_DECLARATION_INVALID` |
|
|
1006
|
-
| a driver with no lease store, for any `concurrency` | `
|
|
1048
|
+
| a driver with no lease store, for any `concurrency` | `jobWorker().start()` | `X_JOB_CONCURRENCY_UNENFORCEABLE` |
|
|
1007
1049
|
| a `LeaseStore` with no `holders(key)` | the build — the member is required | `TS2741` |
|
|
1008
1050
|
|
|
1009
1051
|
What `'fail'` decides, exactly:
|
|
@@ -1071,7 +1113,9 @@ the pg and the memory driver alike. `x jobs`, `/_x` and a dashboard read nothing
|
|
|
1071
1113
|
|---|---|---|
|
|
1072
1114
|
| `list(filter)` | one page, newest first; `filter.after = jobCursor(lastRow)` for the next, `filter.before = jobCursor(firstRow)` for the previous (the rows nearest the cursor, still newest first; never both). Filters: `queue`, `name`, `state`, `tenantId`, `idPrefix`, `createdFrom`, `createdTo` | `MAX_JOB_PAGE` = 200 rows; `DEFAULT_JOB_PAGE` = 100. A page shorter than its limit is the last. A larger `limit`, or a cursor no page produced, is `X_JOB_PAGE_INVALID` |
|
|
1073
1115
|
| `remove(id)` | delete one job and its steps. A running one is `X_JOB_NOT_REMOVABLE` | — |
|
|
1074
|
-
| `
|
|
1116
|
+
| `requeue(id, { fromStep? })` | re-queue ONE finished job (`REQUEUEABLE_STATES`). A live one is `X_JOB_NOT_REQUEUEABLE`, a key a live job holds `X_JOB_DUPLICATE`, an id nobody queued `X_JOB_NOT_FOUND` — on both drivers. `fromStep` drops that step and every later one in the statement that moves the row | — |
|
|
1117
|
+
| `cancel(id, reason?)` | stop a LIVE job (`LIVE_STATES`: ready, delayed, running, suspended). A done, failed, dead or cancelled row is left exactly as it ended and answers `undefined` — `cancelJob` turns that into `X_JOB_NOT_CANCELLABLE` | — |
|
|
1118
|
+
| `requeueMany({ state, queue?, name?, tenantId? })` | re-queue finished rows; a row whose key a live job holds stays, and of several finished rows sharing one key the oldest is requeued | `MAX_BULK_ROWS` = 1,000 per call, spent only on rows it can move; answers `{ affected, remaining }`, where `remaining` is what a second call could still move — a held row is not counted, so "call until zero" ends |
|
|
1075
1119
|
| `removeMany({ state, queue?, name?, tenantId? })` | delete rows in any state but `running` | the same |
|
|
1076
1120
|
| `promoteMany({ state, queue?, name?, tenantId? })` | `promote` over a set: every row still waiting on its `runAt` is due now. `state` is `delayed` or `ready` (`PROMOTABLE_STATES`) | the same; `remaining` is the rows still waiting |
|
|
1077
1121
|
| `promote(id)` | a job waiting on its `runAt` — delayed, or backing off — is due now | — |
|
|
@@ -1089,7 +1133,6 @@ the pg and the memory driver alike. `x jobs`, `/_x` and a dashboard read nothing
|
|
|
1089
1133
|
| the counter moves in the statement that settles the row | a count that can disagree with the rows is worse than none — and no second round trip per job |
|
|
1090
1134
|
| a shed, a suspension and a drained attempt add nothing | they are handed back uncounted; counted, every `step.sleep` would read as a failure |
|
|
1091
1135
|
| `ack` / `nack` / `heartbeat` / `recordProgress` are fenced on the CLAIM | `{ workerId, claim }`: a body whose lease lapsed must not settle, renew or report on the run that replaced it — claimed by another worker, or by the SAME one, whose id did not change. A miss answers `false` and logs `jobs.settle.unowned` |
|
|
1092
|
-
| `ack(id, { …, counted: false })` adds nothing to the counters | `x jobs drain` settles the row it moved: `done` here, and not a completed run |
|
|
1093
1136
|
| a pause is a row the claim reads | never a column on every queued row: pausing a million-row queue takes effect within one poll |
|
|
1094
1137
|
| an occurrence fires in one statement | the watermark and its jobs move together (`SchedulerState.fire`), so a crash between them cannot fire it twice |
|
|
1095
1138
|
|
|
@@ -1133,6 +1176,7 @@ export const syncAccountWatched = job({
|
|
|
1133
1176
|
|---|---|---|---|
|
|
1134
1177
|
| the body returned and the ack landed | `completed` | `input`, `result` — what `run` returned, typed from it | `done` |
|
|
1135
1178
|
| failed for good, parked in the dead-letter queue | `dead-lettered` | `input` (or `undefined`: the stored payload no longer parses), `error`, `code` | `dead` |
|
|
1179
|
+
| its lease lapsed on the final attempt and the claim buried it — every worker that took it died | `dead-lettered` | `input`, `error` (`LEASE_LAPSED_FINAL_ATTEMPT`), `code: undefined` — nothing was thrown | `dead` |
|
|
1136
1180
|
| failed for good on `retry.deadLetter: false` | `dropped` | the same | `failed` |
|
|
1137
1181
|
| `whenBusy: 'fail'` over a busy key — the body never ran | `refused` | the same, `code: 'X_JOB_KEY_BUSY'` | `failed` |
|
|
1138
1182
|
|
|
@@ -1198,15 +1242,16 @@ FOR a user takes that user's id in its input and re-authorises it in the body.
|
|
|
1198
1242
|
| `X_ABORTED` | a cancelled attempt tried to write a step — core's code, not a second name for it |
|
|
1199
1243
|
| `X_JOB_LEASE_LOST` | the job was cancelled, or its lease lapsed and the queue re-delivered it, while this worker was still running it |
|
|
1200
1244
|
| `X_JOB_SLOT_LOST` | the fleet `concurrency` slot this run held was taken by another worker — a different row on a different clock from the lease above |
|
|
1201
|
-
| `X_JOB_NOT_CANCELLABLE` | `cancelJob` reached a job that already
|
|
1245
|
+
| `X_JOB_NOT_CANCELLABLE` | `cancelJob` reached a job that is not live — done, failed, dead or already cancelled — an id no queue holds, or a driver with no `cancel` |
|
|
1246
|
+
| `X_JOB_NOT_FOUND` | `requeue` was asked for an id the queue does not hold — one answer on both drivers. Fix: `x jobs ls --json` |
|
|
1202
1247
|
| `X_JOB_CONCURRENCY_UNENFORCEABLE` | a registered job declares `concurrency`, plain or keyed, and the driver has no lease store |
|
|
1203
1248
|
| `X_JOB_KEY_BUSY` | a run claimed under `whenBusy: 'fail'` while its key already held `limit` runs. Classified `terminal`. Fix: `x jobs ls --name <job> --state running --json` |
|
|
1204
1249
|
| `X_JOB_DECLARATION_INVALID` | `job()` is missing a required field, declares a `concurrency` no worker can honour (plain or keyed), or its `concurrency.key` answered an empty, non-string or over-long key at enqueue |
|
|
1205
1250
|
| `X_JOB_NOT_REMOVABLE` | `remove` / `removeMany` reached a running job. Fix: `x jobs cancel <id> --json` |
|
|
1206
1251
|
| `X_JOB_NOT_PROMOTABLE` | `promote` reached a job not waiting on its run time |
|
|
1207
|
-
| `X_JOB_PAGE_INVALID` | `list()` was asked for more than `MAX_JOB_PAGE` rows, or handed a cursor no page produced. Fix: `
|
|
1252
|
+
| `X_JOB_PAGE_INVALID` | `list()` was asked for more than `MAX_JOB_PAGE` rows, or handed a cursor no page produced. The cause names which cursor (`after` / `before`). Fix: `x jobs ls --limit 200 --json` |
|
|
1208
1253
|
| `X_JOB_ON_SETTLED_FAILED` | a declared `onSettled` threw on every one of its tries; logged and reported, never thrown |
|
|
1209
|
-
| `X_NOT_IMPLEMENTED` |
|
|
1254
|
+
| `X_NOT_IMPLEMENTED` | introspection or the operator surface on a driver with no `introspect` (a hand-rolled one) |
|
|
1210
1255
|
|
|
1211
1256
|
### Error classes
|
|
1212
1257
|
|
|
@@ -1240,6 +1285,7 @@ a job boundary the class is gone and the `code` is what survives — match on th
|
|
|
1240
1285
|
| `JobMaxAttemptsError` | `X_JOB_MAX_ATTEMPTS` | `src/errors.ts` |
|
|
1241
1286
|
| `JobNameTakenError` | `X_JOB_DUPLICATE` | `src/errors.ts` |
|
|
1242
1287
|
| `JobNotCancellableError` | `X_JOB_NOT_CANCELLABLE` | `src/errors.ts` |
|
|
1288
|
+
| `JobNotFoundError` | `X_JOB_NOT_FOUND` | `src/errors-requeue.ts` |
|
|
1243
1289
|
| `JobNotPromotableError` | `X_JOB_NOT_PROMOTABLE` | `src/errors-operator.ts` |
|
|
1244
1290
|
| `JobNotRemovableError` | `X_JOB_NOT_REMOVABLE` | `src/errors-operator.ts` |
|
|
1245
1291
|
| `JobNotRequeueableError` | `X_JOB_NOT_REQUEUEABLE` | `src/errors-requeue.ts` |
|
|
@@ -1247,7 +1293,6 @@ a job boundary the class is gone and the `code` is what survives — match on th
|
|
|
1247
1293
|
| `JobPageInvalidError` | `X_JOB_PAGE_INVALID` | `src/errors-operator.ts` |
|
|
1248
1294
|
| `JobRowStatusUnknownError` | `X_JOB_ROW_STATUS_UNKNOWN` | `src/errors.ts` |
|
|
1249
1295
|
| `JobSlotLostError` | `X_JOB_SLOT_LOST` | `src/errors.ts` |
|
|
1250
|
-
| `JobsNotImplementedError` | `X_NOT_IMPLEMENTED` | `src/errors.ts` |
|
|
1251
1296
|
| `JobTenantRequiredError` | `X_JOB_TENANT_REQUIRED` | `src/errors.ts` |
|
|
1252
1297
|
| `JobTimeoutError` | `X_JOB_TIMEOUT` | `src/errors.ts` |
|
|
1253
1298
|
| `LeaseLostError` | `X_JOB_LEASE_LOST` | `src/errors.ts` |
|