@ultimat3/jobs 24.0.0 → 25.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +25 -27
- package/README.md +50 -34
- package/package.json +7 -7
- package/src/backfill-errors.ts +3 -2
- package/src/backfill-ledger.ts +1 -1
- package/src/backfill-rate.ts +2 -2
- package/src/backfill.ts +2 -2
- package/src/counters.ts +1 -1
- package/src/driver-memory-operator.ts +3 -3
- package/src/driver-memory.ts +14 -17
- package/src/driver-pg-settle-sql.ts +2 -3
- package/src/driver-pg-sql.ts +1 -1
- package/src/driver-pg.ts +18 -18
- package/src/driver.ts +9 -15
- package/src/errors-concurrency.ts +1 -1
- package/src/errors.ts +5 -5
- package/src/events-pg.ts +4 -4
- package/src/events.ts +6 -6
- package/src/execute.ts +46 -5
- package/src/export.ts +2 -2
- package/src/heartbeat.ts +7 -2
- package/src/index.ts +37 -97
- package/src/inspect-operator.ts +1 -1
- package/src/inspect.ts +1 -1
- package/src/job.ts +1 -1
- package/src/leases.ts +1 -1
- package/src/limits.ts +13 -8
- package/src/outbox-lease.ts +1 -1
- package/src/outbox-pg.ts +5 -5
- package/src/outbox-relay.ts +6 -6
- package/src/outbox.ts +13 -13
- package/src/renewal-timer.ts +1 -1
- package/src/retry-classification.ts +52 -15
- package/src/scheduler-leader.ts +2 -2
- package/src/scheduler-pg.ts +16 -14
- package/src/scheduler-state.ts +2 -2
- package/src/scheduler.ts +5 -5
- package/src/steps-memory.ts +2 -2
- package/src/steps.ts +35 -7
- package/src/task.ts +1 -1
- package/src/webhook-attempt.ts +206 -0
- package/src/webhook-ledger.ts +15 -4
- package/src/webhook.ts +64 -118
- package/src/worker-admit.ts +7 -3
- package/src/worker-drain-cutoff.ts +110 -0
- package/src/worker-fleet-slots.ts +5 -2
- package/src/worker-held.ts +100 -0
- package/src/worker-options.ts +19 -15
- package/src/worker-run.ts +38 -8
- package/src/worker-tally.ts +53 -0
- package/src/worker-types.ts +7 -4
- package/src/worker.ts +85 -77
- package/src/driver-nats.ts +0 -89
- package/src/driver-redis.ts +0 -92
package/CLAUDE.md
CHANGED
|
@@ -47,7 +47,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
47
47
|
- **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`,
|
|
48
48
|
`x_outbox`, `x_scheduler_*`). A NEW table is also a name in
|
|
49
49
|
`packages/cli/src/framework-schema.ts`. A shipped table grows by `alter table ... add column if
|
|
50
|
-
not exists`. Its comments carry NO apostrophes and NO semicolons (
|
|
50
|
+
not exists`. Its comments carry NO apostrophes and NO semicolons (it splits on `;`; `driver-pg-sql.test.ts` checks
|
|
51
51
|
quote parity).
|
|
52
52
|
- **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
|
|
53
53
|
`X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's reads are `async` so a refusal rejects on both.
|
|
@@ -67,13 +67,13 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
67
67
|
- Drivers implement the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`. New
|
|
68
68
|
capabilities go behind the interface. `inspect.ts` returns plain JSON for CLI, `/_x` and MCP.
|
|
69
69
|
- Step results are persisted BEFORE the step returns. All time is epoch ms (`nowMs()`, `clock.ts`).
|
|
70
|
-
- **`
|
|
70
|
+
- **`postgresLeader` is correct only on a DEDICATED connection**; boot uses `postgresLeaseLeader`.
|
|
71
71
|
|
|
72
72
|
## Rules — numbers and limits
|
|
73
73
|
|
|
74
74
|
- **Every numeric knob is refused when not FINITE** — core's `finiteOption()` (a bound) and
|
|
75
|
-
`finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `
|
|
76
|
-
**`bun run finite-bounds` is a floor, never proof**: `
|
|
75
|
+
`finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `jobWorker` reads them.
|
|
76
|
+
**`bun run finite-bounds` is a floor, never proof**: `concurrencyLimiter`'s five numbers are screened
|
|
77
77
|
with min 0 (zero is a HARD STOP). **A row count is `finiteCount` (min 0)**; `retry.attempts` is one with min 1.
|
|
78
78
|
- **`WorkerOptions.concurrency` is read by OWN key** — a queue named `constructor`
|
|
79
79
|
(`worker-slots.test.ts`).
|
|
@@ -98,16 +98,14 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
98
98
|
- The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
|
|
99
99
|
job's plain key.
|
|
100
100
|
- **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
|
|
101
|
-
the poll timer,
|
|
101
|
+
the poll timer, arm the drain cut-off — it aborts nothing); `close` is the teardown. `stop()` hands both back in a `finally`;
|
|
102
102
|
`start()` refuses while draining.
|
|
103
|
-
- **A claimed job is counted with core's `beginWork()
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
`
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
drained attempt settles as **`interrupted`** (`nack`, `countsAsAttempt: false`, no park, no dead
|
|
110
|
-
letter), read off the SIGNAL. `WorkerStats.interrupted`; `worker-drain-signal.test.ts`.
|
|
103
|
+
- **A claimed job is counted with core's `beginWork()`**: the drain's in-flight phase waits. The
|
|
104
|
+
teardown's wait is bounded on SIGTERM (`drain-wait.ts`), unbounded on `stop()`; one teardown, joined.
|
|
105
|
+
- **A deploy lets a running job FINISH**: SIGTERM only stops claiming. `drainSignal` aborts
|
|
106
|
+
(`X_DRAINING`) at `deadlineAt − margin` (`worker-drain-cutoff.ts`), then held claims go back
|
|
107
|
+
uncounted (`worker-held.ts`). Settles **`interrupted`**; `docs/history/jobs.md`, "Drain, 2026-10".
|
|
108
|
+
- **`X_RATE_LIMITED` defers uncounted** (`rateLimitDeferralMs`).
|
|
111
109
|
- **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
|
|
112
110
|
synchronously with its guard.
|
|
113
111
|
- **A fleet slot is taken INSIDE a `try`, released AWAITED, and HELD, not owned**: `false`, or a TTL
|
|
@@ -123,7 +121,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
123
121
|
- **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
|
|
124
122
|
- **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
|
|
125
123
|
the attempt is cancelled in `executeJob`'s `finally`.
|
|
126
|
-
- **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`)
|
|
124
|
+
- **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`) — save a fenced
|
|
125
|
+
`X_DRAINING` on a live attempt: a step done in the drain is recorded once. The STORE fences
|
|
127
126
|
the claim (`StepFence`, `SQL_STEP_PUT` as `SQL_ACK`: `X_JOB_LEASE_LOST`). `executeJob` passes it.
|
|
128
127
|
- **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
|
|
129
128
|
- **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
|
|
@@ -140,8 +139,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
140
139
|
- **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
|
|
141
140
|
`nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
|
|
142
141
|
stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
|
|
143
|
-
- **A
|
|
144
|
-
|
|
142
|
+
- **A wiring throw hands back the run's holdings, and its ROW until `executeJob` starts**
|
|
143
|
+
(`worker-run.ts`), as a failed shed does its job (`worker-admit.ts`).
|
|
145
144
|
- **`step.run` hydrates the run's steps from ONE `store.list(runId)`**; `sleep` and `waitForEvent`
|
|
146
145
|
record replays through `trace()`. **`claimName` reads a Set**; `usedNames()` keeps
|
|
147
146
|
`MAX_TRACE_NAMES` (200).
|
|
@@ -155,9 +154,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
155
154
|
suite (memory + real pg).
|
|
156
155
|
- **A settle answers whether it landed**; a miss is `jobs.settle.unowned`, not a throw.
|
|
157
156
|
- **The counter moves in the SETTLING statement** (`driver-pg-settle-sql.ts`): one-minute bucket per
|
|
158
|
-
job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing)
|
|
159
|
-
|
|
160
|
-
(`rollupCounters`), at most once a bucket.
|
|
157
|
+
job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing).
|
|
158
|
+
The scheduler leader folds tiers (`rollupCounters`), at most once a bucket.
|
|
161
159
|
- **`list()` is KEYSET** (`after: jobCursor(lastRow)`, newest first by `(created_at, id)`); the pg
|
|
162
160
|
seek reads the cursor row's own `created_at` — the cursor's ms is rounded. A page past
|
|
163
161
|
`MAX_JOB_PAGE` or a foreign cursor is `X_JOB_PAGE_INVALID`, never a bare invariant.
|
|
@@ -197,7 +195,7 @@ Long form: `docs/history/jobs.md`, "the cross-process wake".
|
|
|
197
195
|
- **`idle-cost.test.ts` / `queue-wake.test.ts` count statements per idle minute** — scheduler 6,
|
|
198
196
|
worker and relay 30 with no wake, 12 with. Raising one is a regression.
|
|
199
197
|
- **The memory driver stores a payload's JSON form**, as pg binds it (`driver-settle-parity.test.ts`).
|
|
200
|
-
- **The event bus has ONE clock**: `
|
|
198
|
+
- **The event bus has ONE clock**: `postgresEventBus` takes no `clock`; `EventLookup.now()` is
|
|
201
199
|
REQUIRED, stamps a NEW wait, and calls the TIMEOUT (asked once the runner thinks time is up).
|
|
202
200
|
`purgeExpired()` is awaited and counted; `eventsPurgeTarget(bus)` is its `PurgeTarget`.
|
|
203
201
|
|
|
@@ -228,8 +226,8 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
228
226
|
- **`purge()`** is the one caller of every `purgeExpired()`: one table per `step.run`, one clock
|
|
229
227
|
reading per pass, duplicate names refused.
|
|
230
228
|
- **`exportRows()`**: one object per PAGE, named by page index; NO cross-tenant escape.
|
|
231
|
-
- **`webhook()
|
|
232
|
-
throw. **The wire format is core's
|
|
229
|
+
- **`webhook()`**: ONE event to ONE endpoint, no steps; each attempt but a CANCELLED one recorded
|
|
230
|
+
before the throw; `ledger.isDisabled` gates the socket. **The wire format is core's.**
|
|
233
231
|
- A `-fixture.ts` file does not ship; `backfill-pass-fixture.ts` raises a plain `Error` subclass on
|
|
234
232
|
purpose (it stands in for app code).
|
|
235
233
|
|
|
@@ -255,28 +253,28 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
255
253
|
| `steps-timeout.ts` / `steps-suspension.ts` | a step's ceiling; `StepSuspension` and the timed-out marker |
|
|
256
254
|
| `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
|
|
257
255
|
| `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
|
|
258
|
-
| `outbox-pg.ts` | `
|
|
256
|
+
| `outbox-pg.ts` | `postgresOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
|
|
259
257
|
| `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
|
|
260
258
|
| `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
|
|
261
259
|
| `concurrency.ts` | `job.concurrency` as declared, resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
|
|
262
260
|
| `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `errors.ts` |
|
|
263
261
|
| `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
|
|
264
262
|
| `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
|
|
265
|
-
| `scheduler-pg.ts` | `
|
|
266
|
-
| `events-pg.ts` | `
|
|
263
|
+
| `scheduler-pg.ts` | `postgresSchedulerState` (the durable watermark, the atomic fire) + `postgresLeaseLeader` |
|
|
264
|
+
| `events-pg.ts` | `postgresEventBus` — `step.waitForEvent` across processes |
|
|
267
265
|
| `driver.ts` | `JobDriver` contract + wire records |
|
|
268
|
-
| `driver-pg.ts` | default driver, and `
|
|
266
|
+
| `driver-pg.ts` | default driver, and `postgresLeader` |
|
|
269
267
|
| `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the ONE install point. Its comments carry no `;` and no `'` |
|
|
270
268
|
| `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and `JOB_ROW_COLUMNS`; re-exported from `driver-pg-sql.ts` |
|
|
271
269
|
| `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
|
|
272
270
|
| `driver-memory.ts` / `claim-exhausted.ts` | `x dev` / tests; a buried row's log line and `onSettled` |
|
|
273
|
-
| `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
|
|
274
271
|
| `retry.ts` | the dead-letter decision, and this package's option names over core's `backoffDelay` — no curve of its own |
|
|
275
272
|
| `retry-classification.ts` | the other half: what the thrown error says, and the stop reason |
|
|
276
273
|
| `execute.ts` / `run-deadline.ts` | `executeJob` — one claimed job run and settled; the run's deadline: cancel, then fail |
|
|
277
274
|
| `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
|
|
278
275
|
| `renewal-timer.ts` | the renewal interval and its `stopped()` latch |
|
|
279
276
|
| `worker.ts` | `worker` role, claim round, drain |
|
|
277
|
+
| `worker-drain-cutoff.ts` / `worker-held.ts` / `worker-tally.ts` | drain cut-off · held claims · counters |
|
|
280
278
|
| `worker-admit.ts` | may this claimed job start: the limiter, the fleet slot, and the shed / refusal |
|
|
281
279
|
| `worker-registry.ts` | one worker's registry row: announce, heartbeat, forget |
|
|
282
280
|
| `worker-types.ts` | the worker's public contract: `WorkerOptions`, `WorkerStats`, `Worker` |
|
package/README.md
CHANGED
|
@@ -192,7 +192,7 @@ input. The tenant is a fact about the work, so the job declares it.
|
|
|
192
192
|
expires, so on a bus shared by one `bun test` process an answer one test published resumes the
|
|
193
193
|
next test's wait — call it between tests.
|
|
194
194
|
|
|
195
|
-
The event bus has ONE clock. `
|
|
195
|
+
The event bus has ONE clock. `postgresEventBus({ executor })` stamps `publishedAt` and the expiry
|
|
196
196
|
with the database's `now()` — it takes no `clock` — and `bus.now()` answers that same clock. A
|
|
197
197
|
consumer that wants "published after I asked" (`@ultimat3/scraping`'s `eventPrompt`) reads its
|
|
198
198
|
"asked at" from `bus.now()`, never from its own process: a skew between two pods must not decide
|
|
@@ -537,9 +537,11 @@ x-ultimate-webhook-signature: t=1700000000,v1=<hex hmac-sha256>
|
|
|
537
537
|
|
|
538
538
|
The receiving half is `verifyWebhookSignature(request, { secret })` in `@ultimat3/http`. The format
|
|
539
539
|
itself — the canonical string, the mac and the header names — is **one module in `@ultimat3/core`**
|
|
540
|
-
(`webhook-signature.ts`),
|
|
541
|
-
signs, `http` verifies, and neither may import the other, so the one copy
|
|
542
|
-
can reach.
|
|
540
|
+
(`webhook-signature.ts`), imported from `@ultimat3/core` by both packages and re-declared by
|
|
541
|
+
neither: this package signs, `http` verifies, and neither may import the other, so the one copy
|
|
542
|
+
lives at the tier both can reach. A receiver or a test imports `WEBHOOK_SIGNATURE_HEADER`,
|
|
543
|
+
`webhookSignature` and the rest from `@ultimat3/core` — 25.0.0 dropped this package's
|
|
544
|
+
re-exports (`X_HELPER_COPY`). Same argument `timing-safe-equal.ts` makes for itself.
|
|
543
545
|
|
|
544
546
|
| Rule | Why |
|
|
545
547
|
|---|---|
|
|
@@ -550,10 +552,21 @@ can reach. Same argument `timing-safe-equal.ts` makes for itself.
|
|
|
550
552
|
| `redirect: 'manual'` | following a 3xx would re-POST a body signed for one host to whatever the receiver named |
|
|
551
553
|
| the endpoint is never checkpointed | a `step.run` output lands in `x_job_steps`, and the endpoint carries the secret |
|
|
552
554
|
| every attempt is recorded **before** the throw | a failure the ledger cannot see is a failure the consecutive count cannot see, which is an endpoint that never gets disabled |
|
|
553
|
-
| a `Retry-After` the receiver names is honoured | `X_WEBHOOK_DELIVERY_THROTTLED` carries `meta.retryAfterSeconds`, which the nack waits out (clamped by `retry.maxDelay`
|
|
555
|
+
| a `Retry-After` the receiver names is honoured | `X_WEBHOOK_DELIVERY_THROTTLED` carries `meta.retryAfterSeconds`, which the nack waits out — read by core's `retryAfterSecondsOf` (delta-seconds, or an HTTP-date against the response's own `Date`; `0` or a past date is unstated), clamped by `retry.maxDelay`, plus core's `jitterStatedDelay` spread so deliveries handed one instant do not wake together |
|
|
556
|
+
| `WebhookLedger.isDisabled(endpointId)` is asked before every socket, beside the row's own `disabled` | `disableAfter` writes only the ledger: a verdict nobody asks for again is an endpoint that keeps receiving POSTs. Answer it from the column your `disable` sets |
|
|
557
|
+
| the deadline and a lost lease abort `fetch`; a failure that abort caused writes no ledger row | the nack behind them hands the job to another worker at once, so a request left open is a duplicate delivery, and an abandoned attempt's failure would count toward `disableAfter` |
|
|
558
|
+
| the DRAIN never aborts a POST already on the wire; it stops one not yet sent | torn down, the job goes to the next pod, which re-POSTs the same event on every deploy. Finished, it is recorded like any other outcome — a real failure during the drain included |
|
|
554
559
|
| re-enabling is always yours | an endpoint the framework un-disabled on its own is a retry loop with no end |
|
|
555
560
|
| `WebhookLedger` is a seam, not a table | retention is seven years for one business and thirty days for the next, so shipping a schema would ship one of those answers |
|
|
556
561
|
|
|
562
|
+
**Set `timeout` below the drain budget.** A webhook has no timeout unless you declare one, and a
|
|
563
|
+
POST on the wire when SIGTERM lands runs until the receiver answers. With a `timeout`, a drained
|
|
564
|
+
request is still torn down at the attempt's deadline — where `executeJob` hands the job on — and
|
|
565
|
+
writes no ledger row. Without one, the only bound is `configureLifecycle({ deadlineMs })`, 25 s by
|
|
566
|
+
default: past it the worker logs `jobs.worker.drain-abandoned` and exits, and the row comes back
|
|
567
|
+
only when its lease lapses, as a COUNTED attempt the receiver may already have processed.
|
|
568
|
+
`timeout: '10s'` against the default budget ends every drained delivery inside it.
|
|
569
|
+
|
|
557
570
|
## The deadline cancels
|
|
558
571
|
|
|
559
572
|
A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that
|
|
@@ -670,15 +683,15 @@ things have to be true in a process:
|
|
|
670
683
|
| Step | Call |
|
|
671
684
|
|---|---|
|
|
672
685
|
| the table exists | ships in `SQL_JOBS_TABLE` — applying the queue DDL is enough |
|
|
673
|
-
| the facade is installed | `setJobsFacade(
|
|
674
|
-
| the relay is running | `
|
|
686
|
+
| the facade is installed | `setJobsFacade(outboxJobsFacade({ store, driver }, currentTx))` |
|
|
687
|
+
| the relay is running | `outboxRelay({ store, driver }).start()` |
|
|
675
688
|
|
|
676
|
-
`start()` registers the same two shutdown hooks `
|
|
689
|
+
`start()` registers the same two shutdown hooks `jobWorker` does — `accept` stops polling
|
|
677
690
|
(the worker's also aborts every held job's `ctx.signal`), `close` waits out the pass in flight
|
|
678
691
|
under the drain's deadline — and `stop()` hands both back.
|
|
679
692
|
`drainOnShutdown: false` opts out, for a caller that drives its own teardown.
|
|
680
693
|
|
|
681
|
-
with `store =
|
|
694
|
+
with `store = postgresOutboxStore({ executor, txExecutor })`. `txExecutor` is what makes it
|
|
682
695
|
transactional: `stage()` runs on the CALLER'S connection, never the pool. With nothing installed,
|
|
683
696
|
`jobsFacade()` answers a fallback whose `currentTx` is `() => undefined` and every enqueue
|
|
684
697
|
publishes straight to the driver — deliberate, so a script and a test enqueue with no wiring, but
|
|
@@ -713,7 +726,7 @@ What the lease buys, precisely:
|
|
|
713
726
|
| a relay that died mid-batch stranding its rows forever | anything a **non-idempotent** handler does on its second run |
|
|
714
727
|
| — | a lapsed visibility lease re-delivering a job that is still running |
|
|
715
728
|
|
|
716
|
-
The memory store (`
|
|
729
|
+
The memory store (`memoryOutboxStore`, `x dev` and tests) **drops** a published row —
|
|
717
730
|
`retained()` is the relay's backlog, not a running total; the pg store keeps `published_at` as
|
|
718
731
|
the audit trail this map is not. A relay pass that throws is logged as `jobs.outbox.tick-failed`
|
|
719
732
|
and the loop re-arms: an unobserved rejection would end the process with rows still staged.
|
|
@@ -730,17 +743,21 @@ One interface: `enqueue`, `claim` (visibility timeout), `ack(jobId, { workerId,
|
|
|
730
743
|
`backfills` and `leases`. Both settles are fenced on the CLAIM — `claimOf(claimed)` is the
|
|
731
744
|
`{ workerId, claim }` a `ClaimedJob` carries — and answer whether they landed. Zero job-code change
|
|
732
745
|
between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driver` config line**:
|
|
733
|
-
`JobsConfig.driver` has no reader and boot always builds `
|
|
746
|
+
`JobsConfig.driver` has no reader and boot always builds `postgresJobDriver`.
|
|
734
747
|
|
|
735
748
|
| Driver | Status | Backing | Use |
|
|
736
749
|
|---|---|---|---|
|
|
737
750
|
| `pg` | **default** | `SELECT ... FOR UPDATE SKIP LOCKED`, a partial unique index on `(name, coalesce(tenant_id, ''), idempotency_key)` over live rows (`x_jobs_name_tenant_idempotency_live_idx`), lease-based leader, `x_job_leases` | zero-infra start, most apps |
|
|
738
751
|
| `memory` | complete | in-process maps | `x dev`, tests |
|
|
739
|
-
| `redis` | interface-complete, `X_NOT_IMPLEMENTED` | Streams + consumer groups | planned |
|
|
740
|
-
| `nats` | interface-complete, `X_NOT_IMPLEMENTED` | JetStream work queue | planned |
|
|
741
752
|
|
|
742
|
-
|
|
743
|
-
|
|
753
|
+
There is **no NATS or Redis jobs driver**: `createNatsDriver` and `createRedisDriver`, all-throw
|
|
754
|
+
stubs, were deleted in 25.0.0. A real `redisJobDriver()` would arrive as a minor.
|
|
755
|
+
|
|
756
|
+
The pg SQL lives verbatim in `src/driver-pg-*sql.ts` (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …)
|
|
757
|
+
so an agent debugging a stuck queue can read and run the exact statement. The barrel exports
|
|
758
|
+
only the statement another package runs: `SQL_JOBS_TABLE` (the install) — 25.0.0 took the rest
|
|
759
|
+
off, `SQL_CLAIM` and `SQL_OUTBOX_RELEASE` included (their array-parameter proofs are this package's
|
|
760
|
+
`driver-pg-array.live.test.ts`).
|
|
744
761
|
|
|
745
762
|
**A lease that lapses on a row's final attempt is buried by the claim.** `claim` re-takes a
|
|
746
763
|
`running` row whose lease lapsed — the worker died — but one already at `attempt >= maxAttempts`
|
|
@@ -754,7 +771,7 @@ the job's `onSettled`, and the worker's claim round does exactly that. The row c
|
|
|
754
771
|
`maxAttempts` and not the policy, so the CALLER names the jobs declaring `retry.deadLetter: false`
|
|
755
772
|
(`ClaimOptions.dropExhausted` — the worker reads them off its registry): those rows are buried
|
|
756
773
|
`failed` and announced `dropped`, every other one `dead` / `dead-lettered`. A caller that names
|
|
757
|
-
none
|
|
774
|
+
none buries every such row `dead`.
|
|
758
775
|
|
|
759
776
|
A driver you write reads a nack's target state through `nackState(options)` — `dead`
|
|
760
777
|
(`deadLetter`), `failed` (`fail`, terminal and out of the dead-letter queue), `suspended` (`park`)
|
|
@@ -765,8 +782,8 @@ or `ready` — so a new branch cannot land in one driver and not another. Its le
|
|
|
765
782
|
|
|
766
783
|
| Role | Entry | Behaviour |
|
|
767
784
|
|---|---|---|
|
|
768
|
-
| `worker` | `
|
|
769
|
-
| `scheduler` | `
|
|
785
|
+
| `worker` | `jobWorker({ driver, context, queues, concurrency })` | per-queue pools, lease heartbeat, SIGTERM drain: stop claiming → finish in-flight → close. `concurrency` is a number for every queue or a table per queue; a queue it does not name gets core's `JOBS_CONCURRENCY_DEFAULT` (8 — this package's own default was 5 before 25.0.0) |
|
|
786
|
+
| `scheduler` | `jobScheduler({ driver, leader, state })` | one dispatch round at a time, catch-up policy, SIGTERM drain: stop dispatching → finish the round → release the lock |
|
|
770
787
|
|
|
771
788
|
Both roles register the same **pair** of hooks and bound the `close` half the same way. The
|
|
772
789
|
scheduler's abandoned case differs in one respect: a round still enqueueing keeps the lease rather
|
|
@@ -798,24 +815,24 @@ update runs two leaders.
|
|
|
798
815
|
|
|
799
816
|
```ts
|
|
800
817
|
import {
|
|
801
|
-
|
|
802
|
-
|
|
818
|
+
postgresLeaseLeader,
|
|
819
|
+
jobScheduler,
|
|
803
820
|
type JobDriver,
|
|
804
821
|
type PgExecutor,
|
|
805
|
-
|
|
822
|
+
postgresSchedulerState,
|
|
806
823
|
} from '@ultimat3/jobs';
|
|
807
824
|
|
|
808
825
|
declare const driver: JobDriver;
|
|
809
826
|
declare const executor: PgExecutor; // see "A `PgExecutor`" below
|
|
810
827
|
|
|
811
|
-
|
|
828
|
+
jobScheduler({
|
|
812
829
|
driver,
|
|
813
|
-
state:
|
|
814
|
-
leader:
|
|
830
|
+
state: postgresSchedulerState(executor),
|
|
831
|
+
leader: postgresLeaseLeader({ executor }),
|
|
815
832
|
});
|
|
816
833
|
```
|
|
817
834
|
|
|
818
|
-
`
|
|
835
|
+
`postgresLeaseLeader` and not `postgresLeader`: `pg_try_advisory_lock` is scoped to a Postgres
|
|
819
836
|
*session*, and the executor this package is handed is a **pool** — the lock is released the moment
|
|
820
837
|
that connection goes back to it, so every node reads itself as leader. The lease is a row with an
|
|
821
838
|
expiry and needs no connection affinity. `acquire()` is also the renewal, called every round, which
|
|
@@ -855,10 +872,10 @@ hourly task fire 24 catch-ups a second apart.
|
|
|
855
872
|
client, which already speaks `(text, values)` and decodes `jsonb`:
|
|
856
873
|
|
|
857
874
|
```ts
|
|
858
|
-
import {
|
|
875
|
+
import { postgresClient } from '@ultimat3/db';
|
|
859
876
|
import type { PgExecutor } from '@ultimat3/jobs';
|
|
860
877
|
|
|
861
|
-
const client =
|
|
878
|
+
const client = postgresClient({ url: 'postgres://localhost:5432/app_test' });
|
|
862
879
|
export const executor: PgExecutor = {
|
|
863
880
|
query: <R>(text: string, values: readonly unknown[]) => client.query<R>({ text, values }),
|
|
864
881
|
};
|
|
@@ -886,11 +903,11 @@ A queue with nothing to do costs almost nothing, and a job starts when it is com
|
|
|
886
903
|
`startQueueWake({ listener, executor })` holds ONE `LISTEN` session per worker pod and turns "another process committed" into the signal the idle loops already hear. The boot starts it for the `worker` role.
|
|
887
904
|
|
|
888
905
|
```ts
|
|
889
|
-
import {
|
|
906
|
+
import { postgresClient } from '@ultimat3/db';
|
|
890
907
|
import type { PgExecutor } from '@ultimat3/jobs';
|
|
891
908
|
import { startQueueWake } from '@ultimat3/jobs';
|
|
892
909
|
|
|
893
|
-
const client =
|
|
910
|
+
const client = postgresClient({ url: 'postgres://localhost:5432/app_test' });
|
|
894
911
|
const executor: PgExecutor = {
|
|
895
912
|
query: <R>(text: string, values: readonly unknown[]) => client.query<R>({ text, values }),
|
|
896
913
|
};
|
|
@@ -959,7 +976,7 @@ exactly as it had before.
|
|
|
959
976
|
| Thrown | What the queue does |
|
|
960
977
|
|---|---|
|
|
961
978
|
| a `terminal` code (`X_SCRAPE_AUTH_FAILED`, a validation fault, a permission denial) | dead-lettered on the attempt it happened, `attempt` recorded, remaining attempts unspent — a rotated password retried five times is five more wrong passwords at a site that locks the account after three |
|
|
962
|
-
| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay` — instead of the backoff. Still an attempt, still under the ceiling |
|
|
979
|
+
| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay`, plus core's `jitterStatedDelay` spread unless `jitter: false` — instead of the backoff. Still an attempt, still under the ceiling |
|
|
963
980
|
| a `retryable` code (`X_TIMEOUT`, `X_DRAINING`) | the backoff schedule above, unchanged |
|
|
964
981
|
| an **unclassified** code, or anything that is not an `UltimateError` | the backoff schedule above, unchanged. Most codes are unclassified and `retryFor` answers `terminal` for all of them, so reading that would have stopped every transient retry in every app |
|
|
965
982
|
|
|
@@ -990,7 +1007,7 @@ contractual rate.
|
|
|
990
1007
|
`job.concurrency` is the one that is fleet-wide, and it is enforced by a row every replica can
|
|
991
1008
|
see — one per held slot, keyed `job:<name>`, renewed by the same heartbeat that renews the
|
|
992
1009
|
visibility lease and reclaimed by TTL when a worker is SIGKILLed. A driver with no lease store
|
|
993
|
-
cannot hold the cap, so `
|
|
1010
|
+
cannot hold the cap, so `jobWorker().start()` **refuses to boot**
|
|
994
1011
|
(`X_JOB_CONCURRENCY_UNENFORCEABLE`) rather than let a documented guarantee do nothing.
|
|
995
1012
|
|
|
996
1013
|
Over any cap the claim is handed straight back without burning an attempt — one org's 50k-row
|
|
@@ -1028,7 +1045,7 @@ export const syncAccount = job({
|
|
|
1028
1045
|
|---|---|---|
|
|
1029
1046
|
| a cap of `0`, negative, fractional, `NaN` or `Infinity` — `concurrency: 0` and `limit: 0` alike; a `key` that is not a function; a `whenBusy` outside the two | where the job is declared | `X_JOB_DECLARATION_INVALID` |
|
|
1030
1047
|
| `key(input)` answering `''`, a non-string, or more than 200 characters | at that **enqueue** — the facade and the scheduler both ask | `X_JOB_DECLARATION_INVALID` |
|
|
1031
|
-
| a driver with no lease store, for any `concurrency` | `
|
|
1048
|
+
| a driver with no lease store, for any `concurrency` | `jobWorker().start()` | `X_JOB_CONCURRENCY_UNENFORCEABLE` |
|
|
1032
1049
|
| a `LeaseStore` with no `holders(key)` | the build — the member is required | `TS2741` |
|
|
1033
1050
|
|
|
1034
1051
|
What `'fail'` decides, exactly:
|
|
@@ -1116,7 +1133,6 @@ the pg and the memory driver alike. `x jobs`, `/_x` and a dashboard read nothing
|
|
|
1116
1133
|
| the counter moves in the statement that settles the row | a count that can disagree with the rows is worse than none — and no second round trip per job |
|
|
1117
1134
|
| a shed, a suspension and a drained attempt add nothing | they are handed back uncounted; counted, every `step.sleep` would read as a failure |
|
|
1118
1135
|
| `ack` / `nack` / `heartbeat` / `recordProgress` are fenced on the CLAIM | `{ workerId, claim }`: a body whose lease lapsed must not settle, renew or report on the run that replaced it — claimed by another worker, or by the SAME one, whose id did not change. A miss answers `false` and logs `jobs.settle.unowned` |
|
|
1119
|
-
| `ack(id, { …, counted: false })` adds nothing to the counters | `x jobs drain` settles the row it moved: `done` here, and not a completed run |
|
|
1120
1136
|
| a pause is a row the claim reads | never a column on every queued row: pausing a million-row queue takes effect within one poll |
|
|
1121
1137
|
| an occurrence fires in one statement | the watermark and its jobs move together (`SchedulerState.fire`), so a crash between them cannot fire it twice |
|
|
1122
1138
|
|
|
@@ -1235,7 +1251,7 @@ FOR a user takes that user's id in its input and re-authorises it in the body.
|
|
|
1235
1251
|
| `X_JOB_NOT_PROMOTABLE` | `promote` reached a job not waiting on its run time |
|
|
1236
1252
|
| `X_JOB_PAGE_INVALID` | `list()` was asked for more than `MAX_JOB_PAGE` rows, or handed a cursor no page produced. The cause names which cursor (`after` / `before`). Fix: `x jobs ls --limit 200 --json` |
|
|
1237
1253
|
| `X_JOB_ON_SETTLED_FAILED` | a declared `onSettled` threw on every one of its tries; logged and reported, never thrown |
|
|
1238
|
-
| `X_NOT_IMPLEMENTED` |
|
|
1254
|
+
| `X_NOT_IMPLEMENTED` | introspection or the operator surface on a driver with no `introspect` (a hand-rolled one) |
|
|
1239
1255
|
|
|
1240
1256
|
### Error classes
|
|
1241
1257
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@ultimat3/jobs",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "25.0.0",
|
|
4
4
|
"description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
|
@@ -25,17 +25,17 @@
|
|
|
25
25
|
"LICENSE"
|
|
26
26
|
],
|
|
27
27
|
"engines": {
|
|
28
|
-
"bun": ">=1.4.
|
|
28
|
+
"bun": ">=1.4.2"
|
|
29
29
|
},
|
|
30
30
|
"scripts": {
|
|
31
31
|
"typecheck": "tsc --noEmit -p tsconfig.json",
|
|
32
32
|
"test": "bun test"
|
|
33
33
|
},
|
|
34
34
|
"dependencies": {
|
|
35
|
-
"@ultimat3/core": "
|
|
36
|
-
"@ultimat3/db": "
|
|
37
|
-
"@ultimat3/entity": "
|
|
38
|
-
"@ultimat3/schema": "
|
|
39
|
-
"@ultimat3/time": "
|
|
35
|
+
"@ultimat3/core": "25.0.0",
|
|
36
|
+
"@ultimat3/db": "25.0.0",
|
|
37
|
+
"@ultimat3/entity": "25.0.0",
|
|
38
|
+
"@ultimat3/schema": "25.0.0",
|
|
39
|
+
"@ultimat3/time": "25.0.0"
|
|
40
40
|
}
|
|
41
41
|
}
|
package/src/backfill-errors.ts
CHANGED
|
@@ -5,7 +5,7 @@
|
|
|
5
5
|
// declared in ONE place; only the classes that throw them live here, beside `backfill-ledger.ts`,
|
|
6
6
|
// `backfill-pending.ts` and `backfill-registry.ts`.
|
|
7
7
|
|
|
8
|
-
import { UltimateError } from '@ultimat3/core';
|
|
8
|
+
import { renderFixShellArg, UltimateError } from '@ultimat3/core';
|
|
9
9
|
|
|
10
10
|
/**
|
|
11
11
|
* The seven backfill codes below all answer one question — "why is this sweep not running?" — and
|
|
@@ -64,7 +64,8 @@ export class BackfillEnvironmentError extends UltimateError {
|
|
|
64
64
|
fix:
|
|
65
65
|
target === undefined
|
|
66
66
|
? 'x db backfill --pending --json'
|
|
67
|
-
:
|
|
67
|
+
: // Both are the declaration's own strings and the class is public: screened (sweep 1c).
|
|
68
|
+
`ULTIMATE_ENV=${renderFixShellArg(target, "'<environment>'")} x db backfill ${renderFixShellArg(input.backfill, "'<backfill>'")} --write --json`,
|
|
68
69
|
});
|
|
69
70
|
}
|
|
70
71
|
}
|
package/src/backfill-ledger.ts
CHANGED
|
@@ -153,7 +153,7 @@ export function decideBackfill(
|
|
|
153
153
|
return { run: force, previous: completed, changed: completed.checksum !== checksum };
|
|
154
154
|
}
|
|
155
155
|
|
|
156
|
-
export function
|
|
156
|
+
export function memoryBackfillLedger(clock: Clock = systemClock): BackfillLedger {
|
|
157
157
|
const runs = new Map<string, BackfillRun>();
|
|
158
158
|
const patch = (runId: string, fields: Partial<BackfillRun>): void => {
|
|
159
159
|
const existing = runs.get(runId);
|
package/src/backfill-rate.ts
CHANGED
|
@@ -69,14 +69,14 @@ function timerSleep(ms: number, signal: AbortSignal): Promise<void> {
|
|
|
69
69
|
* rate is a property of the table being swept and the pool it is swept through, not of whichever
|
|
70
70
|
* attempt happens to hold the run.
|
|
71
71
|
*/
|
|
72
|
-
export function
|
|
72
|
+
export function backfillPacer(options: PacerOptions): Pacer {
|
|
73
73
|
// Refused HERE and not only at `backfill()`: `rate: 0` makes `intervalMs` Infinity, which the
|
|
74
74
|
// timer clamps to about a millisecond — so an unvalidated zero reads as "no throttle at all",
|
|
75
75
|
// which is the one setting this module exists to make unreachable. A negative rate is the same
|
|
76
76
|
// bug with a negative wait.
|
|
77
77
|
assert(
|
|
78
78
|
Number.isFinite(options.rate) && options.rate > 0,
|
|
79
|
-
`
|
|
79
|
+
`backfillPacer({ rate: ${String(options.rate)} }) for "${options.job}" — a rate is batches per second, greater than zero`,
|
|
80
80
|
`pass rate: ${DEFAULT_BACKFILL_RATE} — to sweep faster raise the number, there is no unthrottled mode`,
|
|
81
81
|
);
|
|
82
82
|
const clock = options.clock ?? systemClock;
|
package/src/backfill.ts
CHANGED
|
@@ -15,7 +15,7 @@ import type { ReadBuilder } from '@ultimat3/entity';
|
|
|
15
15
|
import { t } from '@ultimat3/schema';
|
|
16
16
|
import { backfillChecksum } from './backfill-ledger';
|
|
17
17
|
import { backfillPass } from './backfill-pass';
|
|
18
|
-
import {
|
|
18
|
+
import { backfillPacer, DEFAULT_BACKFILL_RATE } from './backfill-rate';
|
|
19
19
|
import type { BackfillCount } from './backfill-registry';
|
|
20
20
|
import { stampBackfill } from './backfill-registry';
|
|
21
21
|
import type { DurationInput } from './clock';
|
|
@@ -176,7 +176,7 @@ export function backfill<Row>(definition: BackfillDefinition<Row>): JobHandle<Ba
|
|
|
176
176
|
const checksum = backfillChecksum(definition.source, definition.handle);
|
|
177
177
|
// Built here rather than per attempt: the interval belongs to the table and the pool, not to
|
|
178
178
|
// whichever attempt holds the run, so a retrying pass keeps the pace it was declared with.
|
|
179
|
-
const pace =
|
|
179
|
+
const pace = backfillPacer({ rate, job: definition.name });
|
|
180
180
|
|
|
181
181
|
// Bound to the definition rather than passed bare: `count` is declared as a method, so a
|
|
182
182
|
// reference torn off the object literal would run with `this` undefined the first time an
|
package/src/counters.ts
CHANGED
|
@@ -45,7 +45,7 @@ export interface MemoryCounters {
|
|
|
45
45
|
}
|
|
46
46
|
|
|
47
47
|
/** The memory driver's counters. Same buckets, same tiers, same fold as `x_job_counters`. */
|
|
48
|
-
export function
|
|
48
|
+
export function memoryCounters(): MemoryCounters {
|
|
49
49
|
const buckets = new Map<string, JobCounter>();
|
|
50
50
|
const keyOf = (job: string, bucketMs: number, bucketStart: number): string =>
|
|
51
51
|
`${bucketMs}\u0000${bucketStart}\u0000${job}`;
|
|
@@ -7,7 +7,7 @@ import type { Clock } from '@ultimat3/core';
|
|
|
7
7
|
import { assert, finiteOption } from '@ultimat3/core';
|
|
8
8
|
import { nowMs } from './clock';
|
|
9
9
|
import type { MemoryCounters } from './counters';
|
|
10
|
-
import {
|
|
10
|
+
import { memoryCounters } from './counters';
|
|
11
11
|
import type { JobRecord } from './driver';
|
|
12
12
|
import { REQUEUEABLE_STATES } from './driver';
|
|
13
13
|
import { signalEnqueued } from './enqueue-signal';
|
|
@@ -92,9 +92,9 @@ function pauseSet(clock: Clock): {
|
|
|
92
92
|
};
|
|
93
93
|
}
|
|
94
94
|
|
|
95
|
-
export function
|
|
95
|
+
export function memoryOperator(state: MemoryQueueState): MemoryOperator {
|
|
96
96
|
const { jobs, clock } = state;
|
|
97
|
-
const counters =
|
|
97
|
+
const counters = memoryCounters();
|
|
98
98
|
const queues = pauseSet(clock);
|
|
99
99
|
const tasks = pauseSet(clock);
|
|
100
100
|
const workers = new Map<string, WorkerRecord & { readonly expiresAt: number }>();
|
package/src/driver-memory.ts
CHANGED
|
@@ -3,9 +3,9 @@
|
|
|
3
3
|
// real claim/ack/nack paths rather than a mock that always succeeds.
|
|
4
4
|
|
|
5
5
|
import type { Clock } from '@ultimat3/core';
|
|
6
|
-
import { finiteCount, systemClock,
|
|
6
|
+
import { finiteCount, systemClock, uuidV7 } from '@ultimat3/core';
|
|
7
7
|
import type { BackfillLedger } from './backfill-ledger';
|
|
8
|
-
import {
|
|
8
|
+
import { memoryBackfillLedger } from './backfill-ledger';
|
|
9
9
|
import { nowMs } from './clock';
|
|
10
10
|
import { nackOutcome } from './counters';
|
|
11
11
|
import type {
|
|
@@ -30,19 +30,19 @@ import {
|
|
|
30
30
|
nackState,
|
|
31
31
|
REQUEUEABLE_STATES,
|
|
32
32
|
} from './driver';
|
|
33
|
-
import {
|
|
33
|
+
import { memoryOperator } from './driver-memory-operator';
|
|
34
34
|
import { signalEnqueued } from './enqueue-signal';
|
|
35
35
|
import { JobDuplicateError, LeaseLostError } from './errors';
|
|
36
36
|
import { JobNotFoundError, JobNotRequeueableError, requeueKeyTaken } from './errors-requeue';
|
|
37
37
|
import type { JobIntrospection } from './introspection';
|
|
38
38
|
import { MAX_ERROR_STACK_LENGTH } from './introspection';
|
|
39
39
|
import type { LeaseStore } from './leases';
|
|
40
|
-
import {
|
|
40
|
+
import { memoryLeaseStore } from './leases';
|
|
41
41
|
import { isFinalAttempt } from './retry';
|
|
42
42
|
import type { StepStore } from './steps';
|
|
43
|
-
import {
|
|
43
|
+
import { memoryStepStore } from './steps-memory';
|
|
44
44
|
|
|
45
|
-
export interface
|
|
45
|
+
export interface MemoryJobDriverOptions {
|
|
46
46
|
readonly clock?: Clock;
|
|
47
47
|
readonly steps?: StepStore;
|
|
48
48
|
/** Injectable for the same reason `steps` is: two drivers in one test sharing one ledger. */
|
|
@@ -62,9 +62,9 @@ export interface MemoryDriverOptions {
|
|
|
62
62
|
*/
|
|
63
63
|
export type MemoryJobDriver = JobDriver & { close(): Promise<void> };
|
|
64
64
|
|
|
65
|
-
export function
|
|
65
|
+
export function memoryJobDriver(options: MemoryJobDriverOptions = {}): MemoryJobDriver {
|
|
66
66
|
const clock = options.clock ?? systemClock;
|
|
67
|
-
const stepStore = options.steps ??
|
|
67
|
+
const stepStore = options.steps ?? memoryStepStore();
|
|
68
68
|
// `SQL_STEP_PUT`'s fence: a write made under a claim lands only while that claim holds the row.
|
|
69
69
|
const steps: StepStore = {
|
|
70
70
|
...stepStore,
|
|
@@ -79,9 +79,8 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
|
|
|
79
79
|
return stepStore.put(record);
|
|
80
80
|
},
|
|
81
81
|
};
|
|
82
|
-
const backfills = options.backfills ??
|
|
83
|
-
const leases =
|
|
84
|
-
options.leases ?? createMemoryLeaseStore(options.clock === undefined ? {} : { clock });
|
|
82
|
+
const backfills = options.backfills ?? memoryBackfillLedger(clock);
|
|
83
|
+
const leases = options.leases ?? memoryLeaseStore(options.clock === undefined ? {} : { clock });
|
|
85
84
|
const jobs = new Map<string, JobRecord>();
|
|
86
85
|
|
|
87
86
|
// Keyed by NAME, TENANT and key, exactly as `x_jobs_name_tenant_idempotency_live_idx` is. A
|
|
@@ -134,7 +133,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
|
|
|
134
133
|
});
|
|
135
134
|
};
|
|
136
135
|
|
|
137
|
-
const operator =
|
|
136
|
+
const operator = memoryOperator({
|
|
138
137
|
jobs,
|
|
139
138
|
steps,
|
|
140
139
|
clock,
|
|
@@ -252,7 +251,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
|
|
|
252
251
|
const at = nowMs(clock);
|
|
253
252
|
const runAt = request.runAt ?? at;
|
|
254
253
|
const record: JobRecord = {
|
|
255
|
-
id: request.id ??
|
|
254
|
+
id: request.id ?? uuidV7(),
|
|
256
255
|
name: request.name,
|
|
257
256
|
queue: request.queue || DEFAULT_QUEUE,
|
|
258
257
|
// Through JSON, exactly as the pg driver binds it (`JSON.stringify(request.input ?? null)`):
|
|
@@ -260,7 +259,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
|
|
|
260
259
|
// queue in production never stores, so a job that read one passed here and failed there.
|
|
261
260
|
input: JSON.parse(JSON.stringify(request.input ?? null)) as unknown,
|
|
262
261
|
idempotencyKey: request.idempotencyKey,
|
|
263
|
-
runId: request.runId ??
|
|
262
|
+
runId: request.runId ?? uuidV7(),
|
|
264
263
|
attempt: 0,
|
|
265
264
|
maxAttempts: request.maxAttempts,
|
|
266
265
|
state: runAt > at ? 'delayed' : 'ready',
|
|
@@ -341,9 +340,7 @@ export function createMemoryDriver(options: MemoryDriverOptions = {}): MemoryJob
|
|
|
341
340
|
const record = heldBy(jobId, by);
|
|
342
341
|
if (record === undefined) return Promise.resolve(false);
|
|
343
342
|
settle(jobId, { state: 'done' });
|
|
344
|
-
|
|
345
|
-
operator.counters.add(record.name, 'done', by.durationMs ?? 0, nowMs(clock));
|
|
346
|
-
}
|
|
343
|
+
operator.counters.add(record.name, 'done', by.durationMs ?? 0, nowMs(clock));
|
|
347
344
|
return Promise.resolve(true);
|
|
348
345
|
},
|
|
349
346
|
|
|
@@ -17,8 +17,7 @@ const BUCKET = `${COUNTER_BUCKET_MS}, date_trunc('minute', now())`;
|
|
|
17
17
|
* that claimed it again is the SAME one: its id did not change, only the claim's ordinal did.
|
|
18
18
|
* `returning` is how the caller learns whether its settle landed.
|
|
19
19
|
*
|
|
20
|
-
* $1 id, $2 worker, $3 the attempt's duration in ms, $4 the claim
|
|
21
|
-
* row `x jobs drain` moved is `done` here and is not a completed run.
|
|
20
|
+
* $1 id, $2 worker, $3 the attempt's duration in ms, $4 the claim.
|
|
22
21
|
*/
|
|
23
22
|
export const SQL_ACK = `
|
|
24
23
|
with settled as (
|
|
@@ -28,7 +27,7 @@ with settled as (
|
|
|
28
27
|
returning name
|
|
29
28
|
), counted as (
|
|
30
29
|
insert into x_job_counters (job, bucket_ms, bucket_start, done, duration_ms)
|
|
31
|
-
select name, ${BUCKET}, 1, $3::bigint from settled
|
|
30
|
+
select name, ${BUCKET}, 1, $3::bigint from settled
|
|
32
31
|
on conflict (job, bucket_ms, bucket_start) do update
|
|
33
32
|
set done = x_job_counters.done + 1,
|
|
34
33
|
duration_ms = x_job_counters.duration_ms + excluded.duration_ms
|
package/src/driver-pg-sql.ts
CHANGED
|
@@ -216,7 +216,7 @@ select queue,
|
|
|
216
216
|
|
|
217
217
|
/**
|
|
218
218
|
* Session-scoped, so a crashed node's lock releases itself — and released just as surely when a
|
|
219
|
-
* POOLED connection goes back to the pool, which is why `
|
|
219
|
+
* POOLED connection goes back to the pool, which is why `postgresLeader` is not what a scheduler
|
|
220
220
|
* running on a shared executor should use. See `SQL_LEADER_ACQUIRE` below.
|
|
221
221
|
*/
|
|
222
222
|
export const SQL_TRY_ADVISORY_LOCK = 'select pg_try_advisory_lock($1) as locked';
|