@ultimat3/jobs 30.9.0 → 31.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +72 -107
- package/README.md +202 -55
- package/package.json +8 -7
- package/src/adapter.ts +37 -0
- package/src/backfill-ledger.ts +2 -2
- package/src/backfill-rate.ts +1 -0
- package/src/clock.ts +2 -2
- package/src/drain-wait.ts +1 -0
- package/src/driver-memory.ts +2 -2
- package/src/driver-pg-ddl.ts +19 -0
- package/src/driver-pg-event-wake-sql.ts +43 -22
- package/src/driver-pg-sql.ts +68 -27
- package/src/driver-pg.ts +11 -10
- package/src/driver-redis.ts +2 -2
- package/src/driver.ts +14 -8
- package/src/error-codes.ts +27 -0
- package/src/errors-concurrency.ts +1 -1
- package/src/errors.ts +1 -1
- package/src/events-cursor.ts +31 -0
- package/src/events-pg.ts +52 -6
- package/src/events.ts +72 -36
- package/src/export.ts +2 -2
- package/src/gdpr-audit.ts +75 -0
- package/src/gdpr-coverage.ts +94 -0
- package/src/gdpr-errors.ts +35 -0
- package/src/import.ts +2 -2
- package/src/index.ts +45 -206
- package/src/inspect-operator.ts +2 -2
- package/src/inspect.ts +1 -1
- package/src/job-instrument.ts +115 -0
- package/src/job.ts +41 -4
- package/src/jobs-retention.ts +23 -6
- package/src/leases.ts +13 -3
- package/src/ledger-book.ts +168 -0
- package/src/ledger-errors.ts +48 -0
- package/src/ledger.ts +145 -0
- package/src/limits.ts +57 -19
- package/src/lock-errors.ts +54 -0
- package/src/lock.ts +176 -0
- package/src/outbox-relay.ts +4 -0
- package/src/outbox.ts +3 -3
- package/src/partitions.ts +119 -0
- package/src/reindex-sweep.ts +94 -0
- package/src/reindex.ts +147 -0
- package/src/retry-classification.ts +3 -2
- package/src/retry-jitter.ts +32 -0
- package/src/retry.ts +13 -16
- package/src/scheduler.ts +10 -6
- package/src/steps-api.ts +44 -5
- package/src/steps-suspension.ts +34 -3
- package/src/steps-wait.ts +52 -4
- package/src/steps.ts +1 -0
- package/src/subject-jobs.ts +12 -4
- package/src/webhook.ts +2 -2
- package/src/worker-admit.ts +8 -5
- package/src/worker-drain-cutoff.ts +1 -0
- package/src/worker-loop.ts +28 -5
- package/src/worker-registry.ts +7 -5
- package/src/worker-run.ts +53 -9
- package/src/worker.ts +8 -8
package/CLAUDE.md
CHANGED
|
@@ -1,16 +1,20 @@
|
|
|
1
1
|
# @ultimat3/jobs — agent notes
|
|
2
2
|
|
|
3
3
|
Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queue drivers.
|
|
4
|
+
A rule or row marked `†` has its full text in [`docs/history/jobs-rules.md`](../../docs/history/jobs-rules.md).
|
|
4
5
|
|
|
5
6
|
## Boundary
|
|
6
7
|
|
|
7
8
|
- May import: `core`, `schema`, `entity`, `policy`, `cache`, `time` — and `db`, for
|
|
8
|
-
`expectedQueryLoop` (`steps.ts
|
|
9
|
-
`
|
|
10
|
-
(`consumer-loop.ts`: a batch and its cursor commit together) —
|
|
11
|
-
from it, the ambient `db()`. Never `render`, `ui`. `http` only from `webhook-receiver*.ts`, for the route
|
|
9
|
+
`expectedQueryLoop` (`steps.ts`, the per-step write to the N+1 detector), `outsideTransaction`
|
|
10
|
+
(`execute.ts`), `currentTx` (`events-pg.ts`), `refuseAbandoned` (`outbox.ts`) and
|
|
11
|
+
`withTransaction` (`consumer-loop.ts`: a batch and its cursor commit together) — a client is
|
|
12
|
+
taken from it in ONE place, the ambient `db()`. Never `render`, `ui`. `http` only from `webhook-receiver*.ts`, for the route
|
|
12
13
|
(`Route`/`Bucket` types, `DEFAULT_WEBHOOK_BODY_LIMIT`, `HttpError`, `jsonResponse`).
|
|
13
14
|
- Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
|
|
15
|
+
- **Installers are on `@ultimat3/jobs/adapter`, never the `.` barrel** (`adapter.test.ts`):
|
|
16
|
+
`installJobDriver`/`installEventBus` + resets, the port interfaces, the errors a driver throws.
|
|
17
|
+
An app swaps a port in `apps/<app>/runtime.ts`; a `fix:` names that line, never an installer.
|
|
14
18
|
- External deps: none. Postgres access goes through the injected `PgExecutor` (`@ultimat3/core`'s,
|
|
15
19
|
re-exported from `index.ts`).
|
|
16
20
|
|
|
@@ -23,7 +27,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
23
27
|
`X_JOB_DUPLICATE` at the earliest decidable point (same `name:` inside `job()`/`task()`; a
|
|
24
28
|
different handle, or the same handle under a second export name, at registration).
|
|
25
29
|
- `registerJob`/`registerJobs`/`registerTask`/`registerTasks`/`nameJobs`/`nameTasks` are NOT in
|
|
26
|
-
`src/index.ts
|
|
30
|
+
`src/index.ts`, and there is no `./loader` subpath (action's and query's have one, for `x`'s
|
|
31
|
+
module scan): jobs and tasks reach their registries only through core's registrar table. `register.ts` announces the registrars in core's table at import — never remove it
|
|
27
32
|
(`X_REGISTRAR_MISSING`). `isJobHandle`/`isTaskHandle` are structural plus proof `job()`/`task()`
|
|
28
33
|
built it.
|
|
29
34
|
- `idempotencyKey` is NON-OPTIONAL. **The namespace is `(name, coalesce(tenant_id, ''),
|
|
@@ -45,32 +50,21 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
45
50
|
- **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`, `x_outbox`, `x_scheduler_*`). Full rule: `docs/history/jobs.md`.
|
|
46
51
|
- **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
|
|
47
52
|
`X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's reads are `async` so a refusal rejects on both.
|
|
48
|
-
- **`ack`, `nack`, `heartbeat
|
|
49
|
-
|
|
50
|
-
never reset. `heartbeat` answers a boolean, read as `held === false` (never `!held`).
|
|
53
|
+
- **`ack`, `nack`, `heartbeat`, `recordProgress` are FENCED on `running` AND the claim**
|
|
54
|
+
(`claimOf(claimed)`); read `heartbeat` as `held === false`. †
|
|
51
55
|
- **A driver's semantics are pinned beside the pg statement** (`driver-parity*.test.ts`);
|
|
52
56
|
`driver-*-fixture.ts` scenarios run on memory AND Postgres.
|
|
53
|
-
- **The claim BURIES a lease that lapsed on the final attempt** (`
|
|
54
|
-
|
|
55
|
-
— counted, never returned; the worker's round hears `onExhausted` and calls `announceExhausted`.
|
|
57
|
+
- **The claim BURIES a lease that lapsed on the final attempt** (`dead`, or `failed` under
|
|
58
|
+
`dropExhausted`); the worker calls `announceExhausted`. †
|
|
56
59
|
- **`cancel` is fenced on `LIVE_STATES`.** **`requeue`** refuses (`X_JOB_NOT_FOUND`, not requeueable,
|
|
57
60
|
duplicate), then ONE statement. **`enqueue` inserts twice at most.** **`BulkResult.remaining` is
|
|
58
61
|
what a second call would move.**
|
|
59
|
-
- **`enqueueMany` is one statement per `ENQUEUE_CHUNK` (1,000)
|
|
60
|
-
|
|
61
|
-
are answered by ONE `SQL_ENQUEUE_MANY_RESOLVE`, a row neither answers goes back through `enqueue`.
|
|
62
|
-
**`JobDriver.enqueueMany` is REQUIRED** and opens with `assertEnqueueMany` (no `onConflict:
|
|
63
|
-
'error'` in a batch, `X_INVARIANT`); Redis's `REDIS_ENQUEUE_MANY` shares `enqueue_one` with
|
|
64
|
-
`REDIS_ENQUEUE`. `OutboxStore.stageMany`/`markPublishedMany` are OPTIONAL: absent, a loop. Every
|
|
65
|
-
refusal precedes the first write; `refuseAbandoned()` before every direct chunk. `runAt` crosses
|
|
66
|
-
as `numeric` then `bigint`: a fractional ms rounds, never a `22P02`.
|
|
62
|
+
- **`enqueueMany` is one statement per `ENQUEUE_CHUNK` (1,000)**; `JobDriver.enqueueMany` is
|
|
63
|
+
REQUIRED; every refusal precedes the first write. †
|
|
67
64
|
- **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
|
|
68
65
|
(discovered, comments stripped) for `select *`/`returning *`.
|
|
69
|
-
- Drivers:
|
|
70
|
-
|
|
71
|
-
barrel) is one Lua script per operation: a semantic change lands in memory, pg AND the scripts.
|
|
72
|
-
Its client is core's `redisClient()`, never `Bun.redis`; `ERR_REDIS_CONNECTION_CLOSED` is
|
|
73
|
-
`X_DRIVER_UNAVAILABLE` at once (`typedRefusals`), never a wait.
|
|
66
|
+
- Drivers: `JobDriver` + optional `introspect`/`backfills`/`leases`; `redisJobDriver`
|
|
67
|
+
(`@ultimat3/jobs/redis`) is one Lua script per operation — a change lands in memory, pg AND Lua. †
|
|
74
68
|
- **`demand` is the seventh method, required**; a draining worker's `queue_demand` goes to 0.
|
|
75
69
|
- **The claim SEEKS** (`SQL_CLAIM`); correct only because a claim autocommits.
|
|
76
70
|
- Step results are persisted BEFORE the step returns, and it returns what it persisted: `jsonForm` (`steps-output.ts`), typed `StepOutput<T>`, one stringify + parse per EXECUTED step. Full rule: `docs/history/jobs.md`.
|
|
@@ -85,64 +79,49 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
85
79
|
`limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
|
|
86
80
|
throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
|
|
87
81
|
- **`concurrency` is resolved ONCE** (`concurrency.ts`): a number, or `{ key, limit, whenBusy }` per key. Full rule: `docs/history/jobs.md`.
|
|
88
|
-
- **`whenBusy: 'fail'` refuses only on EVIDENCE another run holds the key**
|
|
89
|
-
|
|
90
|
-
false })` → `failed`, outcome `refused` (`worker-key-busy.ts`), never `executeJob`.
|
|
82
|
+
- **`whenBusy: 'fail'` refuses only on EVIDENCE another run holds the key**
|
|
83
|
+
(`LeaseStore.holders`); settled `failed`, outcome `refused`. †
|
|
91
84
|
- **A key underivable at claim is `SlotGrant` `undecidable`** → `executeJob({ refusal })`: the
|
|
92
85
|
ATTEMPT fails, never the round.
|
|
93
86
|
- **`finalAttempt` is `isFinalAttempt(handle.retry, attempt)`**, REQUIRED on `JobRunArgs`.
|
|
94
87
|
|
|
95
88
|
## Rules — the worker and the scheduler
|
|
96
89
|
|
|
97
|
-
- **One enqueue implementation**: `jobsFacade()` owns `enqueue` AND `enqueueMany`, over
|
|
98
|
-
|
|
99
|
-
the
|
|
90
|
+
- **One enqueue implementation**: `jobsFacade()` owns `enqueue` AND `enqueueMany`, over
|
|
91
|
+
`enqueue-record.ts`'s row builders; the only other `driver.enqueue` sites are the outbox relay and
|
|
92
|
+
the scheduler. `handle.as(actor, input)` QUEUES; `executeJob` is the one execution path.
|
|
100
93
|
- The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
|
|
101
94
|
job's plain key.
|
|
102
95
|
- **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
|
|
103
|
-
the poll timer, arm the
|
|
104
|
-
`start()` refuses while draining.
|
|
96
|
+
the poll timer, arm the cut-off; aborts nothing); `close` is the teardown. `stop()` releases both
|
|
97
|
+
in a `finally`; `start()` refuses while draining.
|
|
105
98
|
- **A claimed job is counted with core's `beginWork()`**: the drain's in-flight phase waits. The
|
|
106
99
|
teardown's wait is bounded on SIGTERM (`drain-wait.ts`), unbounded on `stop()`; one teardown, joined.
|
|
107
|
-
- **A deploy lets a running job FINISH**: SIGTERM only stops claiming
|
|
108
|
-
|
|
109
|
-
uncounted (`worker-held.ts`). Settles **`interrupted`**; `docs/history/jobs.md`, "Drain, 2026-10".
|
|
100
|
+
- **A deploy lets a running job FINISH**: SIGTERM only stops claiming; `X_DRAINING` at
|
|
101
|
+
`deadlineAt − margin`; settles `interrupted`. †
|
|
110
102
|
- **`X_RATE_LIMITED` defers uncounted** (`rateLimitDeferralMs`).
|
|
111
103
|
- **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
|
|
112
104
|
synchronously with its guard.
|
|
113
|
-
- **A fleet slot is
|
|
114
|
-
with no renewal landing, CANCELS the run (`X_JOB_SLOT_LOST`); held per job id as a LIST.
|
|
105
|
+
- **A fleet slot is HELD, not owned**: a lost one CANCELS the run (`X_JOB_SLOT_LOST`). †
|
|
115
106
|
- **The run's signal is a controller this worker owns** (`run-signal.ts`).
|
|
116
|
-
- **A lease is HELD, not owned** (`heartbeat.ts`):
|
|
117
|
-
|
|
107
|
+
- **A lease is HELD, not owned** (`heartbeat.ts`): a whole window without a renewal is
|
|
108
|
+
`jobs.lease.lost`. †
|
|
118
109
|
- **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
|
|
119
110
|
interval is `unref`ed; every renewal is armed through `WorkerOptions.schedule`.
|
|
120
111
|
- **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
|
|
121
112
|
- **The retry decision reads the ERROR, on core's backoff curve** (`retry-classification.ts`,
|
|
122
113
|
`backoffDelayMs`): `docs/history/jobs.md`, "Moved 2026-10-01 — retry and bounds".
|
|
123
114
|
- **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
|
|
124
|
-
- **A
|
|
125
|
-
`ctx.signal` aborts unless the body completed AND its ack landed; steps fence on `ended`.
|
|
126
|
-
- **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`) — save a fenced
|
|
127
|
-
`X_DRAINING` on a live attempt: a step done in the drain is recorded once. The STORE fences
|
|
128
|
-
the claim (`StepFence`, `SQL_STEP_PUT` as `SQL_ACK`: `X_JOB_LEASE_LOST`). `executeJob` passes it.
|
|
129
|
-
- **A cancelled body's writes refuse too**, `enqueue` included (`outbox-abandon.test.ts`); a store is on the pool: a fenced `put` past `X_DRAINING` lands.
|
|
130
|
-
- **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
|
|
115
|
+
- **A FULL pass marks a backlog**: the next freed slot claims at once (`worker-backlog.test.ts`).
|
|
131
116
|
- **The scheduler runs one dispatch round at a time**; `stop()` waits it out before `leader.release()`. Full rule: `docs/history/jobs.md`.
|
|
132
117
|
- **`run-once` fires ONE catch-up**: the watermark rides the fire (`ScheduledFire.watermarkMs`),
|
|
133
118
|
never a `markFired` behind it; `skip` fires the latest missed occurrence (`latestOccurrenceBy`).
|
|
134
|
-
- **A lease for a run that never STARTED is `abandon()`ed, never `release()`d
|
|
135
|
-
|
|
136
|
-
catch); `timers` is the test seam.
|
|
119
|
+
- **A lease for a run that never STARTED is `abandon()`ed, never `release()`d**; a failed pass
|
|
120
|
+
re-arms at the floor. †
|
|
137
121
|
- **Every timer body catches before it finalises.**
|
|
138
|
-
-
|
|
139
|
-
`
|
|
140
|
-
|
|
141
|
-
- **A wiring throw hands back the run's holdings, and its ROW until `executeJob` starts**
|
|
142
|
-
(`worker-run.ts`), as a failed shed does its job (`worker-admit.ts`).
|
|
143
|
-
- **`step.run` hydrates the run's steps from ONE `store.list(runId)`**; `sleep` and `waitForEvent`
|
|
144
|
-
record replays through `trace()`. **`claimName` reads a Set**; `usedNames()` keeps
|
|
145
|
-
`MAX_TRACE_NAMES` (200).
|
|
122
|
+
- Cancellation, deadlines, `traceparent`, suspension vs shed, wiring throws:
|
|
123
|
+
`docs/history/jobs-worker-cancellation.md` (moved 2026-11-09).
|
|
124
|
+
- **`step.run` hydrates from ONE `store.list(runId)`**; `claimName` reads a Set. †
|
|
146
125
|
|
|
147
126
|
## Rules — long form in `docs/history/jobs.md`
|
|
148
127
|
|
|
@@ -156,41 +135,32 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
156
135
|
|
|
157
136
|
Long form: `docs/history/jobs.md`, "Moved 2026-10-01 — the outbox".
|
|
158
137
|
|
|
159
|
-
- **The relay drains in two phases**; `relay.stop()` JOINS the pass in flight
|
|
160
|
-
a later shutdown binds LATE (the worker's shape); its timer is `unref`ed.
|
|
138
|
+
- **The relay drains in two phases**; `relay.stop()` JOINS the pass in flight. †
|
|
161
139
|
- **The claim is a LEASE in one statement**, fenced on every mutation (`and claimed_by = $n`);
|
|
162
140
|
`claimLeaseMs` is normalised in `outbox-lease.ts`. `outbox-claim.test.ts` pins both stores.
|
|
163
141
|
- **The memory outbox store DELETES a published row** (`retained()` is the seam).
|
|
164
|
-
- **A staged row's id IS its job's id
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
`markPublishedMany` when the store has it; any failure drops that batch to the row loop, which
|
|
169
|
-
keeps the order. `RELAY_BATCH_SIZE` (500) is measured in its doc comment.
|
|
142
|
+
- **A staged row's id IS its job's id; its `runId` is allocated at stage**; a repeated publish is
|
|
143
|
+
`deduped`. †
|
|
144
|
+
- **The relay publishes a batch in two statements** (`publishBatch`); a failure falls back to the
|
|
145
|
+
row loop. †
|
|
170
146
|
|
|
171
147
|
## Rules — factories over `job()`
|
|
172
148
|
|
|
173
149
|
Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-01 — factories".
|
|
174
150
|
|
|
175
151
|
- **`backfill()`**: a step persists the CURSOR, never the page; **`handle` is AT LEAST ONCE**. Full rule: `docs/history/jobs.md`.
|
|
176
|
-
- **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff
|
|
177
|
-
`
|
|
178
|
-
only; `count` after the last batch (`X_BACKFILL_STALLED`). `inspectBackfills()` is the ONE projection.
|
|
152
|
+
- **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff**;
|
|
153
|
+
`inspectBackfills()` is the ONE projection. †
|
|
179
154
|
- **`rollup()`/`refreshView()`**: ONE `upsertAll`; `rollup.job.test.ts`.
|
|
180
155
|
- **`purge()`** is the one caller of every `purgeExpired()`: one table per `step.run`, one clock
|
|
181
156
|
reading per pass, duplicate names refused.
|
|
182
157
|
- **`exportRows()`**: one object per PAGE, named by page index; NO cross-tenant escape.
|
|
183
|
-
- **`importRows()`**: one step per PAGE, checkpointed at the BYTE OFFSET
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
the ack (503 otherwise); the payload is stored `.sensitive()`, never re-verified.
|
|
190
|
-
`webhookReceiverRoutes()` is what the boots mount.
|
|
191
|
-
- **`webhook()`**: ONE event to ONE endpoint, no steps; each attempt but a CANCELLED one recorded
|
|
192
|
-
before the throw; `ledger.isDisabled` gates the socket. **The wire format is core's.** An org
|
|
193
|
-
tenant reads `orgId` OFF THE INPUT; none, or not the run's org: TERMINAL `X_JOB_TENANT_MISMATCH`.
|
|
158
|
+
- **`importRows()`**: one step per PAGE, checkpointed at the BYTE OFFSET; AT LEAST ONCE per page;
|
|
159
|
+
no cross-tenant escape. †
|
|
160
|
+
- **`webhookReceiver()`**: read under `maxBytes`, `verify` before any parse, the enqueue before the
|
|
161
|
+
ack (503 otherwise); `webhookReceiverRoutes()` is what the boots mount. †
|
|
162
|
+
- **`webhook()`**: ONE event to ONE endpoint; core's wire format; tenant from the input, else
|
|
163
|
+
terminal `X_JOB_TENANT_MISMATCH`. †
|
|
194
164
|
|
|
195
165
|
## Files
|
|
196
166
|
|
|
@@ -199,30 +169,21 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
199
169
|
| `job.ts` | the `job()` primitive + registry + the handle's fluent surface + `registerJob` |
|
|
200
170
|
| `tenant.ts` | what tenant a run acts under: `JobTenant`, the declaration's backstop, the actor `executeJob` installs |
|
|
201
171
|
| `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input |
|
|
202
|
-
| `backfill
|
|
203
|
-
| `backfill-scope.ts` | which sweeps run across tenants (`tenant: 'none'` only) |
|
|
204
|
-
| `backfill-ledger.ts` | `x_backfills` — the contract, `BACKFILL_STATUSES`, the checksum, the verdict, the memory ledger |
|
|
205
|
-
| `backfill-registry.ts` | what was DECLARED: the `origin` stamp, `isBackfill`, `registeredBackfills` |
|
|
206
|
-
| `backfill-gate.ts` | may this sweep run here and now — environment, `requires`, already-applied |
|
|
207
|
-
| `backfill-pending.ts` | declared minus completed, per environment: the alarm `--pending` reads |
|
|
208
|
-
| `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait |
|
|
209
|
-
| `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP |
|
|
210
|
-
| `backfill-errors.ts` | the eight `X_BACKFILL_*` classes; the codes stay declared in `error-codes.ts` |
|
|
211
|
-
| `backfill-definition.ts` / `backfill-source.ts` / `backfill-reader.ts` | the declaration types; exactly one of `source`/`pages`, the `<name>:<scope>` ledger key; reading either |
|
|
172
|
+
| `backfill-*.ts` | one pass + cursor checkpoints, cross-tenant scope, the `x_backfills` ledger, the declared registry, the gate, pending, rate, the `inspectBackfills` projection, `X_BACKFILL_*` classes, declaration/source/reader † |
|
|
212
173
|
| `consumer.ts` / `consumer-loop.ts` / `consumer-types.ts` | `consumer()` — a job + `.supervisor` task; one run's turns; the shapes |
|
|
213
|
-
| `consumer-registry.ts` /
|
|
174
|
+
| `consumer-registry.ts` / `-guard.ts` / `-errors.ts` | the `origin` stamp + `registeredConsumers`; the self-enqueue refusal; `X_CONSUMER_*` |
|
|
214
175
|
| `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements; an `action-job` projection is `X_ACTION_JOB_UNBRIDGED` |
|
|
215
176
|
| `describe.ts` | one handle's JSON projection; `describeJobs()` maps it |
|
|
216
177
|
| `steps.ts` / `-core.ts` / `-wait.ts` / `-unwind.ts` / `step-scope.ts` | replay runner; core; `waitForEvent`; compensation; `currentStep()` |
|
|
217
|
-
| `steps-timeout.ts` /
|
|
218
|
-
| `steps-api.ts` /
|
|
178
|
+
| `steps-timeout.ts` / `-suspension.ts` | a step's ceiling; `StepSuspension`, `UnwindReached`, timeout marker |
|
|
179
|
+
| `steps-api.ts` / `-store.ts` / `-output.ts` | `StepApi`, `EventLookup`; `StepRecord`, `StepFence`, `StepStore`; what a step answers, `StepOutput` and `jsonForm` |
|
|
219
180
|
| `type-pins.ts` | compile-time pins for `StepOutput` and `waitForEvent` |
|
|
220
181
|
| `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
|
|
221
182
|
| `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
|
|
222
183
|
| `outbox-pg.ts` | `postgresOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
|
|
223
184
|
| `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
|
|
224
185
|
| `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
|
|
225
|
-
| `concurrency.ts` | `job.concurrency`
|
|
186
|
+
| `concurrency.ts` | `job.concurrency` resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
|
|
226
187
|
| `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `error-codes.ts` |
|
|
227
188
|
| `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
|
|
228
189
|
| `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
|
|
@@ -231,58 +192,62 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
|
|
|
231
192
|
| `driver.ts` | `JobDriver` contract + wire records |
|
|
232
193
|
| `driver-pg.ts` | default driver, and `postgresLeader` |
|
|
233
194
|
| `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the ONE install point. Its comments carry no `;` and no `'` |
|
|
234
|
-
| `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and `JOB_ROW_COLUMNS`; re-exported from `driver-pg-sql.ts` |
|
|
235
195
|
| `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
|
|
236
196
|
| `driver-memory.ts` / `claim-exhausted.ts` | `x dev` / tests; a buried row's log line and `onSettled` |
|
|
237
|
-
| `retry.ts` | the dead-letter decision
|
|
197
|
+
| `retry.ts` | the dead-letter decision; this package's option names over core's `backoffDelay`, no curve |
|
|
238
198
|
| `retry-classification.ts` | the other half: what the thrown error says, and the stop reason |
|
|
199
|
+
| `retry-jitter.ts` | `retry.jitter` refused at `job()` unless a `JitterMode` (a v30 boolean names its mapping) |
|
|
239
200
|
| `execute-unwind.ts` / `execute-ctx.ts` | a saga failed for good; caller ctx |
|
|
240
201
|
| `execute.ts` (+`-result.ts`) / `run-deadline.ts` | `executeJob`: one claimed job run and settled; its deadline: cancel, then fail |
|
|
241
202
|
| `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
|
|
242
203
|
| `renewal-timer.ts` | the renewal interval and its `stopped()` latch |
|
|
243
204
|
| `worker.ts` | `worker` role, claim round, drain |
|
|
244
|
-
| `worker-drain-cutoff.ts` /
|
|
205
|
+
| `worker-drain-cutoff.ts` / `-held.ts` / `-tally.ts` | drain cut-off · held claims · counters |
|
|
245
206
|
| `worker-admit.ts` | may this claimed job start: the limiter, the fleet slot, and the shed / refusal |
|
|
246
207
|
| `worker-registry.ts` | one worker's registry row: announce, heartbeat, forget |
|
|
247
|
-
| `worker-types.ts` | the worker's
|
|
208
|
+
| `worker-types.ts` | the worker's contract: `WorkerOptions`, `WorkerStats`, `Worker` |
|
|
248
209
|
| `drain-wait.ts` | the drain's wait, shared by both roles |
|
|
249
|
-
| `worker-run.ts` | one claimed job
|
|
210
|
+
| `worker-run.ts` | one claimed job: heartbeat, slot renewal, run signal, consumer span, `job.perform` |
|
|
211
|
+
| `job-instrument.ts` | the producer span + `job.enqueue` per handle enqueue; both spans' attributes |
|
|
250
212
|
| `run-signal.ts` | the signal ONE run is cancelled by |
|
|
251
|
-
| `worker-fleet-slots.ts` |
|
|
213
|
+
| `worker-fleet-slots.ts` | an in-flight job's fleet slot: take, renew, hand back; the boot refusal |
|
|
252
214
|
| `worker-process-cap.ts` / `worker-claim-sweep.ts` | `maxInFlight`'s count; the pass's asks, rotated |
|
|
253
215
|
| `purge.ts` | `purge()` — a factory over `job()`: the retention sweep and its target seam |
|
|
254
216
|
| `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
|
|
255
217
|
| `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
|
|
256
218
|
| `limits.ts` | per-tenant / per-queue / global concurrency + rate |
|
|
257
|
-
| `events.ts` |
|
|
219
|
+
| `events.ts` / `-cursor.ts` | memory bus; the wait cursor, both buses |
|
|
258
220
|
| `inspect.ts` | `--json` introspection. `operatorInput`: no registered handle gives `[redacted]` whole; otherwise `redactInput(maskSensitive(handle.input, …))` |
|
|
259
221
|
| `inspect-operator.ts` | the verbs `x jobs` binds: remove, promote, pause, resume |
|
|
260
222
|
| `introspection.ts` | `JobIntrospection`, every operator type, and the bounds |
|
|
261
223
|
| `counters.ts` | bucket arithmetic, `nackOutcome`, the memory counters |
|
|
262
224
|
| `driver-memory-operator.ts` / `driver-pg-operator.ts` | each driver's operator members |
|
|
263
|
-
| `driver-pg-operator-sql.ts` / `driver-pg-settle-sql.ts` | their statements; `SQL_ACK` / `SQL_NACK` with the counter |
|
|
264
225
|
| `progress.ts` / `settled.ts` / `redact-input.ts` | the progress throttle; the `onSettled` runner; a payload as an operator reads it |
|
|
265
226
|
| `scheduler-state.ts` | `SchedulerState`, `fire`, the memory watermark |
|
|
266
227
|
| `scheduler-leader.ts` / `scheduler-occurrences.ts` | `LeaderElection` + `soleLeader`; which occurrences fall in a window |
|
|
267
228
|
| `idle-backoff.ts` / `enqueue-signal.ts` | the wait both polling loops share; the two wake signals and `wakeIsLive` |
|
|
268
229
|
| `queue-wake.ts` / `driver-pg-wake-sql.ts` | the `LISTEN` session and its proof; which statement notifies |
|
|
269
230
|
| `worker-loop.ts` | the claim loop's timer: wake, `again`, the kicks |
|
|
270
|
-
| `driver-pg-outbox-sql.ts` | the `x_outbox` statements |
|
|
271
231
|
| `errors-operator.ts` / `errors-requeue.ts` | the operator verbs' refusals; `requeue`'s |
|
|
272
232
|
| `errors-tenant.ts` | `JobTenantMismatchError` |
|
|
273
233
|
| `errors-steps.ts` / `errors-saga.ts` / `step-resolve.ts` | step, saga + `once` refusals; `resolveStep` |
|
|
274
|
-
| `worker-queue-demand.ts` | the `queue_demand` gauge: `driver.demand()`, skipped
|
|
275
|
-
| `driver-pg
|
|
234
|
+
| `worker-queue-demand.ts` | the `queue_demand` gauge: `driver.demand()`, skipped on no work, 0 at the drain |
|
|
235
|
+
| `driver-pg-*-sql.ts` | one file per statement family: jobs rows (`JOB_ROW_COLUMNS`), operator, settle (`SQL_ACK`/`SQL_NACK`), outbox, demand (`SQL_QUEUE_DEMAND`) † |
|
|
276
236
|
| `driver-pg-claim-sql.ts` | `SQL_CLAIM` and why it seeks; re-exported from `driver-pg-sql.ts` |
|
|
277
237
|
| `error-codes.ts` / `errors.ts` | codes, titles, their registration (`new-error-code` writes here) / the classes |
|
|
278
238
|
| `job-enqueue-many.ts` / `enqueue-record.ts` / `driver-pg-enqueue-many.ts` | `enqueueMany`, `MAX_ENQUEUE_BATCH`, `ENQUEUE_CHUNK`; one enqueue's row, every path; `SQL_ENQUEUE_MANY` + resolve |
|
|
279
239
|
| `import.ts` / `import-pass.ts` | `importRows()`: the declaration; one pass, its byte-offset checkpoints, the error report |
|
|
280
|
-
| `import-format.ts` /
|
|
240
|
+
| `import-format.ts` / `-source.ts` / `-errors.ts` | record boundaries on bytes; the `ImportSource` seam; the `X_IMPORT_*` classes |
|
|
281
241
|
| `driver-redis-refusal.ts` | `typedRefusals`: a send refused for want of a connection → `X_DRIVER_UNAVAILABLE` |
|
|
282
242
|
| `subject*.ts` | `subjectExport`/`subjectErase`; reach only via DELETED tables (README) |
|
|
243
|
+
| `gdpr-*.ts` | `.personalData()` coverage (thrown by `plannedSubject`, listed by `x exercise`); the erase's audit record |
|
|
244
|
+
| `ledger*.ts` | `ledger()`: balanced per currency, ONE step; a key is one posting per tenant, re-used with other lines `X_LEDGER_POSTING_CONFLICT` |
|
|
245
|
+
| `lock*.ts` | `withLock`: one `lock:<key>` slot; token = the store's `fence`; renewal as `worker-fleet-slots.ts` |
|
|
246
|
+
| `reindex.ts` | `reindex()`: a `backfill()` into the `searchIndex` port via `searchDocumentOf` |
|
|
247
|
+
| `partitions.ts` | `partitions()` + `.schedule`: db's `partitionMaintenance` per partitioned entity, job clock |
|
|
283
248
|
| `webhook-receiver.ts` / `-route.ts` / `-types.ts` / `-errors.ts` | `webhookReceiver()` and `webhookReceiverRoutes`; the route; `ReceivedEvent`, `WebhookInbound`; its three refusals |
|
|
284
249
|
|
|
285
|
-
Commands: `bun test packages/jobs
|
|
250
|
+
Commands: `bun test ./packages/jobs/` · `bun run --filter @ultimat3/jobs typecheck`. Test harnesses:
|
|
286
251
|
history, "Moved 2026-10-01 — test harnesses".
|
|
287
252
|
|
|
288
|
-
Why
|
|
253
|
+
Why: [`docs/history/jobs.md`](../../docs/history/jobs.md).
|