@ultimat3/jobs 30.9.0 → 31.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (60) hide show
  1. package/CLAUDE.md +72 -107
  2. package/README.md +202 -55
  3. package/package.json +8 -7
  4. package/src/adapter.ts +37 -0
  5. package/src/backfill-ledger.ts +2 -2
  6. package/src/backfill-rate.ts +1 -0
  7. package/src/clock.ts +2 -2
  8. package/src/drain-wait.ts +1 -0
  9. package/src/driver-memory.ts +2 -2
  10. package/src/driver-pg-ddl.ts +19 -0
  11. package/src/driver-pg-event-wake-sql.ts +43 -22
  12. package/src/driver-pg-sql.ts +68 -27
  13. package/src/driver-pg.ts +11 -10
  14. package/src/driver-redis.ts +2 -2
  15. package/src/driver.ts +14 -8
  16. package/src/error-codes.ts +27 -0
  17. package/src/errors-concurrency.ts +1 -1
  18. package/src/errors.ts +1 -1
  19. package/src/events-cursor.ts +31 -0
  20. package/src/events-pg.ts +52 -6
  21. package/src/events.ts +72 -36
  22. package/src/export.ts +2 -2
  23. package/src/gdpr-audit.ts +75 -0
  24. package/src/gdpr-coverage.ts +94 -0
  25. package/src/gdpr-errors.ts +35 -0
  26. package/src/import.ts +2 -2
  27. package/src/index.ts +45 -206
  28. package/src/inspect-operator.ts +2 -2
  29. package/src/inspect.ts +1 -1
  30. package/src/job-instrument.ts +115 -0
  31. package/src/job.ts +41 -4
  32. package/src/jobs-retention.ts +23 -6
  33. package/src/leases.ts +13 -3
  34. package/src/ledger-book.ts +168 -0
  35. package/src/ledger-errors.ts +48 -0
  36. package/src/ledger.ts +145 -0
  37. package/src/limits.ts +57 -19
  38. package/src/lock-errors.ts +54 -0
  39. package/src/lock.ts +176 -0
  40. package/src/outbox-relay.ts +4 -0
  41. package/src/outbox.ts +3 -3
  42. package/src/partitions.ts +119 -0
  43. package/src/reindex-sweep.ts +94 -0
  44. package/src/reindex.ts +147 -0
  45. package/src/retry-classification.ts +3 -2
  46. package/src/retry-jitter.ts +32 -0
  47. package/src/retry.ts +13 -16
  48. package/src/scheduler.ts +10 -6
  49. package/src/steps-api.ts +44 -5
  50. package/src/steps-suspension.ts +34 -3
  51. package/src/steps-wait.ts +52 -4
  52. package/src/steps.ts +1 -0
  53. package/src/subject-jobs.ts +12 -4
  54. package/src/webhook.ts +2 -2
  55. package/src/worker-admit.ts +8 -5
  56. package/src/worker-drain-cutoff.ts +1 -0
  57. package/src/worker-loop.ts +28 -5
  58. package/src/worker-registry.ts +7 -5
  59. package/src/worker-run.ts +53 -9
  60. package/src/worker.ts +8 -8
package/CLAUDE.md CHANGED
@@ -1,16 +1,20 @@
1
1
  # @ultimat3/jobs — agent notes
2
2
 
3
3
  Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queue drivers.
4
+ A rule or row marked `†` has its full text in [`docs/history/jobs-rules.md`](../../docs/history/jobs-rules.md).
4
5
 
5
6
  ## Boundary
6
7
 
7
8
  - May import: `core`, `schema`, `entity`, `policy`, `cache`, `time` — and `db`, for
8
- `expectedQueryLoop` (`steps.ts` declares the per-step write one-per-step to the N+1 detector),
9
- `outsideTransaction` (`execute.ts`), `refuseAbandoned` (`outbox.ts`) and `withTransaction`
10
- (`consumer-loop.ts`: a batch and its cursor commit together) — the ONE place a client is taken
11
- from it, the ambient `db()`. Never `render`, `ui`. `http` only from `webhook-receiver*.ts`, for the route
9
+ `expectedQueryLoop` (`steps.ts`, the per-step write to the N+1 detector), `outsideTransaction`
10
+ (`execute.ts`), `currentTx` (`events-pg.ts`), `refuseAbandoned` (`outbox.ts`) and
11
+ `withTransaction` (`consumer-loop.ts`: a batch and its cursor commit together) — a client is
12
+ taken from it in ONE place, the ambient `db()`. Never `render`, `ui`. `http` only from `webhook-receiver*.ts`, for the route
12
13
  (`Route`/`Bucket` types, `DEFAULT_WEBHOOK_BODY_LIMIT`, `HttpError`, `jsonResponse`).
13
14
  - Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
15
+ - **Installers are on `@ultimat3/jobs/adapter`, never the `.` barrel** (`adapter.test.ts`):
16
+ `installJobDriver`/`installEventBus` + resets, the port interfaces, the errors a driver throws.
17
+ An app swaps a port in `apps/<app>/runtime.ts`; a `fix:` names that line, never an installer.
14
18
  - External deps: none. Postgres access goes through the injected `PgExecutor` (`@ultimat3/core`'s,
15
19
  re-exported from `index.ts`).
16
20
 
@@ -23,7 +27,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
23
27
  `X_JOB_DUPLICATE` at the earliest decidable point (same `name:` inside `job()`/`task()`; a
24
28
  different handle, or the same handle under a second export name, at registration).
25
29
  - `registerJob`/`registerJobs`/`registerTask`/`registerTasks`/`nameJobs`/`nameTasks` are NOT in
26
- `src/index.ts`. `register.ts` announces the registrars in core's table at import — never remove it
30
+ `src/index.ts`, and there is no `./loader` subpath (action's and query's have one, for `x`'s
31
+ module scan): jobs and tasks reach their registries only through core's registrar table. `register.ts` announces the registrars in core's table at import — never remove it
27
32
  (`X_REGISTRAR_MISSING`). `isJobHandle`/`isTaskHandle` are structural plus proof `job()`/`task()`
28
33
  built it.
29
34
  - `idempotencyKey` is NON-OPTIONAL. **The namespace is `(name, coalesce(tenant_id, ''),
@@ -45,32 +50,21 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
45
50
  - **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`, `x_outbox`, `x_scheduler_*`). Full rule: `docs/history/jobs.md`.
46
51
  - **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
47
52
  `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's reads are `async` so a refusal rejects on both.
48
- - **`ack`, `nack`, `heartbeat` and `recordProgress` are FENCED on `state = 'running'` AND the
49
- CLAIM** — `{ workerId, claim }`, `claimOf(claimed)`; `x_jobs.claims` is moved by the claim and
50
- never reset. `heartbeat` answers a boolean, read as `held === false` (never `!held`).
53
+ - **`ack`, `nack`, `heartbeat`, `recordProgress` are FENCED on `running` AND the claim**
54
+ (`claimOf(claimed)`); read `heartbeat` as `held === false`. †
51
55
  - **A driver's semantics are pinned beside the pg statement** (`driver-parity*.test.ts`);
52
56
  `driver-*-fixture.ts` scenarios run on memory AND Postgres.
53
- - **The claim BURIES a lease that lapsed on the final attempt** (`attempt >= max_attempts`): `dead`
54
- in `SQL_CLAIM` — `failed` for a name in `ClaimOptions.dropExhausted` (`retry.deadLetter: false`)
55
- — counted, never returned; the worker's round hears `onExhausted` and calls `announceExhausted`.
57
+ - **The claim BURIES a lease that lapsed on the final attempt** (`dead`, or `failed` under
58
+ `dropExhausted`); the worker calls `announceExhausted`. †
56
59
  - **`cancel` is fenced on `LIVE_STATES`.** **`requeue`** refuses (`X_JOB_NOT_FOUND`, not requeueable,
57
60
  duplicate), then ONE statement. **`enqueue` inserts twice at most.** **`BulkResult.remaining` is
58
61
  what a second call would move.**
59
- - **`enqueueMany` is one statement per `ENQUEUE_CHUNK` (1,000)** (`job-enqueue-many.ts`;
60
- `SQL_ENQUEUE_MANY`: `SQL_ENQUEUE`'s conflict target and id fence, one wake per queue); skipped rows
61
- are answered by ONE `SQL_ENQUEUE_MANY_RESOLVE`, a row neither answers goes back through `enqueue`.
62
- **`JobDriver.enqueueMany` is REQUIRED** and opens with `assertEnqueueMany` (no `onConflict:
63
- 'error'` in a batch, `X_INVARIANT`); Redis's `REDIS_ENQUEUE_MANY` shares `enqueue_one` with
64
- `REDIS_ENQUEUE`. `OutboxStore.stageMany`/`markPublishedMany` are OPTIONAL: absent, a loop. Every
65
- refusal precedes the first write; `refuseAbandoned()` before every direct chunk. `runAt` crosses
66
- as `numeric` then `bigint`: a fractional ms rounds, never a `22P02`.
62
+ - **`enqueueMany` is one statement per `ENQUEUE_CHUNK` (1,000)**; `JobDriver.enqueueMany` is
63
+ REQUIRED; every refusal precedes the first write. †
67
64
  - **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
68
65
  (discovered, comments stripped) for `select *`/`returning *`.
69
- - Drivers: the `JobDriver` methods plus optional `introspect`, `backfills`, `leases`; new
70
- capabilities go behind the interface. **`redisJobDriver`** (`@ultimat3/jobs/redis`, off the
71
- barrel) is one Lua script per operation: a semantic change lands in memory, pg AND the scripts.
72
- Its client is core's `redisClient()`, never `Bun.redis`; `ERR_REDIS_CONNECTION_CLOSED` is
73
- `X_DRIVER_UNAVAILABLE` at once (`typedRefusals`), never a wait.
66
+ - Drivers: `JobDriver` + optional `introspect`/`backfills`/`leases`; `redisJobDriver`
67
+ (`@ultimat3/jobs/redis`) is one Lua script per operation — a change lands in memory, pg AND Lua. †
74
68
  - **`demand` is the seventh method, required**; a draining worker's `queue_demand` goes to 0.
75
69
  - **The claim SEEKS** (`SQL_CLAIM`); correct only because a claim autocommits.
76
70
  - Step results are persisted BEFORE the step returns, and it returns what it persisted: `jsonForm` (`steps-output.ts`), typed `StepOutput<T>`, one stringify + parse per EXECUTED step. Full rule: `docs/history/jobs.md`.
@@ -85,64 +79,49 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
85
79
  `limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
86
80
  throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
87
81
  - **`concurrency` is resolved ONCE** (`concurrency.ts`): a number, or `{ key, limit, whenBusy }` per key. Full rule: `docs/history/jobs.md`.
88
- - **`whenBusy: 'fail'` refuses only on EVIDENCE another run holds the key** (`LeaseStore.holders`,
89
- required): a run meets its own leftover slot. The refusal is `nack({ fail, countsAsAttempt:
90
- false })` → `failed`, outcome `refused` (`worker-key-busy.ts`), never `executeJob`.
82
+ - **`whenBusy: 'fail'` refuses only on EVIDENCE another run holds the key**
83
+ (`LeaseStore.holders`); settled `failed`, outcome `refused`. †
91
84
  - **A key underivable at claim is `SlotGrant` `undecidable`** → `executeJob({ refusal })`: the
92
85
  ATTEMPT fails, never the round.
93
86
  - **`finalAttempt` is `isFinalAttempt(handle.retry, attempt)`**, REQUIRED on `JobRunArgs`.
94
87
 
95
88
  ## Rules — the worker and the scheduler
96
89
 
97
- - **One enqueue implementation**: `jobsFacade()` owns `enqueue` AND `enqueueMany`, over the row
98
- builders in `enqueue-record.ts`, which the relay shares; the only other `driver.enqueue` sites are
99
- the outbox relay and the scheduler. `handle.as(actor, input)` QUEUES; `executeJob` is the one execution path.
90
+ - **One enqueue implementation**: `jobsFacade()` owns `enqueue` AND `enqueueMany`, over
91
+ `enqueue-record.ts`'s row builders; the only other `driver.enqueue` sites are the outbox relay and
92
+ the scheduler. `handle.as(actor, input)` QUEUES; `executeJob` is the one execution path.
100
93
  - The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
101
94
  job's plain key.
102
95
  - **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
103
- the poll timer, arm the drain cut-off — it aborts nothing); `close` is the teardown. `stop()` hands both back in a `finally`;
104
- `start()` refuses while draining.
96
+ the poll timer, arm the cut-off; aborts nothing); `close` is the teardown. `stop()` releases both
97
+ in a `finally`; `start()` refuses while draining.
105
98
  - **A claimed job is counted with core's `beginWork()`**: the drain's in-flight phase waits. The
106
99
  teardown's wait is bounded on SIGTERM (`drain-wait.ts`), unbounded on `stop()`; one teardown, joined.
107
- - **A deploy lets a running job FINISH**: SIGTERM only stops claiming. `drainSignal` aborts
108
- (`X_DRAINING`) at `deadlineAt − margin` (`worker-drain-cutoff.ts`), then held claims go back
109
- uncounted (`worker-held.ts`). Settles **`interrupted`**; `docs/history/jobs.md`, "Drain, 2026-10".
100
+ - **A deploy lets a running job FINISH**: SIGTERM only stops claiming; `X_DRAINING` at
101
+ `deadlineAt − margin`; settles `interrupted`. †
110
102
  - **`X_RATE_LIMITED` defers uncounted** (`rateLimitDeferralMs`).
111
103
  - **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
112
104
  synchronously with its guard.
113
- - **A fleet slot is taken INSIDE a `try`, released AWAITED, and HELD, not owned**: `false`, or a TTL
114
- with no renewal landing, CANCELS the run (`X_JOB_SLOT_LOST`); held per job id as a LIST.
105
+ - **A fleet slot is HELD, not owned**: a lost one CANCELS the run (`X_JOB_SLOT_LOST`). †
115
106
  - **The run's signal is a controller this worker owns** (`run-signal.ts`).
116
- - **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal warns; a whole window without
117
- one landing is `jobs.lease.lost`, on this process's clock, asked both sides of the call.
107
+ - **A lease is HELD, not owned** (`heartbeat.ts`): a whole window without a renewal is
108
+ `jobs.lease.lost`. †
118
109
  - **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
119
110
  interval is `unref`ed; every renewal is armed through `WorkerOptions.schedule`.
120
111
  - **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
121
112
  - **The retry decision reads the ERROR, on core's backoff curve** (`retry-classification.ts`,
122
113
  `backoffDelayMs`): `docs/history/jobs.md`, "Moved 2026-10-01 — retry and bounds".
123
114
  - **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
124
- - **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
125
- `ctx.signal` aborts unless the body completed AND its ack landed; steps fence on `ended`.
126
- - **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`) — save a fenced
127
- `X_DRAINING` on a live attempt: a step done in the drain is recorded once. The STORE fences
128
- the claim (`StepFence`, `SQL_STEP_PUT` as `SQL_ACK`: `X_JOB_LEASE_LOST`). `executeJob` passes it.
129
- - **A cancelled body's writes refuse too**, `enqueue` included (`outbox-abandon.test.ts`); a store is on the pool: a fenced `put` past `X_DRAINING` lands.
130
- - **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
115
+ - **A FULL pass marks a backlog**: the next freed slot claims at once (`worker-backlog.test.ts`).
131
116
  - **The scheduler runs one dispatch round at a time**; `stop()` waits it out before `leader.release()`. Full rule: `docs/history/jobs.md`.
132
117
  - **`run-once` fires ONE catch-up**: the watermark rides the fire (`ScheduledFire.watermarkMs`),
133
118
  never a `markFired` behind it; `skip` fires the latest missed occurrence (`latestOccurrenceBy`).
134
- - **A lease for a run that never STARTED is `abandon()`ed, never `release()`d** — the rate stamp
135
- goes with it (`worker-admit.ts`). **A failed pass re-arms at the floor** (`worker-loop.ts`
136
- catch); `timers` is the test seam.
119
+ - **A lease for a run that never STARTED is `abandon()`ed, never `release()`d**; a failed pass
120
+ re-arms at the floor. †
137
121
  - **Every timer body catches before it finalises.**
138
- - **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
139
- `nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
140
- stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
141
- - **A wiring throw hands back the run's holdings, and its ROW until `executeJob` starts**
142
- (`worker-run.ts`), as a failed shed does its job (`worker-admit.ts`).
143
- - **`step.run` hydrates the run's steps from ONE `store.list(runId)`**; `sleep` and `waitForEvent`
144
- record replays through `trace()`. **`claimName` reads a Set**; `usedNames()` keeps
145
- `MAX_TRACE_NAMES` (200).
122
+ - Cancellation, deadlines, `traceparent`, suspension vs shed, wiring throws:
123
+ `docs/history/jobs-worker-cancellation.md` (moved 2026-11-09).
124
+ - **`step.run` hydrates from ONE `store.list(runId)`**; `claimName` reads a Set. †
146
125
 
147
126
  ## Rules — long form in `docs/history/jobs.md`
148
127
 
@@ -156,41 +135,32 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
156
135
 
157
136
  Long form: `docs/history/jobs.md`, "Moved 2026-10-01 — the outbox".
158
137
 
159
- - **The relay drains in two phases**; `relay.stop()` JOINS the pass in flight, under a `DrainBudget`
160
- a later shutdown binds LATE (the worker's shape); its timer is `unref`ed.
138
+ - **The relay drains in two phases**; `relay.stop()` JOINS the pass in flight. †
161
139
  - **The claim is a LEASE in one statement**, fenced on every mutation (`and claimed_by = $n`);
162
140
  `claimLeaseMs` is normalised in `outbox-lease.ts`. `outbox-claim.test.ts` pins both stores.
163
141
  - **The memory outbox store DELETES a published row** (`retained()` is the seam).
164
- - **A staged row's id IS its job's id, and its `runId` is allocated at stage** (`x_outbox.run_id`),
165
- so a staged enqueue answers real ids. `SQL_ENQUEUE` inserts nothing under an id that names a
166
- row — a repeated publish is `deduped`, live or finished (`outbox-run-id.test.ts`).
167
- - **The relay publishes a batch in two statements** (`publishBatch`): `driver.enqueueMany`, then
168
- `markPublishedMany` when the store has it; any failure drops that batch to the row loop, which
169
- keeps the order. `RELAY_BATCH_SIZE` (500) is measured in its doc comment.
142
+ - **A staged row's id IS its job's id; its `runId` is allocated at stage**; a repeated publish is
143
+ `deduped`. †
144
+ - **The relay publishes a batch in two statements** (`publishBatch`); a failure falls back to the
145
+ row loop. †
170
146
 
171
147
  ## Rules — factories over `job()`
172
148
 
173
149
  Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-01 — factories".
174
150
 
175
151
  - **`backfill()`**: a step persists the CURSOR, never the page; **`handle` is AT LEAST ONCE**. Full rule: `docs/history/jobs.md`.
176
- - **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff.**
177
- `environments` is checked in `backfillPass()` and `gateBackfill()`; `requires` in `gateBackfill()`
178
- only; `count` after the last batch (`X_BACKFILL_STALLED`). `inspectBackfills()` is the ONE projection.
152
+ - **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff**;
153
+ `inspectBackfills()` is the ONE projection. †
179
154
  - **`rollup()`/`refreshView()`**: ONE `upsertAll`; `rollup.job.test.ts`.
180
155
  - **`purge()`** is the one caller of every `purgeExpired()`: one table per `step.run`, one clock
181
156
  reading per pass, duplicate names refused.
182
157
  - **`exportRows()`**: one object per PAGE, named by page index; NO cross-tenant escape.
183
- - **`importRows()`**: one step per PAGE, checkpointed at the BYTE OFFSET, no carry between steps;
184
- `pageBytes` > `maxRecordBytes` is refused at declaration. `handle` is AT LEAST ONCE per page. The
185
- error report never carries the refused value. NO cross-tenant escape.
186
- - **`webhookReceiver()`**: a plain `Route` read under `maxBytes` (core's `readWithinLimit`, never
187
- `bodyBytes()`); `verify` runs before any parse and answers `{ eventId, payload }` from verified
188
- data; the id is bounded (`WEBHOOK_EVENT_ID_PATTERN`, 422); the enqueue is `outbox: false` before
189
- the ack (503 otherwise); the payload is stored `.sensitive()`, never re-verified.
190
- `webhookReceiverRoutes()` is what the boots mount.
191
- - **`webhook()`**: ONE event to ONE endpoint, no steps; each attempt but a CANCELLED one recorded
192
- before the throw; `ledger.isDisabled` gates the socket. **The wire format is core's.** An org
193
- tenant reads `orgId` OFF THE INPUT; none, or not the run's org: TERMINAL `X_JOB_TENANT_MISMATCH`.
158
+ - **`importRows()`**: one step per PAGE, checkpointed at the BYTE OFFSET; AT LEAST ONCE per page;
159
+ no cross-tenant escape. †
160
+ - **`webhookReceiver()`**: read under `maxBytes`, `verify` before any parse, the enqueue before the
161
+ ack (503 otherwise); `webhookReceiverRoutes()` is what the boots mount. †
162
+ - **`webhook()`**: ONE event to ONE endpoint; core's wire format; tenant from the input, else
163
+ terminal `X_JOB_TENANT_MISMATCH`. †
194
164
 
195
165
  ## Files
196
166
 
@@ -199,30 +169,21 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
199
169
  | `job.ts` | the `job()` primitive + registry + the handle's fluent surface + `registerJob` |
200
170
  | `tenant.ts` | what tenant a run acts under: `JobTenant`, the declaration's backstop, the actor `executeJob` installs |
201
171
  | `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input |
202
- | `backfill-pass.ts` | one pass: the batched iteration, its cursor checkpoints and its ledger row |
203
- | `backfill-scope.ts` | which sweeps run across tenants (`tenant: 'none'` only) |
204
- | `backfill-ledger.ts` | `x_backfills` — the contract, `BACKFILL_STATUSES`, the checksum, the verdict, the memory ledger |
205
- | `backfill-registry.ts` | what was DECLARED: the `origin` stamp, `isBackfill`, `registeredBackfills` |
206
- | `backfill-gate.ts` | may this sweep run here and now — environment, `requires`, already-applied |
207
- | `backfill-pending.ts` | declared minus completed, per environment: the alarm `--pending` reads |
208
- | `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait |
209
- | `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP |
210
- | `backfill-errors.ts` | the eight `X_BACKFILL_*` classes; the codes stay declared in `error-codes.ts` |
211
- | `backfill-definition.ts` / `backfill-source.ts` / `backfill-reader.ts` | the declaration types; exactly one of `source`/`pages`, the `<name>:<scope>` ledger key; reading either |
172
+ | `backfill-*.ts` | one pass + cursor checkpoints, cross-tenant scope, the `x_backfills` ledger, the declared registry, the gate, pending, rate, the `inspectBackfills` projection, `X_BACKFILL_*` classes, declaration/source/reader † |
212
173
  | `consumer.ts` / `consumer-loop.ts` / `consumer-types.ts` | `consumer()` — a job + `.supervisor` task; one run's turns; the shapes |
213
- | `consumer-registry.ts` / `consumer-guard.ts` / `consumer-errors.ts` | the `origin` stamp + `registeredConsumers`; the self-enqueue refusal; `X_CONSUMER_*` |
174
+ | `consumer-registry.ts` / `-guard.ts` / `-errors.ts` | the `origin` stamp + `registeredConsumers`; the self-enqueue refusal; `X_CONSUMER_*` |
214
175
  | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements; an `action-job` projection is `X_ACTION_JOB_UNBRIDGED` |
215
176
  | `describe.ts` | one handle's JSON projection; `describeJobs()` maps it |
216
177
  | `steps.ts` / `-core.ts` / `-wait.ts` / `-unwind.ts` / `step-scope.ts` | replay runner; core; `waitForEvent`; compensation; `currentStep()` |
217
- | `steps-timeout.ts` / `steps-suspension.ts` | a step's ceiling; `StepSuspension`, `UnwindReached`, timeout marker |
218
- | `steps-api.ts` / `steps-store.ts` / `steps-output.ts` | `StepApi`, `EventLookup`; `StepRecord`, `StepFence`, `StepStore`; what a step answers, `StepOutput` and `jsonForm` |
178
+ | `steps-timeout.ts` / `-suspension.ts` | a step's ceiling; `StepSuspension`, `UnwindReached`, timeout marker |
179
+ | `steps-api.ts` / `-store.ts` / `-output.ts` | `StepApi`, `EventLookup`; `StepRecord`, `StepFence`, `StepStore`; what a step answers, `StepOutput` and `jsonForm` |
219
180
  | `type-pins.ts` | compile-time pins for `StepOutput` and `waitForEvent` |
220
181
  | `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
221
182
  | `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
222
183
  | `outbox-pg.ts` | `postgresOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
223
184
  | `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
224
185
  | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
225
- | `concurrency.ts` | `job.concurrency` as declared, resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
186
+ | `concurrency.ts` | `job.concurrency` resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
226
187
  | `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `error-codes.ts` |
227
188
  | `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
228
189
  | `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
@@ -231,58 +192,62 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
231
192
  | `driver.ts` | `JobDriver` contract + wire records |
232
193
  | `driver-pg.ts` | default driver, and `postgresLeader` |
233
194
  | `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the ONE install point. Its comments carry no `;` and no `'` |
234
- | `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and `JOB_ROW_COLUMNS`; re-exported from `driver-pg-sql.ts` |
235
195
  | `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
236
196
  | `driver-memory.ts` / `claim-exhausted.ts` | `x dev` / tests; a buried row's log line and `onSettled` |
237
- | `retry.ts` | the dead-letter decision, and this package's option names over core's `backoffDelay` — no curve of its own |
197
+ | `retry.ts` | the dead-letter decision; this package's option names over core's `backoffDelay`, no curve |
238
198
  | `retry-classification.ts` | the other half: what the thrown error says, and the stop reason |
199
+ | `retry-jitter.ts` | `retry.jitter` refused at `job()` unless a `JitterMode` (a v30 boolean names its mapping) |
239
200
  | `execute-unwind.ts` / `execute-ctx.ts` | a saga failed for good; caller ctx |
240
201
  | `execute.ts` (+`-result.ts`) / `run-deadline.ts` | `executeJob`: one claimed job run and settled; its deadline: cancel, then fail |
241
202
  | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
242
203
  | `renewal-timer.ts` | the renewal interval and its `stopped()` latch |
243
204
  | `worker.ts` | `worker` role, claim round, drain |
244
- | `worker-drain-cutoff.ts` / `worker-held.ts` / `worker-tally.ts` | drain cut-off · held claims · counters |
205
+ | `worker-drain-cutoff.ts` / `-held.ts` / `-tally.ts` | drain cut-off · held claims · counters |
245
206
  | `worker-admit.ts` | may this claimed job start: the limiter, the fleet slot, and the shed / refusal |
246
207
  | `worker-registry.ts` | one worker's registry row: announce, heartbeat, forget |
247
- | `worker-types.ts` | the worker's public contract: `WorkerOptions`, `WorkerStats`, `Worker` |
208
+ | `worker-types.ts` | the worker's contract: `WorkerOptions`, `WorkerStats`, `Worker` |
248
209
  | `drain-wait.ts` | the drain's wait, shared by both roles |
249
- | `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span |
210
+ | `worker-run.ts` | one claimed job: heartbeat, slot renewal, run signal, consumer span, `job.perform` |
211
+ | `job-instrument.ts` | the producer span + `job.enqueue` per handle enqueue; both spans' attributes |
250
212
  | `run-signal.ts` | the signal ONE run is cancelled by |
251
- | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back — and the boot refusal |
213
+ | `worker-fleet-slots.ts` | an in-flight job's fleet slot: take, renew, hand back; the boot refusal |
252
214
  | `worker-process-cap.ts` / `worker-claim-sweep.ts` | `maxInFlight`'s count; the pass's asks, rotated |
253
215
  | `purge.ts` | `purge()` — a factory over `job()`: the retention sweep and its target seam |
254
216
  | `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
255
217
  | `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
256
218
  | `limits.ts` | per-tenant / per-queue / global concurrency + rate |
257
- | `events.ts` | stored event bus for `step.waitForEvent` |
219
+ | `events.ts` / `-cursor.ts` | memory bus; the wait cursor, both buses |
258
220
  | `inspect.ts` | `--json` introspection. `operatorInput`: no registered handle gives `[redacted]` whole; otherwise `redactInput(maskSensitive(handle.input, …))` |
259
221
  | `inspect-operator.ts` | the verbs `x jobs` binds: remove, promote, pause, resume |
260
222
  | `introspection.ts` | `JobIntrospection`, every operator type, and the bounds |
261
223
  | `counters.ts` | bucket arithmetic, `nackOutcome`, the memory counters |
262
224
  | `driver-memory-operator.ts` / `driver-pg-operator.ts` | each driver's operator members |
263
- | `driver-pg-operator-sql.ts` / `driver-pg-settle-sql.ts` | their statements; `SQL_ACK` / `SQL_NACK` with the counter |
264
225
  | `progress.ts` / `settled.ts` / `redact-input.ts` | the progress throttle; the `onSettled` runner; a payload as an operator reads it |
265
226
  | `scheduler-state.ts` | `SchedulerState`, `fire`, the memory watermark |
266
227
  | `scheduler-leader.ts` / `scheduler-occurrences.ts` | `LeaderElection` + `soleLeader`; which occurrences fall in a window |
267
228
  | `idle-backoff.ts` / `enqueue-signal.ts` | the wait both polling loops share; the two wake signals and `wakeIsLive` |
268
229
  | `queue-wake.ts` / `driver-pg-wake-sql.ts` | the `LISTEN` session and its proof; which statement notifies |
269
230
  | `worker-loop.ts` | the claim loop's timer: wake, `again`, the kicks |
270
- | `driver-pg-outbox-sql.ts` | the `x_outbox` statements |
271
231
  | `errors-operator.ts` / `errors-requeue.ts` | the operator verbs' refusals; `requeue`'s |
272
232
  | `errors-tenant.ts` | `JobTenantMismatchError` |
273
233
  | `errors-steps.ts` / `errors-saga.ts` / `step-resolve.ts` | step, saga + `once` refusals; `resolveStep` |
274
- | `worker-queue-demand.ts` | the `queue_demand` gauge: `driver.demand()`, skipped when `stats()` shows no work, retracted to 0 at the drain |
275
- | `driver-pg-demand-sql.ts` | `QUEUE_DEMAND_ROWS` and `SQL_QUEUE_DEMAND`; the scaler fixture's `SQL_SCALER_DEMAND` reads the same predicate |
234
+ | `worker-queue-demand.ts` | the `queue_demand` gauge: `driver.demand()`, skipped on no work, 0 at the drain |
235
+ | `driver-pg-*-sql.ts` | one file per statement family: jobs rows (`JOB_ROW_COLUMNS`), operator, settle (`SQL_ACK`/`SQL_NACK`), outbox, demand (`SQL_QUEUE_DEMAND`) † |
276
236
  | `driver-pg-claim-sql.ts` | `SQL_CLAIM` and why it seeks; re-exported from `driver-pg-sql.ts` |
277
237
  | `error-codes.ts` / `errors.ts` | codes, titles, their registration (`new-error-code` writes here) / the classes |
278
238
  | `job-enqueue-many.ts` / `enqueue-record.ts` / `driver-pg-enqueue-many.ts` | `enqueueMany`, `MAX_ENQUEUE_BATCH`, `ENQUEUE_CHUNK`; one enqueue's row, every path; `SQL_ENQUEUE_MANY` + resolve |
279
239
  | `import.ts` / `import-pass.ts` | `importRows()`: the declaration; one pass, its byte-offset checkpoints, the error report |
280
- | `import-format.ts` / `import-source.ts` / `import-errors.ts` | record boundaries on bytes; the `ImportSource` seam; the `X_IMPORT_*` classes |
240
+ | `import-format.ts` / `-source.ts` / `-errors.ts` | record boundaries on bytes; the `ImportSource` seam; the `X_IMPORT_*` classes |
281
241
  | `driver-redis-refusal.ts` | `typedRefusals`: a send refused for want of a connection → `X_DRIVER_UNAVAILABLE` |
282
242
  | `subject*.ts` | `subjectExport`/`subjectErase`; reach only via DELETED tables (README) |
243
+ | `gdpr-*.ts` | `.personalData()` coverage (thrown by `plannedSubject`, listed by `x exercise`); the erase's audit record |
244
+ | `ledger*.ts` | `ledger()`: balanced per currency, ONE step; a key is one posting per tenant, re-used with other lines `X_LEDGER_POSTING_CONFLICT` |
245
+ | `lock*.ts` | `withLock`: one `lock:<key>` slot; token = the store's `fence`; renewal as `worker-fleet-slots.ts` |
246
+ | `reindex.ts` | `reindex()`: a `backfill()` into the `searchIndex` port via `searchDocumentOf` |
247
+ | `partitions.ts` | `partitions()` + `.schedule`: db's `partitionMaintenance` per partitioned entity, job clock |
283
248
  | `webhook-receiver.ts` / `-route.ts` / `-types.ts` / `-errors.ts` | `webhookReceiver()` and `webhookReceiverRoutes`; the route; `ReceivedEvent`, `WebhookInbound`; its three refusals |
284
249
 
285
- Commands: `bun test packages/jobs` · `bun run --filter @ultimat3/jobs typecheck`. Test harnesses:
250
+ Commands: `bun test ./packages/jobs/` · `bun run --filter @ultimat3/jobs typecheck`. Test harnesses:
286
251
  history, "Moved 2026-10-01 — test harnesses".
287
252
 
288
- Why each rule above is shaped the way it is: [`docs/history/jobs.md`](../../docs/history/jobs.md).
253
+ Why: [`docs/history/jobs.md`](../../docs/history/jobs.md).