@ultimat3/jobs 29.4.0 → 30.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (109) hide show
  1. package/CLAUDE.md +68 -89
  2. package/README.md +304 -10
  3. package/package.json +18 -6
  4. package/src/backfill-definition.ts +184 -0
  5. package/src/backfill-errors.ts +18 -2
  6. package/src/backfill-pass.ts +43 -40
  7. package/src/backfill-pending.ts +40 -17
  8. package/src/backfill-reader.ts +94 -0
  9. package/src/backfill-registry.ts +5 -0
  10. package/src/backfill-scope.ts +1 -2
  11. package/src/backfill-source.ts +113 -0
  12. package/src/backfill.ts +54 -135
  13. package/src/consumer-errors.ts +42 -0
  14. package/src/consumer-guard.ts +19 -0
  15. package/src/consumer-loop.ts +171 -0
  16. package/src/consumer-registry.ts +103 -0
  17. package/src/consumer-types.ts +81 -0
  18. package/src/consumer.ts +147 -0
  19. package/src/driver-memory-operator.ts +25 -2
  20. package/src/driver-memory.ts +35 -1
  21. package/src/driver-pg-claim-sql.ts +128 -0
  22. package/src/driver-pg-ddl.ts +11 -2
  23. package/src/driver-pg-demand-sql.ts +27 -0
  24. package/src/driver-pg-enqueue-many.ts +153 -0
  25. package/src/driver-pg-event-wake-sql.ts +51 -0
  26. package/src/driver-pg-operator-sql.ts +34 -6
  27. package/src/driver-pg-operator.ts +36 -2
  28. package/src/driver-pg-outbox-sql.ts +34 -0
  29. package/src/driver-pg-sql.ts +24 -91
  30. package/src/driver-pg.ts +17 -1
  31. package/src/driver-redis-refusal.ts +25 -0
  32. package/src/driver-redis-scripts.ts +79 -30
  33. package/src/driver-redis.ts +70 -42
  34. package/src/driver.ts +38 -1
  35. package/src/enqueue-record.ts +137 -0
  36. package/src/error-codes.ts +233 -0
  37. package/src/errors-concurrency.ts +2 -2
  38. package/src/errors-declaration.ts +15 -2
  39. package/src/errors-operator.ts +1 -1
  40. package/src/errors-requeue.ts +1 -1
  41. package/src/errors-saga.ts +87 -0
  42. package/src/errors-steps.ts +35 -0
  43. package/src/errors-tenant.ts +1 -1
  44. package/src/errors.ts +25 -165
  45. package/src/events-pg.ts +20 -8
  46. package/src/events.ts +14 -3
  47. package/src/execute-ctx.ts +51 -0
  48. package/src/execute-unwind.ts +191 -0
  49. package/src/execute.ts +50 -51
  50. package/src/export-errors.ts +1 -1
  51. package/src/failure-row.ts +43 -0
  52. package/src/import-errors.ts +102 -0
  53. package/src/import-format.ts +232 -0
  54. package/src/import-pass.ts +255 -0
  55. package/src/import-source.ts +49 -0
  56. package/src/import.ts +175 -0
  57. package/src/index.ts +163 -7
  58. package/src/inspect-operator.ts +24 -3
  59. package/src/inspect.ts +17 -4
  60. package/src/introspection.ts +24 -1
  61. package/src/job-enqueue-many.ts +108 -0
  62. package/src/job-types.ts +229 -0
  63. package/src/job.ts +19 -208
  64. package/src/metrics.ts +10 -0
  65. package/src/outbox-pg.ts +35 -0
  66. package/src/outbox-relay.ts +58 -19
  67. package/src/outbox.ts +65 -73
  68. package/src/purge-entities.ts +92 -0
  69. package/src/purge.ts +10 -2
  70. package/src/queue-wake.ts +1 -1
  71. package/src/refresh-view.ts +76 -0
  72. package/src/retry-classification.ts +27 -8
  73. package/src/retry.ts +26 -0
  74. package/src/rollup-errors.ts +27 -0
  75. package/src/rollup.ts +190 -0
  76. package/src/run-deadline.ts +11 -7
  77. package/src/scheduler.ts +2 -1
  78. package/src/settled.ts +56 -15
  79. package/src/step-resolve.ts +37 -0
  80. package/src/steps-api.ts +111 -0
  81. package/src/steps-core.ts +26 -0
  82. package/src/steps-output.ts +142 -0
  83. package/src/steps-store.ts +68 -0
  84. package/src/steps-suspension.ts +51 -0
  85. package/src/steps-timeout.ts +10 -5
  86. package/src/steps-unwind.ts +94 -0
  87. package/src/steps-wait.ts +163 -0
  88. package/src/steps.ts +174 -210
  89. package/src/subject-erase-pass.ts +134 -0
  90. package/src/subject-errors.ts +48 -0
  91. package/src/subject-export-pass.ts +228 -0
  92. package/src/subject-jobs.ts +121 -0
  93. package/src/subject-plan.ts +287 -0
  94. package/src/subject-select.ts +106 -0
  95. package/src/subject.ts +119 -0
  96. package/src/tenant.ts +69 -1
  97. package/src/type-pins.ts +90 -0
  98. package/src/webhook-errors.ts +1 -1
  99. package/src/webhook-receiver-errors.ts +59 -0
  100. package/src/webhook-receiver-route.ts +127 -0
  101. package/src/webhook-receiver-types.ts +97 -0
  102. package/src/webhook-receiver.ts +132 -0
  103. package/src/webhook-target.ts +20 -43
  104. package/src/webhook.ts +7 -4
  105. package/src/worker-queue-demand.ts +57 -0
  106. package/src/worker-queue-depth.ts +15 -3
  107. package/src/worker-registry.ts +4 -1
  108. package/src/worker-types.ts +2 -0
  109. package/src/worker.ts +2 -2
package/CLAUDE.md CHANGED
@@ -5,8 +5,11 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
5
5
  ## Boundary
6
6
 
7
7
  - May import: `core`, `schema`, `entity`, `policy`, `cache`, `time` — and `db`, for
8
- `expectedQueryLoop` ONLY: `steps.ts` declares the per-step write one-per-step to the N+1
9
- detector there, and no client is ever taken from it. Never `http`, `render`, `ui`.
8
+ `expectedQueryLoop` (`steps.ts` declares the per-step write one-per-step to the N+1 detector),
9
+ `outsideTransaction` (`execute.ts`), `refuseAbandoned` (`outbox.ts`) and `withTransaction`
10
+ (`consumer-loop.ts`: a batch and its cursor commit together) — the ONE place a client is taken
11
+ from it, the ambient `db()`. Never `render`, `ui`. `http` only from `webhook-receiver*.ts`, for the route
12
+ (`Route`/`Bucket` types, `DEFAULT_WEBHOOK_BODY_LIMIT`, `HttpError`, `jsonResponse`).
10
13
  - Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
11
14
  - External deps: none. Postgres access goes through the injected `PgExecutor` (`@ultimat3/core`'s,
12
15
  re-exported from `index.ts`).
@@ -27,12 +30,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
27
30
  idempotency_key)`** — the index, the conflict target (spelling the index EXPRESSION exactly) and the
28
31
  live-row lookup agree, pinned by a test; `driver-memory.ts` mirrors it. The DDL drops both
29
32
  superseded indexes.
30
- - **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — do not re-litigate.**
31
- `tenant` is REQUIRED (`(input) => input.orgId` or `'none'`; `X_JOB_TENANT_REQUIRED` backstop).
32
- `executeJob` puts the derived org on the ctx's actor (`jobRunActor`, org only) and runs the body in
33
- `runWithContext`; `'none'` STRIPS the org (fail-closed).
34
- `tenantFor` is a METHOD on `JobHandle` (variance). The queue row's `tenantId` stays the ENQUEUER's
35
- (the limiter's bucket). `tenancy-cross-surface.test.ts` asserts HTTP and job verdicts are EQUAL.
33
+ - **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — do not re-litigate.** `tenant` is REQUIRED (`(input) => input.orgId` or `'none'`; `X_JOB_TENANT_REQUIRED` backstop). Full rule: `docs/history/jobs.md`.
36
34
  - **`enqueuedBy` is ATTRIBUTION, never authority** — do not re-litigate. A job acting FOR a user takes
37
35
  the user's id in its input and re-authorises it in the body.
38
36
  - `tz` is NON-OPTIONAL in `TaskDefinition`, validated by `@ultimat3/time`'s `isValidTimeZone`. A task
@@ -44,11 +42,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
44
42
 
45
43
  ## Rules — the driver and SQL
46
44
 
47
- - **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`,
48
- `x_outbox`, `x_scheduler_*`). A NEW table is also a name in
49
- `packages/cli/src/framework-schema.ts`. A shipped table grows by `alter table ... add column if
50
- not exists`. Its comments carry NO apostrophes and NO semicolons (it splits on `;`; `driver-pg-sql.test.ts` checks
51
- quote parity).
45
+ - **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`, `x_outbox`, `x_scheduler_*`). Full rule: `docs/history/jobs.md`.
52
46
  - **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
53
47
  `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's reads are `async` so a refusal rejects on both.
54
48
  - **`ack`, `nack`, `heartbeat` and `recordProgress` are FENCED on `state = 'running'` AND the
@@ -62,29 +56,35 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
62
56
  - **`cancel` is fenced on `LIVE_STATES`.** **`requeue`** refuses (`X_JOB_NOT_FOUND`, not requeueable,
63
57
  duplicate), then ONE statement. **`enqueue` inserts twice at most.** **`BulkResult.remaining` is
64
58
  what a second call would move.**
59
+ - **`enqueueMany` is one statement per `ENQUEUE_CHUNK` (1,000)** (`job-enqueue-many.ts`;
60
+ `SQL_ENQUEUE_MANY`: `SQL_ENQUEUE`'s conflict target and id fence, one wake per queue); skipped rows
61
+ are answered by ONE `SQL_ENQUEUE_MANY_RESOLVE`, a row neither answers goes back through `enqueue`.
62
+ **`JobDriver.enqueueMany` is REQUIRED** and opens with `assertEnqueueMany` (no `onConflict:
63
+ 'error'` in a batch, `X_INVARIANT`); Redis's `REDIS_ENQUEUE_MANY` shares `enqueue_one` with
64
+ `REDIS_ENQUEUE`. `OutboxStore.stageMany`/`markPublishedMany` are OPTIONAL: absent, a loop. Every
65
+ refusal precedes the first write; `refuseAbandoned()` before every direct chunk. `runAt` crosses
66
+ as `numeric` then `bigint`: a fractional ms rounds, never a `22P02`.
65
67
  - **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
66
68
  (discovered, comments stripped) for `select *`/`returning *`.
67
- - Drivers: the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`; new
69
+ - Drivers: the `JobDriver` methods plus optional `introspect`, `backfills`, `leases`; new
68
70
  capabilities go behind the interface. **`redisJobDriver`** (`@ultimat3/jobs/redis`, off the
69
71
  barrel) is one Lua script per operation: a semantic change lands in memory, pg AND the scripts.
70
- - Step results are persisted BEFORE the step returns. All time is epoch ms (`nowMs()`, `clock.ts`).
72
+ Its client is core's `redisClient()`, never `Bun.redis`; `ERR_REDIS_CONNECTION_CLOSED` is
73
+ `X_DRIVER_UNAVAILABLE` at once (`typedRefusals`), never a wait.
74
+ - **`demand` is the seventh method, required**; a draining worker's `queue_demand` goes to 0.
75
+ - **The claim SEEKS** (`SQL_CLAIM`); correct only because a claim autocommits.
76
+ - Step results are persisted BEFORE the step returns, and it returns what it persisted: `jsonForm` (`steps-output.ts`), typed `StepOutput<T>`, one stringify + parse per EXECUTED step. Full rule: `docs/history/jobs.md`.
71
77
  - **`postgresLeader` is correct only on a DEDICATED connection**; boot uses `postgresLeaseLeader`.
72
78
 
73
79
  ## Rules — numbers and limits
74
80
 
75
- - **Every numeric knob is refused when not FINITE** — core's `finiteOption()` (a bound) and
76
- `finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `jobWorker` reads them.
77
- **`bun run finite-bounds` is a floor, never proof**: `concurrencyLimiter`'s five numbers are screened
78
- with min 0 (zero is a HARD STOP). **A row count is `finiteCount` (min 0)**; `retry.attempts` is one with min 1.
81
+ - **Every numeric knob is refused when not FINITE** (`finiteOption()`, `finiteCount()`, read in `worker-options.ts`); `bun run finite-bounds` is a floor. Full rule: `docs/history/jobs.md`.
79
82
  - **`WorkerOptions.concurrency` is read by OWN key** (`worker-slots.test.ts`); `maxInFlight` BEFORE
80
83
  the claim, its count back with the lease, before the `kick` (history).
81
84
  - **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
82
85
  `limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
83
86
  throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
84
- - **`concurrency` is resolved ONCE** (`concurrency.ts`): a number, or `{ key, limit, whenBusy }` per
85
- key. The handle carries `concurrency` (the limit), `whenBusy` (set exactly when keyed) and
86
- `concurrencyKeyFor()` — a METHOD. Lease key `job-key:<encoded name>:<key>`; plain `job:<name>`.
87
- A bad cap is `X_JOB_DECLARATION_INVALID`.
87
+ - **`concurrency` is resolved ONCE** (`concurrency.ts`): a number, or `{ key, limit, whenBusy }` per key. Full rule: `docs/history/jobs.md`.
88
88
  - **`whenBusy: 'fail'` refuses only on EVIDENCE another run holds the key** (`LeaseStore.holders`,
89
89
  required): a run meets its own leftover slot. The refusal is `nack({ fail, countsAsAttempt:
90
90
  false })` → `failed`, outcome `refused` (`worker-key-busy.ts`), never `executeJob`.
@@ -94,8 +94,9 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
94
94
 
95
95
  ## Rules — the worker and the scheduler
96
96
 
97
- - **One enqueue implementation**: `jobsFacade()`; the only other `driver.enqueue` sites are the outbox
98
- relay and the scheduler. `handle.as(actor, input)` QUEUES; `executeJob` is the one execution path.
97
+ - **One enqueue implementation**: `jobsFacade()` owns `enqueue` AND `enqueueMany`, over the row
98
+ builders in `enqueue-record.ts`, which the relay shares; the only other `driver.enqueue` sites are
99
+ the outbox relay and the scheduler. `handle.as(actor, input)` QUEUES; `executeJob` is the one execution path.
99
100
  - The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
100
101
  job's plain key.
101
102
  - **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
@@ -125,11 +126,9 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
125
126
  - **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`) — save a fenced
126
127
  `X_DRAINING` on a live attempt: a step done in the drain is recorded once. The STORE fences
127
128
  the claim (`StepFence`, `SQL_STEP_PUT` as `SQL_ACK`: `X_JOB_LEASE_LOST`). `executeJob` passes it.
129
+ - **A cancelled body's writes refuse too**, `enqueue` included (`outbox-abandon.test.ts`); a store is on the pool: a fenced `put` past `X_DRAINING` lands.
128
130
  - **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
129
- - **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
130
- out before `leader.release()`. Same two-hook rule; an ABANDONED round does not release
131
- (`jobs.scheduler.drain-abandoned`). `stillLeading()` is asked every round and before every task,
132
- and answers from memory inside `renewEveryMs`.
131
+ - **The scheduler runs one dispatch round at a time**; `stop()` waits it out before `leader.release()`. Full rule: `docs/history/jobs.md`.
133
132
  - **`run-once` fires ONE catch-up**: the watermark rides the fire (`ScheduledFire.watermarkMs`),
134
133
  never a `markFired` behind it; `skip` fires the latest missed occurrence (`latestOccurrenceBy`).
135
134
  - **A lease for a run that never STARTED is `abandon()`ed, never `release()`d** — the rate stamp
@@ -147,57 +146,12 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
147
146
  - **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which refuses** a finished job or a
148
147
  driver with no `cancel` (`X_JOB_NOT_CANCELLABLE`).
149
148
 
150
- ## Rules — the operator surface
149
+ ## Rules — long form in `docs/history/jobs.md`
151
150
 
152
- - **Every operator capability is a member of `JobIntrospection`** (`introspection.ts`), implemented
153
- by BOTH drivers, bounds as constants beside it; `operator-surface*-fixture.ts` is their parity
154
- suite (memory + real pg).
155
- - **A settle answers whether it landed**; a miss is `jobs.settle.unowned`, not a throw.
156
- - **The counter moves in the SETTLING statement** (`driver-pg-settle-sql.ts`): one-minute bucket per
157
- job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing).
158
- The scheduler leader folds tiers (`rollupCounters`), at most once a bucket.
159
- - **`list()` is KEYSET** (`after: jobCursor(lastRow)`, newest first by `(created_at, id)`); the pg
160
- seek reads the cursor row's own `created_at` — the cursor's ms is rounded. A page past
161
- `MAX_JOB_PAGE` or a foreign cursor is `X_JOB_PAGE_INVALID`, never a bare invariant.
162
- - **`taskFires()` is the last occurrence that DISPATCHED**, never the watermark (arming and skipping
163
- move that): `SQL_SCHEDULER_FIRE` writes it, `fireThroughDriver` calls `recordTaskFire`.
164
- - **A pause is a ROW the claim reads** (`x_job_pauses`, `not exists` in `SQL_CLAIM`), never a column
165
- on `x_jobs`. A paused TASK keeps its watermark, so resume is its own `catchUp`.
166
- - **An occurrence fires in ONE statement** (`SchedulerState.fire`, `SQL_SCHEDULER_FIRE`): the
167
- watermark is the fence, jobs insert only `where exists (select 1 from moved)`.
168
- - **`progress()` is throttled in `progress.ts`** and flushed before every settle. **`onSettled`
169
- (`settled.ts`) is the ONE ending hook** — `completed`, `dead-lettered`, `dropped`, `refused` —
170
- AFTER a settle that LANDED, in the body's tenant scope, `ON_SETTLED_ATTEMPTS` tries, never
171
- rethrown, at most once across a crash. Never for a cancel, a retry or a suspension.
172
- - **Failed for good + `deadLetter: false` is DROPPED**: `nack({ fail })`, outcome `dropped`.
173
- - **Worker registry rows expire** (`worker-registry.ts`, TTL = visibility timeout); `stop()` forgets.
174
-
175
- ## Rules — idle cost and the wake
176
-
177
- Long form: `docs/history/jobs.md`, "the cross-process wake".
178
-
179
- - **An idle round touches no store.** The scheduler keeps each watermark and its next occurrence in
180
- memory while it leads (`known`), trusts a lease for `LeaderElection.renewEveryMs`, reads the pause
181
- table only when a task is DUE, and folds counters every `COUNTER_ROLLUP_INTERVAL_MS`. What makes
182
- the trust safe is the fire statement's watermark fence, never the lease.
183
- - **The worker and the relay back off** (`idle-backoff.ts`): floor → doubling → the ceiling, 2 s, or
184
- 5 s while `wakeIsLive()`. An idle worker pass is ONE claim over every queue with a free slot.
185
- - **The poll is the guarantee, the wake an optimisation.** `startQueueWake` holds one `LISTEN`
186
- session and raises `enqueue-signal.ts`'s two signals; it is live only once a probe has crossed
187
- BOTH channels, and un-proven on every re-dial — a transaction-pooling proxy resolves a `LISTEN`
188
- and delivers nothing.
189
- - **A statement decides whether it notifies** (`driver-pg-wake-sql.ts`): once per queue per
190
- `WAKE_SLOT_MS` for an enqueue, never behind an unclaimed row for a stage. Every notifying commit
191
- serialises behind ONE Postgres lock: never notify unconditionally. Payload: the queue name.
192
- - **A wake resets the backoff**: the floor passes after it find the row a slot silenced.
193
- - **The claim loop is ONE chain** (`worker-loop.ts`): a wake on a pass in flight sets `again`; a
194
- freed slot and a due retry (`JobExecution.resumeAt`) `kick` one pass, no reset.
195
- - **`idle-cost.test.ts` / `queue-wake.test.ts` count statements per idle minute** — scheduler 6,
196
- worker and relay 30 with no wake, 12 with. Raising one is a regression.
197
- - **The memory driver stores a payload's JSON form**, as pg binds it (`driver-settle-parity.test.ts`).
198
- - **The event bus has ONE clock**: `postgresEventBus` takes no `clock`; `EventLookup.now()` is
199
- REQUIRED, stamps a NEW wait, and calls the TIMEOUT (asked once the runner thinks time is up).
200
- `purgeExpired()` is awaited and counted; `eventsPurgeTarget(bus)` is its `PurgeTarget`.
151
+ - "Moved out 2026-11-03": operator verbs on BOTH drivers; an idle round touches no store.
152
+ - "Moved out 2026-11-06": `demand` with no worker registry; how the claim seeks.
153
+ - "Plan 101 wave 6c": compensate ONLY failed for good; `once` writes `started` BEFORE `fn`; a
154
+ waking publish is ONE statement; payload org = enqueue org (`X_JOB_TENANT_MISMATCH`).
201
155
 
202
156
  ## Rules — the outbox
203
157
 
@@ -211,25 +165,33 @@ Long form: `docs/history/jobs.md`, "Moved 2026-10-01 — the outbox".
211
165
  - **A staged row's id IS its job's id, and its `runId` is allocated at stage** (`x_outbox.run_id`),
212
166
  so a staged enqueue answers real ids. `SQL_ENQUEUE` inserts nothing under an id that names a
213
167
  row — a repeated publish is `deduped`, live or finished (`outbox-run-id.test.ts`).
168
+ - **The relay publishes a batch in two statements** (`publishBatch`): `driver.enqueueMany`, then
169
+ `markPublishedMany` when the store has it; any failure drops that batch to the row loop, which
170
+ keeps the order. `RELAY_BATCH_SIZE` (500) is measured in its doc comment.
214
171
 
215
172
  ## Rules — factories over `job()`
216
173
 
217
174
  Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-01 — factories".
218
175
 
219
- - **`backfill()`**: a step persists the CURSOR and a count, never the page; step names are positional;
220
- **`handle` is AT LEAST ONCE** — never invert; a REPLAYED batch writes no ledger row. `tenant: 'none'`
221
- is the only cross-tenant scope (`backfill-scope.ts`). `x_backfills` is what was SWEPT, keyed by RUN;
222
- only `completed` blocks. The throttle is spent INSIDE the batch's `step.run`.
176
+ - **`backfill()`**: a step persists the CURSOR, never the page; **`handle` is AT LEAST ONCE**. Full rule: `docs/history/jobs.md`.
223
177
  - **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff.**
224
178
  `environments` is checked in `backfillPass()` and `gateBackfill()`; `requires` in `gateBackfill()`
225
179
  only; `count` after the last batch (`X_BACKFILL_STALLED`). `inspectBackfills()` is the ONE projection.
180
+ - **`rollup()`/`refreshView()`**: ONE `upsertAll`; `rollup.job.test.ts`.
226
181
  - **`purge()`** is the one caller of every `purgeExpired()`: one table per `step.run`, one clock
227
182
  reading per pass, duplicate names refused.
228
183
  - **`exportRows()`**: one object per PAGE, named by page index; NO cross-tenant escape.
184
+ - **`importRows()`**: one step per PAGE, checkpointed at the BYTE OFFSET, no carry between steps;
185
+ `pageBytes` > `maxRecordBytes` is refused at declaration. `handle` is AT LEAST ONCE per page. The
186
+ error report never carries the refused value. NO cross-tenant escape.
187
+ - **`webhookReceiver()`**: a plain `Route` read under `maxBytes` (core's `readWithinLimit`, never
188
+ `bodyBytes()`); `verify` runs before any parse and answers `{ eventId, payload }` from verified
189
+ data; the id is bounded (`WEBHOOK_EVENT_ID_PATTERN`, 422); the enqueue is `outbox: false` before
190
+ the ack (503 otherwise); the payload is stored `.sensitive()`, never re-verified.
191
+ `webhookReceiverRoutes()` is what the boots mount.
229
192
  - **`webhook()`**: ONE event to ONE endpoint, no steps; each attempt but a CANCELLED one recorded
230
193
  before the throw; `ledger.isDisabled` gates the socket. **The wire format is core's.** An org
231
194
  tenant reads `orgId` OFF THE INPUT; none, or not the run's org: TERMINAL `X_JOB_TENANT_MISMATCH`.
232
- - A `-fixture.ts` file does not ship; `backfill-pass-fixture.ts` raises a plain `Error` on purpose.
233
195
 
234
196
  ## Files
235
197
 
@@ -246,18 +208,23 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
246
208
  | `backfill-pending.ts` | declared minus completed, per environment: the alarm `--pending` reads |
247
209
  | `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait |
248
210
  | `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP |
249
- | `backfill-errors.ts` | the seven `X_BACKFILL_*` classes; the codes stay declared in `errors.ts` |
211
+ | `backfill-errors.ts` | the eight `X_BACKFILL_*` classes; the codes stay declared in `error-codes.ts` |
212
+ | `backfill-definition.ts` / `backfill-source.ts` / `backfill-reader.ts` | the declaration types; exactly one of `source`/`pages`, the `<name>:<scope>` ledger key; reading either |
213
+ | `consumer.ts` / `consumer-loop.ts` / `consumer-types.ts` | `consumer()` — a job + `.supervisor` task; one run's turns; the shapes |
214
+ | `consumer-registry.ts` / `consumer-guard.ts` / `consumer-errors.ts` | the `origin` stamp + `registeredConsumers`; the self-enqueue refusal; `X_CONSUMER_*` |
250
215
  | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements; an `action-job` projection is `X_ACTION_JOB_UNBRIDGED` |
251
216
  | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
252
- | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
253
- | `steps-timeout.ts` / `steps-suspension.ts` | a step's ceiling; `StepSuspension` and the timed-out marker |
217
+ | `steps.ts` / `-core.ts` / `-wait.ts` / `-unwind.ts` | replay runner; shared core; `waitForEvent`; compensation |
218
+ | `steps-timeout.ts` / `steps-suspension.ts` | a step's ceiling; `StepSuspension`, `UnwindReached`, timeout marker |
219
+ | `steps-api.ts` / `steps-store.ts` / `steps-output.ts` | `StepApi`, `EventLookup`; `StepRecord`, `StepFence`, `StepStore`; what a step answers, `StepOutput` and `jsonForm` |
220
+ | `type-pins.ts` | compile-time pins for `StepOutput` and `waitForEvent` |
254
221
  | `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
255
222
  | `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
256
223
  | `outbox-pg.ts` | `postgresOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
257
224
  | `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
258
225
  | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
259
226
  | `concurrency.ts` | `job.concurrency` as declared, resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
260
- | `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `errors.ts` |
227
+ | `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `error-codes.ts` |
261
228
  | `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
262
229
  | `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
263
230
  | `scheduler-pg.ts` | `postgresSchedulerState` (the durable watermark, the atomic fire) + `postgresLeaseLeader` |
@@ -270,6 +237,7 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
270
237
  | `driver-memory.ts` / `claim-exhausted.ts` | `x dev` / tests; a buried row's log line and `onSettled` |
271
238
  | `retry.ts` | the dead-letter decision, and this package's option names over core's `backoffDelay` — no curve of its own |
272
239
  | `retry-classification.ts` | the other half: what the thrown error says, and the stop reason |
240
+ | `execute-unwind.ts` / `execute-ctx.ts` | a saga failed for good; caller ctx |
273
241
  | `execute.ts` / `run-deadline.ts` | `executeJob` — one claimed job run and settled; the run's deadline: cancel, then fail |
274
242
  | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
275
243
  | `renewal-timer.ts` | the renewal interval and its `stopped()` latch |
@@ -288,7 +256,7 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
288
256
  | `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
289
257
  | `limits.ts` | per-tenant / per-queue / global concurrency + rate |
290
258
  | `events.ts` | stored event bus for `step.waitForEvent` |
291
- | `inspect.ts` | `--json` introspection |
259
+ | `inspect.ts` | `--json` introspection. `operatorInput`: no registered handle gives `[redacted]` whole; otherwise `redactInput(maskSensitive(handle.input, …))` |
292
260
  | `inspect-operator.ts` | the verbs `x jobs` binds: remove, promote, pause, resume |
293
261
  | `introspection.ts` | `JobIntrospection`, every operator type, and the bounds |
294
262
  | `counters.ts` | bucket arithmetic, `nackOutcome`, the memory counters |
@@ -303,6 +271,17 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
303
271
  | `driver-pg-outbox-sql.ts` | the `x_outbox` statements |
304
272
  | `errors-operator.ts` / `errors-requeue.ts` | the operator verbs' refusals; `requeue`'s |
305
273
  | `errors-tenant.ts` | `JobTenantMismatchError` |
274
+ | `errors-steps.ts` / `errors-saga.ts` / `step-resolve.ts` | step, saga + `once` refusals; `resolveStep` |
275
+ | `worker-queue-demand.ts` | the `queue_demand` gauge: `driver.demand()`, skipped when `stats()` shows no work, retracted to 0 at the drain |
276
+ | `driver-pg-demand-sql.ts` | `QUEUE_DEMAND_ROWS` and `SQL_QUEUE_DEMAND`; the scaler fixture's `SQL_SCALER_DEMAND` reads the same predicate |
277
+ | `driver-pg-claim-sql.ts` | `SQL_CLAIM` and why it seeks; re-exported from `driver-pg-sql.ts` |
278
+ | `error-codes.ts` / `errors.ts` | codes, titles, their registration (`new-error-code` writes here) / the classes |
279
+ | `job-enqueue-many.ts` / `enqueue-record.ts` / `driver-pg-enqueue-many.ts` | `enqueueMany`, `MAX_ENQUEUE_BATCH`, `ENQUEUE_CHUNK`; one enqueue's row, every path; `SQL_ENQUEUE_MANY` + resolve |
280
+ | `import.ts` / `import-pass.ts` | `importRows()`: the declaration; one pass, its byte-offset checkpoints, the error report |
281
+ | `import-format.ts` / `import-source.ts` / `import-errors.ts` | record boundaries on bytes; the `ImportSource` seam; the `X_IMPORT_*` classes |
282
+ | `driver-redis-refusal.ts` | `typedRefusals`: a send refused for want of a connection → `X_DRIVER_UNAVAILABLE` |
283
+ | `subject*.ts` | `subjectExport`/`subjectErase`; reach only via DELETED tables (README) |
284
+ | `webhook-receiver.ts` / `-route.ts` / `-types.ts` / `-errors.ts` | `webhookReceiver()` and `webhookReceiverRoutes`; the route; `ReceivedEvent`, `WebhookInbound`; its three refusals |
306
285
 
307
286
  Commands: `bun test packages/jobs` · `bun run --filter @ultimat3/jobs typecheck`. Test harnesses:
308
287
  history, "Moved 2026-10-01 — test harnesses".