@ultimat3/jobs 22.14.0 → 23.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (63) hide show
  1. package/CLAUDE.md +129 -87
  2. package/README.md +329 -19
  3. package/package.json +6 -6
  4. package/src/concurrency.ts +144 -0
  5. package/src/counters.ts +105 -0
  6. package/src/describe.ts +29 -0
  7. package/src/driver-memory-operator.ts +297 -0
  8. package/src/driver-memory.ts +79 -37
  9. package/src/driver-nats.ts +2 -2
  10. package/src/driver-pg-ddl.ts +71 -0
  11. package/src/driver-pg-jobs-sql.ts +71 -8
  12. package/src/driver-pg-operator-sql.ts +303 -0
  13. package/src/driver-pg-operator.ts +300 -0
  14. package/src/driver-pg-outbox-sql.ts +110 -0
  15. package/src/driver-pg-rows.ts +31 -0
  16. package/src/driver-pg-settle-sql.ts +71 -0
  17. package/src/driver-pg-sql.ts +80 -129
  18. package/src/driver-pg-wake-sql.ts +76 -0
  19. package/src/driver-pg.ts +74 -34
  20. package/src/driver-redis.ts +2 -2
  21. package/src/driver.ts +96 -34
  22. package/src/enqueue-signal.ts +50 -0
  23. package/src/errors-concurrency.ts +91 -0
  24. package/src/errors-operator.ts +94 -0
  25. package/src/errors.ts +13 -17
  26. package/src/events-pg.ts +38 -28
  27. package/src/events.ts +25 -0
  28. package/src/execute.ts +160 -81
  29. package/src/export.ts +23 -4
  30. package/src/heartbeat.ts +9 -3
  31. package/src/idle-backoff.ts +74 -0
  32. package/src/index.ts +105 -12
  33. package/src/inspect-operator.ts +49 -0
  34. package/src/inspect.ts +42 -2
  35. package/src/introspection.ts +262 -0
  36. package/src/job.ts +78 -15
  37. package/src/leases.ts +49 -9
  38. package/src/metrics.ts +6 -1
  39. package/src/outbox-pg.ts +3 -0
  40. package/src/outbox-relay.ts +114 -39
  41. package/src/outbox.ts +29 -6
  42. package/src/progress.ts +78 -0
  43. package/src/queue-wake.ts +220 -0
  44. package/src/redact-input.ts +26 -0
  45. package/src/renewal-timer.ts +28 -10
  46. package/src/retry-classification.ts +13 -1
  47. package/src/retry.ts +10 -2
  48. package/src/run-deadline.ts +57 -0
  49. package/src/scheduler-leader.ts +26 -0
  50. package/src/scheduler-occurrences.ts +76 -0
  51. package/src/scheduler-pg.ts +55 -2
  52. package/src/scheduler-state.ts +66 -0
  53. package/src/scheduler.ts +143 -103
  54. package/src/settled.ts +161 -0
  55. package/src/worker-admit.ts +163 -0
  56. package/src/worker-fleet-slots.ts +107 -38
  57. package/src/worker-hand-back.ts +6 -1
  58. package/src/worker-key-busy.ts +79 -0
  59. package/src/worker-loop.ts +145 -0
  60. package/src/worker-registry.ts +103 -0
  61. package/src/worker-run.ts +9 -0
  62. package/src/worker-types.ts +23 -0
  63. package/src/worker.ts +103 -134
package/CLAUDE.md CHANGED
@@ -45,14 +45,17 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
45
45
 
46
46
  - **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_steps`,
47
47
  `x_backfills`, `x_outbox`, `x_scheduler_state`, `x_scheduler_leader`, `x_job_leases`,
48
- `x_job_events`). A shipped table grows by `alter table ... add column if not exists`. Its comments
49
- carry NO apostrophes and NO semicolons (`dev-queue.ts` splits on `;`; `driver-pg-sql.test.ts` checks
48
+ `x_job_events`, `x_job_pauses`, `x_job_workers`, `x_job_counters`). A NEW table is also a name in
49
+ `packages/cli/src/framework-schema.ts`. A shipped table grows by `alter table ... add column if
50
+ not exists`. Its comments carry NO apostrophes and NO semicolons (`dev-queue.ts` splits on `;`; `driver-pg-sql.test.ts` checks
50
51
  quote parity).
51
52
  - **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
52
53
  `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's `claim`/`list`/`deadLetters` are `async` so a
53
54
  refusal rejects on both.
54
- - **`ack` and `nack` are FENCED on `state = 'running'`** — that is what makes cancellation possible.
55
- `heartbeat` answers a boolean, read as `held === false` (never `!held`).
55
+ - **`ack`, `nack`, `heartbeat` and `recordProgress` are FENCED on `state = 'running'` AND the
56
+ CLAIM** — `{ workerId, claim }`, `claimOf(claimed)`; `x_jobs.claims` is moved by the claim and
57
+ never reset. A worker id alone let a worker's own earlier body settle its re-claim. `heartbeat`
58
+ answers a boolean, read as `held === false` (never `!held`).
56
59
  - **A driver's semantics are pinned in ONE test beside the pg statement** (`driver-parity.test.ts`):
57
60
  list order, the `attempt` floor, `SQL_LEASE_RENEW` fencing on `expires_at > now()` and `holder`,
58
61
  `SQL_STATS` one bucket per row.
@@ -78,13 +81,16 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
78
81
  - **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
79
82
  `limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
80
83
  throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
81
- - **`limits.ts`'s per-tenant state is BOUNDED**: a zero counter is deleted; `starts`/`refusals` swept
82
- and capped at `DEFAULT_MAX_LIMIT_TENANTS`, evicting the LEAST throttled first. `LimitSnapshot.tracked`
83
- publishes both sizes.
84
- - **`clock.ts`'s conversion is `finiteDurationMs(duration, subject, option)`**: floor `finiteOption`
85
- (negative and zero are shipped behaviour), `subject`/`option` REQUIRED, and `Finite` in the name for
86
- the ratchet. `duration-bounds.test.ts`. `parseDuration` and `nextCronOccurrence` are normalised in
87
- one place each (`finiteDurationMs`, `defaultCronResolver`).
84
+ - **`concurrency` is resolved ONCE** (`concurrency.ts`): a number, or `{ key, limit, whenBusy }` per
85
+ key. The handle carries `concurrency` (the limit), `whenBusy` (set exactly when keyed) and
86
+ `concurrencyKeyFor()` — a METHOD. Lease key `job-key:<encoded name>:<key>`; plain `job:<name>`.
87
+ A bad cap is `X_JOB_DECLARATION_INVALID`.
88
+ - **`whenBusy: 'fail'` refuses only on EVIDENCE another run holds the key** (`LeaseStore.holders`,
89
+ required): a run meets its own leftover slot. The refusal is `nack({ fail, countsAsAttempt:
90
+ false })` → `failed`, outcome `refused` (`worker-key-busy.ts`), never `executeJob`.
91
+ - **A key underivable at claim is `SlotGrant` `undecidable`** → `executeJob({ refusal })`: the
92
+ ATTEMPT fails, never the round.
93
+ - **`finalAttempt` is `isFinalAttempt(handle.retry, attempt)`**, REQUIRED on `JobRunArgs`.
88
94
 
89
95
  ## Rules — the worker and the scheduler
90
96
 
@@ -112,17 +118,10 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
112
118
  whole window without one landing is `jobs.lease.lost` + `recordLeaseLost(queue)`, measured on this
113
119
  process's clock, asked both sides of the call.
114
120
  - **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
115
- interval is `unref`ed.
121
+ interval is `unref`ed; every renewal is armed through `WorkerOptions.schedule`.
116
122
  - **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
117
- - **The retry decision reads the ERROR** (`retry-classification.ts` composes around `nextRetry`;
118
- `nextRetryForError` is `execute.ts`'s only caller). **`classifyThrown` never reads `error.retry`
119
- alone** — core's `declaredErrorRetry(code)`. `retry-after` reuses the nack delay
120
- (`meta.retryAfterSeconds`, clamped). The verdict is published (`stop`, `stopReason`,
121
- `recordedFailure`).
122
- - **The backoff arithmetic is core's** (`backoffDelay` via `backoffDelayMs`), `jitter: true` = EQUAL.
123
- `RetryPolicy`, `DEFAULT_RETRY`, `retrySchedule()` unchanged; `BackoffStrategy` aliases core's
124
- `BackoffCurve`. `retry-core-parity.test.ts`. **`classifyThrown` / `statedDelayMs` are core's,
125
- re-exported** (pinned by identity).
123
+ - **The retry decision reads the ERROR, on core's backoff curve** (`retry-classification.ts`,
124
+ `backoffDelayMs`): `docs/history/jobs.md`, "Moved 2026-10-01 — retry and bounds".
126
125
  - **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
127
126
  - **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
128
127
  the attempt is cancelled in `executeJob`'s `finally`.
@@ -131,8 +130,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
131
130
  - **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
132
131
  - **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
133
132
  out before `leader.release()`. Same two-hook rule; an ABANDONED round does not release
134
- (`jobs.scheduler.drain-abandoned`). **It asks `leader.acquire()` every round AND before every task**
135
- (`stillLeading()`).
133
+ (`jobs.scheduler.drain-abandoned`). `stillLeading()` is asked every round and before every task,
134
+ and answers from memory inside `renewEveryMs`.
136
135
  - **`run-once` fires ONE catch-up** (marks `at`); `skip` fires the true latest missed occurrence
137
136
  (`latestOccurrenceBy`, bisection).
138
137
  - **Every timer body catches before it finalises** (`void work().catch(log).finally(...)`).
@@ -147,63 +146,90 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
147
146
  - **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which refuses** a finished job or a
148
147
  driver with no `cancel` (`X_JOB_NOT_CANCELLABLE`).
149
148
 
149
+ ## Rules — the operator surface
150
+
151
+ - **Every operator capability is a member of `JobIntrospection`** (`introspection.ts`), implemented
152
+ by BOTH drivers, bounds as constants beside it; `operator-surface*-fixture.ts` is their parity
153
+ suite (memory + real pg).
154
+ - **A settle answers whether it landed**; a miss is `jobs.settle.unowned`, not a throw.
155
+ - **The counter moves in the SETTLING statement** (`driver-pg-settle-sql.ts`): one-minute bucket per
156
+ job name; `nackOutcome()` decides what a nack adds (a shed, a suspension, a drain: nothing), and
157
+ `ack({ counted: false })` — `x jobs drain` — adds nothing. The scheduler leader folds tiers
158
+ (`rollupCounters`), at most once a bucket.
159
+ - **`list()` is KEYSET** (`after: jobCursor(lastRow)`, newest first by `(created_at, id)`); the pg
160
+ seek reads the cursor row's own `created_at` — the cursor's ms is rounded. A page past
161
+ `MAX_JOB_PAGE` or a foreign cursor is `X_JOB_PAGE_INVALID`, never a bare invariant.
162
+ - **`taskFires()` is the last occurrence that DISPATCHED**, never the watermark (arming and skipping
163
+ move that): `SQL_SCHEDULER_FIRE` writes it, `fireThroughDriver` calls `recordTaskFire`.
164
+ - **A pause is a ROW the claim reads** (`x_job_pauses`, `not exists` in `SQL_CLAIM`), never a column
165
+ on `x_jobs`. A paused TASK keeps its watermark, so resume is its own `catchUp`.
166
+ - **An occurrence fires in ONE statement** (`SchedulerState.fire`, `SQL_SCHEDULER_FIRE`): the
167
+ watermark is the fence, jobs insert only `where exists (select 1 from moved)`.
168
+ - **`progress()` is throttled in `progress.ts`** and flushed before every settle. **`onSettled`
169
+ (`settled.ts`) is the ONE ending hook** — `completed` (with `result`), `dead-lettered`,
170
+ `dropped`, `refused` — AFTER a settle that LANDED, in the body's tenant scope,
171
+ `ON_SETTLED_ATTEMPTS` tries, never rethrown (`X_JOB_ON_SETTLED_FAILED`), at most once across a
172
+ crash. Never for a cancel, a retry or a suspension; no second hook beside it.
173
+ - **Failed for good + `deadLetter: false` is DROPPED**: `nack({ fail })`, outcome `dropped`.
174
+ - **Worker registry rows expire** (`worker-registry.ts`, TTL = visibility timeout); `stop()` forgets.
175
+
176
+ ## Rules — idle cost and the wake
177
+
178
+ Long form: `docs/history/jobs.md`, "the cross-process wake".
179
+
180
+ - **An idle round touches no store.** The scheduler keeps each watermark and its next occurrence in
181
+ memory while it leads (`known`), trusts a lease for `LeaderElection.renewEveryMs`, reads the pause
182
+ table only when a task is DUE, and folds counters every `COUNTER_ROLLUP_INTERVAL_MS`. What makes
183
+ the trust safe is the fire statement's watermark fence, never the lease.
184
+ - **The worker and the relay back off** (`idle-backoff.ts`): floor → doubling → the ceiling, 2 s, or
185
+ 5 s while `wakeIsLive()`. An idle worker pass is ONE claim over every queue with a free slot.
186
+ - **The poll is the guarantee, the wake an optimisation.** `startQueueWake` holds one `LISTEN`
187
+ session and raises `enqueue-signal.ts`'s two signals; it is live only once a probe has crossed
188
+ BOTH channels, and un-proven on every re-dial — a transaction-pooling proxy resolves a `LISTEN`
189
+ and delivers nothing.
190
+ - **A statement decides whether it notifies** (`driver-pg-wake-sql.ts`): once per queue per
191
+ `WAKE_SLOT_MS` for an enqueue, never behind an unclaimed row for a stage. Every notifying commit
192
+ serialises behind ONE Postgres lock: never notify unconditionally. Payload: the queue name.
193
+ - **A wake resets the backoff**: the floor passes after it find the row a slot silenced.
194
+ - **The claim loop is ONE chain** (`worker-loop.ts`): a wake on a pass in flight sets `again`; a
195
+ freed slot and a due retry (`JobExecution.resumeAt`) `kick` one pass, no reset.
196
+ - **`idle-cost.test.ts` / `queue-wake.test.ts` count statements per idle minute** — scheduler 6,
197
+ worker and relay 30 with no wake, 12 with. Raising one is a regression.
198
+ - **The memory driver stores a payload's JSON form**, as pg binds it (`driver-settle-parity.test.ts`).
199
+ - **The event bus has ONE clock**: `createPgEventBus` takes no `clock`; `bus.now()` is "asked at".
200
+
150
201
  ## Rules — the outbox
151
202
 
152
- - **The relay drains in two phases** (`accept` clears the interval; `close` awaits the pass under
153
- `settleAllBy`), its timer `unref`ed. **`relay.stop()` JOINS the pass in flight.**
154
- - **The claim is a LEASE in one statement** (`claimed_at`/`claimed_by` stamped in the locking CTE;
155
- outer `order by staged_at, id`). `OutboxStore.release` (optional) hands back a failed batch.
156
- `outbox-claim.test.ts` pins both stores.
157
- - **The lease is fenced on every mutation** (`and claimed_by = $n`; `OutboxRecord.claimedBy`);
158
- `markPublished` also requires `published_at is null`. A token-less call is unfenced in memory and
159
- fenced on this relay's id in pg (`outbox-pg.ts` says so). `id` (UUIDv7) is the tiebreak.
160
- - **`claimLeaseMs` is normalised in ONE place** (`outbox-lease.ts`, `resolveClaimLeaseMs`,
161
- `X_INVARIANT` at construction).
203
+ Long form: `docs/history/jobs.md`, "Moved 2026-10-01 — the outbox".
204
+
205
+ - **The relay drains in two phases**; `relay.stop()` JOINS the pass in flight, under a `DrainBudget`
206
+ a later shutdown binds LATE (the worker's shape); its timer is `unref`ed.
207
+ - **The claim is a LEASE in one statement**, fenced on every mutation (`and claimed_by = $n`);
208
+ `claimLeaseMs` is normalised in `outbox-lease.ts`. `outbox-claim.test.ts` pins both stores.
162
209
  - **The memory outbox store DELETES a published row** (`retained()` is the seam).
210
+ - **A staged row's id IS its job's id, and its `runId` is allocated at stage** (`x_outbox.run_id`),
211
+ so a staged enqueue answers real ids. `SQL_ENQUEUE` inserts nothing under an id that names a
212
+ row — a repeated publish is `deduped`, live or finished (`outbox-run-id.test.ts`).
163
213
 
164
214
  ## Rules — factories over `job()`
165
215
 
166
- - **`backfill()`**: a step persists the CURSOR and a count, never the page; step names are positional
167
- (`batch:<index>`), so `handle` gets no `step`; the iteration is rebuilt when `batches.cursor`
168
- disagrees with the checkpoint; a read-back checkpoint is checked. **`handle` is AT LEAST ONCE** (it
169
- runs before its checkpoint) — never invert. **A REPLAYED batch writes no ledger row.**
170
- - **A `backfill()` declaring `tenant: 'none'` gets the cross-tenant scope, and nothing else does**
171
- (`backfill-scope.ts`, `withBackfillScope`): only on the pass's own actor, for the pass's life;
172
- `runWithContext` outside `crossTenant`. `backfill-tenancy.test.ts` drives `executeJob`.
173
- - **`x_backfills` is what was SWEPT; the checkpoints are where a pass resumes**. Keyed by RUN; only
174
- `completed` blocks; `start()` clears `completed_at` on an adopted row. A moved checksum warns and
175
- does not run; `force` rides the input.
176
- - **The throttle is spent INSIDE the batch's `step.run`** (`backfill-rate.ts`, `createPacer` asserts
177
- its own rate).
178
- - **A backfill STAMPS its own handle** (`stampBackfill`, not exported); `registeredBackfills()` derives
179
- from `registeredJobs()`.
180
- - **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff**;
181
- `isPendingBackfillState` is shared with `x db backfill --all`.
182
- - **`environments` is checked in `backfillPass()` (the rail) and `gateBackfill()` (a CLI pre-check);
183
- `requires` in `gateBackfill()` only; `count` in `backfillPass()` after the last batch**
184
- (`X_BACKFILL_STALLED`, its result parsed). `gateBackfill()` RETURNS its refusal.
185
- - **`inspectBackfills()` is the ONE projection of the ledger**, reads no clock, answers `[]` for a
186
- driver with none. The ledger hangs off `driver.backfills`, optional.
187
- - **`purge()`** is the one caller of every `purgeExpired()`: `PurgeTarget` is structural, `targets()` a
188
- thunk, one table per `step.run`, one clock reading for the whole pass, duplicate names refused
189
- (`X_INVARIANT`). `DEFAULT_PURGE_CRON`; `@ultimat3/cli`'s `dev-purge.ts` schedules it.
190
- - **`exportRows()`**: one object per PAGE, named by page index (a rerun rewrites the same bytes); the
191
- interleaving assertion in `export-pass.test.ts` is the memory guard; NO cross-tenant escape; the CSV
192
- formula guard is on strings only.
193
- - **`webhook()` delivers ONE event to ONE endpoint** (key `<name>:<endpointId>:<eventId>`); no steps
194
- (the endpoint carries the secret); the timestamp is SEND time; endpoint headers merge UNDER the
195
- framework's; `redirect: 'manual'`. `WebhookLedger` is a seam; every attempt is recorded before the
196
- throw. **The wire format is core's** (`packages/core/src/webhook-signature.ts`), re-exported.
216
+ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-01 — factories".
217
+
218
+ - **`backfill()`**: a step persists the CURSOR and a count, never the page; step names are positional;
219
+ **`handle` is AT LEAST ONCE** — never invert; a REPLAYED batch writes no ledger row. `tenant: 'none'`
220
+ is the only cross-tenant scope (`backfill-scope.ts`). `x_backfills` is what was SWEPT, keyed by RUN;
221
+ only `completed` blocks. The throttle is spent INSIDE the batch's `step.run`.
222
+ - **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff.**
223
+ `environments` is checked in `backfillPass()` and `gateBackfill()`; `requires` in `gateBackfill()`
224
+ only; `count` after the last batch (`X_BACKFILL_STALLED`). `inspectBackfills()` is the ONE projection.
225
+ - **`purge()`** is the one caller of every `purgeExpired()`: one table per `step.run`, one clock
226
+ reading per pass, duplicate names refused.
227
+ - **`exportRows()`**: one object per PAGE, named by page index; NO cross-tenant escape.
228
+ - **`webhook()`** delivers ONE event to ONE endpoint; no steps; every attempt recorded before the
229
+ throw. **The wire format is core's**, re-exported.
197
230
  - A `-fixture.ts` file does not ship; `backfill-pass-fixture.ts` raises a plain `Error` subclass on
198
231
  purpose (it stands in for app code).
199
232
 
200
- ## Known coupling
201
-
202
- `driver-pg.ts`'s `PgExecutor` is a one-method structural interface (no `@ultimat3/db` dependency).
203
- `packages/cli/src/dev-queue.ts` wraps a real `@ultimat3/db` client, so queue statements pass
204
- `@ultimat3/db`'s statement observer with no `{entity, op}` attribution — future work, see
205
- `packages/db/CLAUDE.md`'s `observe.ts` section.
206
-
207
233
  ## Files
208
234
 
209
235
  | File | Owns |
@@ -219,47 +245,63 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
219
245
  | `backfill-pending.ts` | declared minus completed, per environment: the alarm `--pending` reads |
220
246
  | `backfill-rate.ts` | the `rate` throttle: batches/sec as an interval, and the cancellable wait |
221
247
  | `backfill-inspect.ts` | the ledger projected for `x db backfill`, `x jobs`, `/_x` and MCP |
222
- | `backfill-errors.ts` | the seven `X_BACKFILL_*` classes — split out of `errors.ts`, which was over the 500-line ceiling. The codes themselves stay declared in `errors.ts`: one registry, one place |
223
- | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements. Skips a non-job in silence — a module namespace is full of helpers — EXCEPT an `@ultimat3/action` projection (`kind: 'action-job'`), which is `X_ACTION_JOB_UNBRIDGED` |
248
+ | `backfill-errors.ts` | the seven `X_BACKFILL_*` classes; the codes stay declared in `errors.ts` |
249
+ | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements; an `action-job` projection is `X_ACTION_JOB_UNBRIDGED` |
224
250
  | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
225
251
  | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
226
252
  | `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
227
- | `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks. Split off at `outbox.ts`'s 500-line ceiling |
253
+ | `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
228
254
  | `outbox-pg.ts` | `createPgOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
229
255
  | `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
230
- | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
256
+ | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one (drops a key with its last slot), `jobLeaseKey` |
257
+ | `concurrency.ts` | `job.concurrency` as declared, resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
258
+ | `errors-concurrency.ts` | every concurrency refusal's class — declaration, key, `X_JOB_KEY_BUSY`, the boot refusal. Codes stay declared in `errors.ts` |
259
+ | `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
231
260
  | `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
232
- | `scheduler-pg.ts` | `pgSchedulerState` (the durable watermark) + `createPgLeaseLeader` |
261
+ | `scheduler-pg.ts` | `pgSchedulerState` (the durable watermark, the atomic fire) + `createPgLeaseLeader` |
233
262
  | `events-pg.ts` | `createPgEventBus` — `step.waitForEvent` across processes |
234
263
  | `driver.ts` | `JobDriver` contract + wire records |
235
- | `driver-pg.ts` | default driver, real SQL constants, and `createPgLeader` — the advisory-lock election that is **not** what a scheduler uses; `scheduler-pg.ts` above owns the lease-row one boot wires |
236
- | `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the schema the driver installs, and the ONE install point: every durable table this package owns, `x_outbox` included, is declared in it. Whichever file holds the DDL is the one whose comments may carry no `;` and no `'` |
237
- | `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and the `JOB_ROW_COLUMNS` projection they share. Split off at `driver-pg-sql.ts`'s size ceiling and re-exported from it |
264
+ | `driver-pg.ts` | default driver, and `createPgLeader` — the advisory-lock election a scheduler does NOT use |
265
+ | `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the ONE install point. Its comments carry no `;` and no `'` |
266
+ | `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and `JOB_ROW_COLUMNS`; re-exported from `driver-pg-sql.ts` |
238
267
  | `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
239
268
  | `driver-memory.ts` | `x dev` / tests |
240
269
  | `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
241
270
  | `retry.ts` | the dead-letter decision, and this package's option names over core's `backoffDelay` — no curve of its own |
242
271
  | `retry-classification.ts` | the OTHER half of that decision: what the thrown error says, and the stop reason the row and the log carry |
243
- | `execute.ts` | `executeJob` — one claimed job run and settled, and the run's deadline/cancel |
272
+ | `execute.ts` / `run-deadline.ts` | `executeJob` — one claimed job run and settled; the run's deadline: cancel, then fail |
244
273
  | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
245
274
  | `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
246
275
  | `worker.ts` | `worker` role, claim loop, drain — and the one `AbortController` SIGTERM reaches every held run through |
276
+ | `worker-admit.ts` | may this claimed job start: the limiter, the fleet slot, and the shed / refusal |
277
+ | `worker-registry.ts` | one worker's registry row: announce, heartbeat, forget |
247
278
  | `worker-types.ts` | the worker's public contract: `WorkerOptions`, `WorkerStats`, `Worker` |
248
- | `drain-wait.ts` | the drain's wait, shared by both roles: everything a teardown holds, settled — or abandoned at the budget the `close` hook was handed |
249
- | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
279
+ | `drain-wait.ts` | the drain's wait, shared by both roles: settled, or abandoned at the `close` hook's budget |
280
+ | `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span — handed back in one `finally` |
250
281
  | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |
251
- | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back. The claim loop asks "may I start this one?"; this answers it across the fleet |
282
+ | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back — and the boot refusal |
252
283
  | `purge.ts` | `purge()` — a factory over `job()`: the retention sweep, its structural target seam and the hourly cron a host schedules it on |
253
284
  | `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
254
285
  | `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
255
286
  | `limits.ts` | per-tenant / per-queue / global concurrency + rate |
256
287
  | `events.ts` | stored event bus for `step.waitForEvent` |
257
288
  | `inspect.ts` | `--json` introspection |
289
+ | `inspect-operator.ts` | the verbs `x jobs` binds: remove, promote, pause, resume |
290
+ | `introspection.ts` | `JobIntrospection`, every operator type, and the bounds |
291
+ | `counters.ts` | bucket arithmetic, `nackOutcome`, the memory counters |
292
+ | `driver-memory-operator.ts` / `driver-pg-operator.ts` | each driver's operator members |
293
+ | `driver-pg-operator-sql.ts` / `driver-pg-settle-sql.ts` | their statements; `SQL_ACK` / `SQL_NACK` with the counter |
294
+ | `progress.ts` / `settled.ts` / `redact-input.ts` | the progress throttle; the `onSettled` runner; a payload as an operator reads it |
295
+ | `scheduler-state.ts` | `SchedulerState`, `fire`, the memory watermark |
296
+ | `scheduler-leader.ts` / `scheduler-occurrences.ts` | `LeaderElection` + `soleLeader`; which occurrences fall in a window |
297
+ | `idle-backoff.ts` / `enqueue-signal.ts` | the wait both polling loops share; the two wake signals and `wakeIsLive` |
298
+ | `queue-wake.ts` / `driver-pg-wake-sql.ts` | the `LISTEN` session and its proof; which statement notifies |
299
+ | `worker-loop.ts` | the claim loop's timer: wake, `again`, the kicks |
300
+ | `driver-pg-outbox-sql.ts` | the `x_outbox` statements |
301
+ | `errors-operator.ts` | `X_JOB_NOT_REMOVABLE`, `X_JOB_NOT_PROMOTABLE`, `X_JOB_ON_SETTLED_FAILED` classes |
258
302
 
259
- `backfill-pass-fixture.ts` is the one harness `backfill-pass.test.ts` and
260
- `backfill-pass-ledger.test.ts` share. `*.job.test.ts` is the opt-in `job` step: `replay`,
261
- `idempotency` and `outbox-atomicity` each prove one guarantee through a REAL worker, and
262
- `worker-soak.job.test.ts` kills one worker mid-job and asserts exactly-once completion.
303
+ Which `-fixture.ts` harness each suite shares, and what each `.job.` suite proves:
304
+ `docs/history/jobs.md`, "Moved 2026-10-01 — test harnesses".
263
305
 
264
306
  ## Commands
265
307