@ultimat3/jobs 21.0.0 → 22.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CLAUDE.md CHANGED
@@ -10,918 +10,199 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
10
10
  - Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
11
11
  - External deps: none. Postgres access goes through the injected `PgExecutor`.
12
12
 
13
- ## Rules
14
-
15
- - **The export name IS the job/task name.** `registerJobs(module)` / `registerTasks(module)`
16
- stamp it onto the handle the module exported and re-key the registry, so
17
- `import { sendInvite }` is the object the queue routes to after boot — never a renamed copy.
18
- A definition that supplied its own `name:` keeps it: a job name is a durable queue key that
19
- queued, retrying and dead-lettered rows already carry.
20
- - **`defineApi({ jobs, tasks })` is where a module is handed over.** Nothing else registers.
21
- A job module no call reaches keeps the positional `anonymous-job-<n>` — a name that appears
22
- in no source file, on every queue row and in every dead-letter trace.
23
- - The same handle under the same name is one registration seen twice (`defineApi` and the
24
- framework's module scan both reach the same declaration file), not a collision. Everything
25
- else that puts two things on one durable name is `X_JOB_DUPLICATE`, refused at the earliest
26
- decidable point: two definitions sharing a `name:` collide inside `job()`/`task()`, and a
27
- DIFFERENT handle — or the SAME handle under a second export name — collides at registration.
28
- A handle exported twice matters because the rename is in place: the last alias would silently
29
- move the queue key that queued rows were written to.
13
+ ## Rules — registration and declarations
14
+
15
+ - **The export name IS the job/task name.** `registerJobs(module)` / `registerTasks(module)` stamp
16
+ it onto the exported handle in place. A definition with its own `name:` keeps it (a durable queue
17
+ key). `defineApi({ jobs, tasks })` is where a module is handed over; nothing else registers.
18
+ - The same handle under the same name is one registration; anything else on one durable name is
19
+ `X_JOB_DUPLICATE` at the earliest decidable point (same `name:` inside `job()`/`task()`; a
20
+ different handle, or the same handle under a second export name, at registration).
30
21
  - `registerJob`/`registerJobs`/`registerTask`/`registerTasks`/`nameJobs`/`nameTasks` are NOT in
31
- `src/index.ts`. `defineApi` reaches them through core's registrar table; exporting them would
32
- be a second registration path that bypasses `defineApi`'s own result.
33
- - `registerJobs`/`registerTasks` are announced in core's registrar table at import
34
- (`register.ts`). Never remove the announcement: `defineApi({ jobs })` would then throw
35
- `X_REGISTRAR_MISSING` — `@ultimat3/action` is on this tier and cannot import this package.
36
- - `isJobHandle`/`isTaskHandle` are structural: `kind` plus proof `job()`/`task()` built it.
37
- A look-alike never registers. Deliberately not a registry lookup — the registry is what
38
- registration rewrites.
39
- - `idempotencyKey` is NON-OPTIONAL in `JobDefinition`. Never relax it, never default it.
40
- - **The idempotency namespace is `(name, coalesce(tenant_id, ''), idempotency_key)`, never the key
41
- alone and never name-only** (`As of 2026-08`). Two rounds of the same defect, and both are silent
42
- data loss with no error anywhere.
43
-
44
- It was the key alone: two jobs deriving the same natural key from the same input —
45
- `sendWelcomeEmail` and `provisionWorkspace` both keyed `user:${id}` — shared one namespace, so
46
- the second enqueue hit `on conflict do nothing`, fell through to `SQL_FIND_LIVE_BY_KEY`, found
47
- the FIRST job's row and returned `{ id: <A's>, deduped: true }`. The workspace was never
48
- provisioned and `x jobs ls` showed one healthy job.
49
-
50
- Then it was name-only, while the row already carried `tenant_id` as `$9` of the same insert.
51
- Every natural key the docs suggest is unique only WITHIN a tenant — `` `invoice:${input.invoiceId}` ``,
52
- `` `order:${input.orderNumber}` `` — so tenant B enqueuing while tenant A held that key deduped
53
- into tenant A's row: B's work never ran AND B's caller received A's job id, which is valid on
54
- every id-addressed surface (`cancelJob(driver, jobId)` takes an id with no tenant predicate, so
55
- an app wiring the returned id to a cancel button gave B cancellation of A's job). The sibling
56
- projection in `@ultimat3/action` (`idempotency-key.ts`) had folded the actor's org in all along.
57
- `coalesce`, not the bare column: a null `tenant_id` compares unequal to every other null under a
58
- unique index, so a tenantless queue would lose its dedupe entirely — all tenantless rows share
59
- one namespace instead, which is what they had before tenancy existed.
60
-
61
- Three places have to agree and a test pins them together: the index, the conflict target, and the
62
- live-row lookup. The conflict target must spell the index EXPRESSION exactly —
63
- `(name, (coalesce(tenant_id, '')), idempotency_key)` — or Postgres cannot infer the index at all.
64
- `driver-memory.ts` mirrors it with `(record.tenantId ?? '')`, which is the parity that turned a
65
- gap into a confirmed one rather than catching it. `x_jobs` SHIPPED, so the DDL
66
- `drop index if exists`es BOTH superseded indexes — each is strictly narrower than its successor,
67
- so either left in place would keep enforcing exactly the collision this fixes. The scheduler's
68
- occurrence key already prefixes the task name and is unaffected.
69
- - **`SQL_JOBS_TABLE` is the ONE install point, and every durable table this package owns is in
70
- it** (`As of 2026-08`): `x_jobs`, `x_job_steps`, `x_backfills`, `x_outbox`, `x_scheduler_state`,
71
- `x_scheduler_leader`, `x_job_leases`, `x_job_events`. Four of those were subsystems that shipped
72
- fully built with no table behind them — the outbox, the scheduler's watermark and leader,
73
- `job.concurrency`, and `step.waitForEvent`'s bus — and the outbox's DDL sat in its own exported
74
- constant that no boot code applied, which is exactly how it came to be documented and never
75
- created. A shipped table is extended with `alter table ... add column if not exists`, never by
76
- editing its `create`: `create table if not exists` is a no-op against a database that already
77
- has the table, so a column added only there reaches new installs and nothing else. Comments in
78
- that constant carry NO apostrophes and NO semicolons — `dev-queue.ts` splits it on `;` and
79
- `driver-pg-sql.test.ts` checks quote parity, neither of which can tell prose from a literal.
80
- - **`claim({ queues: [] })` is REFUSED by every driver, `As of 2026-08-24`.** It used to mean two
81
- different things: EVERY queue on `driver-memory.ts` (`wanted.size === 0 ||`) and the `default`
82
- queue on `driver-pg.ts` (`queues.length > 0 ? queues : [DEFAULT_QUEUE]`), with `ClaimOptions.queues`
83
- documenting neither — so the memory driver every test in this repo runs against and the pg driver
84
- production runs against answered one question two ways. Nothing reached it (`createWorker` passes
85
- exactly ONE queue per pass, which is what keeps a slow queue from starving the others), so it could
86
- only ever be found by an embedder, in production. Both meanings are silently wrong in the other's
87
- deployment: claiming every queue is a worker taking work it was never configured for, claiming
88
- `default` is a worker that drains nothing and reads as an idle queue. `assertClaimQueues` in
89
- `driver.ts` is the one refusal (`X_JOB_CLAIM_QUEUES_EMPTY`), called by both drivers, and
90
- `driver-memory.ts`'s `claim` is `async` so an empty list REJECTS on both rather than throwing
91
- synchronously on one — a sync throw out of a method typed `Promise<…>` is itself a divergence.
92
- **Breaking**: `driver-pg.test.ts` was leaning on the pg default and now names its queue.
93
- - **`ack` and `nack` are FENCED on `state = 'running'`, and that fence is what makes cancellation
94
- possible** (`As of 2026-08`). Without it the only way to stop a runaway pass was
95
- `UPDATE x_jobs SET state='dead'`, which the worker's next settle wrote straight over. Same fence
96
- in `driver-memory.ts`. `heartbeat` answers a BOOLEAN for the same reason: no row matched means
97
- this process no longer owns the job, and `heartbeat.ts` turns that into an abort on
98
- `LeaseHeartbeat.signal`, which `worker.ts` folds into the `Ctx` — so `x jobs cancel` reaches a
99
- job that is already running, and `steps.ts`'s `put` fence refuses everything it writes after.
100
- Read as `held === false`, never `!held`: a driver written before the return value existed
101
- resolves `undefined`, and treating that as a loss would cancel every job on every renewal.
102
- - **`job.concurrency` is enforced by `JobDriver.leases`, and `limits.ts` is NOT a fleet gate.**
103
- `limits.ts` is three `Map`s in one heap — `perTenant: 2` on twenty pods is forty concurrent
104
- runs, and `ratePerTenant`'s window resets on every deploy. That is the fast path and it is
105
- correct as such; the docstring that called `global` a "fleet-wide ceiling for this worker
106
- process" was two numbers in one sentence and the code always meant the second. The fleet gate is
107
- one row per HELD SLOT in `x_job_leases`, where the `(lease_key, slot)` primary key is the
108
- serialisation — nothing reads a count and then acts on it. A driver with no lease store makes
109
- `createWorker().start()` THROW `X_JOB_CONCURRENCY_UNENFORCEABLE` when a registered job declares
110
- `concurrency`: a documented guarantee that silently does nothing is the worst of the three
111
- options, and refusing is what axiom 3 asks for.
112
- - **`WorkerOptions.concurrency` is read by OWN key, because a queue NAME is deployment data**
113
- (`worker.ts`, `As of 2026-08-26`). `options.concurrency?.[queue]` answered
114
- `Object.prototype.constructor` for a queue called `constructor`, so
115
- `Math.max(0, <function> - inFlight)` was `NaN`, the `free === 0` guard did not catch it, and the
116
- pass issued `driver.claim({ limit: NaN })`. `bun run proto-index` cannot reach this one — the
117
- table is a **parameter**, not an object literal in the file — so `worker-slots.test.ts` is the
118
- enforcement, over `constructor`, `__proto__` and `toString` at once.
119
- - **Every numeric knob is refused when it is not a FINITE number** — `@ultimat3/core`'s
120
- `finiteOption()` (a bound) and `finiteCount()` (a whole number of things, with the caller's
121
- minimum) are the two refusals, and this package declares none of its own. `worker-options.ts` is
122
- the one place `createWorker` reads them (`As of 2026-08-26`), and `createOutboxRelay` refuses its
123
- own two.
124
- Measured: `visibilityTimeoutMs: NaN` makes `visibleAt` `NaN`, the reclaim scan asks
125
- `visibleAt <= now`, and a job whose worker DIED is never claimable again — at-least-once becomes
126
- never, on a row `x jobs ls` still prints as `running`. `concurrency: NaN` slices `(0, NaN)`, so
127
- the worker claims nothing and reports healthy; `pollIntervalMs: NaN` is `setTimeout(fn, 0)`, so
128
- the claim loop spins on the database. `??` guards only nullish and `Math.max`/`Math.floor`
129
- propagate `NaN`: `Number(process.env.X)` on an unset variable arrives intact. Same refusal
130
- `createLimiter`'s `maxTenants` and `backfill()`'s `batch` already made.
131
-
132
- **`bun run finite-bounds` is a floor, never the answer, and a pin of zero is not proof**
133
- (`As of 2026-08-26`). It matches `x.y ?? CONST`, so it never saw `createLimiter`'s four
134
- ceilings — read as `config.global !== undefined && global >= config.global`, a shape with no
135
- `??` in it — and this package read as clean at **zero** while every one of them was off.
136
- Measured: `createLimiter({ global: Number(process.env.WORKER_GLOBAL_CONCURRENCY) })` with the
137
- variable unset granted **1000 of 1000** acquires where `global: 2` grants 2, and
138
- `snapshot().config` still reported the ceiling to `/_x`. All five numbers (`perTenant`,
139
- `perQueue`, `global`, `ratePerTenant.limit`, `ratePerTenant.windowMs` — the window is half the
140
- same ceiling, since `stamp > at - NaN` empties it on every call) are screened at construction
141
- beside `maxTenants`, `finiteCount` with **min 0**: zero is a HARD STOP here and one this repo's
142
- own suite configures, never "unlimited", which is what omitting the option means.
143
-
144
- **A row count is `finiteCount`, and the reason is driver parity** (`As of 2026-08-26`).
145
- `finiteOption` accepts `-1` and `2.5`, and both diverge: `introspect.list({ limit: -1 })` sliced
146
- every row BUT the newest on `driver-memory.ts` and Postgres answers `ERROR: LIMIT must not be
147
- negative` (probed on pg18), while `2.5` keeps 2 rows here and **3** there with no error on either
148
- side. `list`, `deadLetters`, the backfill ledger's `list` and `assertClaimBounds` (both drivers'
149
- `claim`) take the count screen with **min 0** — `limit: 0` is zero rows on both — and
150
- `claim`'s `visibilityTimeoutMs` takes `finiteOption`, because a lease window is a duration.
151
- The memory driver's `list`/`deadLetters` are `async` for the reason `claim` is: a refusal must
152
- REJECT on both, and a synchronous throw out of a method typed `Promise<…>` is itself the
153
- divergence.
154
- - **`enqueuedBy` is ATTRIBUTION, never authority — decided 2026-08, do not re-litigate.** Both
155
- answers were defensible. Impersonating the enqueuer at claim time gives correct authz and is
156
- rejected because a job that sleeps three days, or dead-letters and is retried next quarter, then
157
- acts as somebody whose role, org membership or employment has changed since. So the row carries
158
- who asked, for audit, and the body is explicitly system-authority — which is how
159
- `docs/idea/02-primitives.md` already frames a job. A job that must act FOR a user takes that
160
- user's id in its INPUT and re-authorises it in the body, where the check is visible in review.
161
- - **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — decided 2026-08, do not
162
- re-litigate.** Two halves of one defect, and neither works alone.
163
-
164
- The defect: `executeJob` built a `Ctx` and handed it to `handle.run({ ctx })` as a PARAMETER —
165
- `runWithContext` appeared nowhere in this package. `@ultimat3/entity`'s tenant guard derives from
166
- `tryUseContext()` (`tenancy.ts:152`), not from the ctx it is given, so inside a job body
167
- `actorTenant` answered `undefined`, `scopedPlan` derived no predicate, `verifyScope` returned
168
- early and `assertRowTenant` could not fire. Proven, same entity, same actor, same call:
169
- `HTTP surface, write naming another org -> X_TENANCY_ACTOR_MISMATCH` /
170
- `JOB surface -> ACCEPTED`. Reachable by any app routing user input into a job — which is what the
171
- bullet above tells an author to do — and by every `backfill()` sweep. Both halves are pinned by
172
- `tenancy-cross-surface.test.ts`, which asserts the two surfaces' verdicts are EQUAL rather than
173
- asserting each one separately: that equality is the only shape of test that could have caught it.
174
-
175
- **`tenant` is REQUIRED on `JobDefinition`**, as `(input) => input.orgId` or the explicit
176
- `tenant: 'none'`. The type is the first rail and `X_JOB_TENANT_REQUIRED` (`assertJobTenant`) is
177
- the runtime backstop for generated code and JS callers — the shape `idempotencyKey` already has.
178
- `executeJob` derives the org through `handle.tenantFor(input)`, puts it on the ctx's actor
179
- (`jobRunActor`, which changes the ORG and nothing else — the identity stays whatever
180
- `WorkerOptions.context()` wired, so this grants no authority) and runs the body inside
181
- `runWithContext`. `'none'` STRIPS the org rather than inheriting the worker's, so a tenant-scoped
182
- read inside such a job is `X_TENANCY_ACTOR_ORG_REQUIRED` — fail-closed, intended, and not to be
183
- weakened; a sweep that genuinely spans tenants says so with `crossTenant(reason, fn)`.
184
-
185
- **A boot-supplied service actor was considered and REJECTED.** It closes the same hole and is one
186
- identity for every job in the app — so the first job that takes an org id in its input reads and
187
- writes another tenant's rows with the framework's blessing, which is precisely the cross-tenant
188
- hole this closes. The tenant is a fact about the WORK, so it is declared per job, from the payload
189
- the author already had to pass. A default was rejected for the same reason in both directions:
190
- `'none'` silently reopens the hole, and inheriting the worker's org is the shared identity again.
191
- `tenantFor` is a METHOD on `JobHandle`, never a `readonly tenant: JobTenant<I>` field — a
192
- function-typed property is contravariant in its parameter, so the field would stop
193
- `JobHandle<OrgInput>` being assignable to `AnyJobHandle` and break the registry.
194
- **The queue row's `tenantId` is unchanged**: it comes from the ENQUEUER's actor and is the
195
- limiter's bucket, never re-derived from the payload (`limits.ts`'s header rule). Two different
196
- questions — "whose rate budget does this enqueue spend" and "whose rows may this run touch".
197
- - **A `backfill()` declaring `tenant: 'none'` gets the cross-tenant scope, and NOTHING else does**
198
- (`As of 2026-08`). The one exception to the bullet above, and it is forced rather than chosen.
199
-
200
- `source` hands back a LAZY chain, so every page's plan is built inside the iteration
201
- (`backfill-pass.ts`'s `iterate()`) — after the declaring frame has closed — and
202
- `@ultimat3/entity` applies `scopedPlan` at plan-build time (`plan.ts:111`). An app author holding
203
- a `ReadBuilder` therefore has nothing to wrap in `crossTenant(reason, fn)`: only the pass is
204
- positioned to open it. Without it, `tenant: 'none'` made a sweep over a tenant-scoped entity fail
205
- on page ONE with `X_TENANCY_ACTOR_ORG_REQUIRED` — the docstring told authors to do something the
206
- API cannot express. `backfill-tenancy.test.ts` drives the real surface (`executeJob`, not
207
- `backfillPass` by hand — the shared fixture calls it directly and so runs with no ambient context
208
- at all, which is why no existing backfill test could see this).
209
-
210
- **A backfill that declared a real `tenant` never gets it.** Granting the escape to a tenanted
211
- sweep is granting it to every backfill in the app, which is the opposite of what declaring a
212
- tenant means; the negative is pinned by two tests that fail the moment the guard in
213
- `withBackfillScope` is dropped. The reason string is derived and names the backfill, because it
214
- lands in the audit trail and `X_TENANCY_CROSS_DENIED` renders it — "backfill" alone would read
215
- identically for every sweep in the app.
216
-
217
- **The capability is granted on the PASS's actor, not on the worker's — and that is the decision.**
218
- Minting a worker identity carrying `tenancy:cross` at boot (`packages/cli/src/dev-roles.ts`
219
- builds that context) would hand it to every job that worker claims, including a plain
220
- `job({ tenant: 'none' })` that declared no sweep, and would move the decision into deployment
221
- config where no reviewer sees it — one identity serving every job, which is the shape this whole
222
- change exists to remove. Here it is bounded four ways: only `backfill()`, only on an explicit
223
- `tenant: 'none'`, only for the duration of that pass, and only on a context that dies with it.
224
- **So `dev-roles.ts` needs no change**, and no framework role's actor carries `tenancy:cross`.
225
- `runWithContext` goes OUTSIDE `crossTenant`, never the reverse: the capability is proved against
226
- the ambient actor at the call and again for every plan built inside. Nesting is safe — an app
227
- `handle` opening its own `crossTenant` replaces the reason and re-proves a capability the actor
228
- already holds — so an app that wraps its own body does not fight the pass.
229
- - **`limits.ts`'s per-tenant state is BOUNDED, and a counter at zero is DELETED** (`As of 2026-08`).
230
- `bump(…, -1)` wrote `0` and kept the key, `refusals` cleared only on a matching acquire, and
231
- `starts` kept an array per tenant — four permanent entries per org in a process that never
232
- restarts, which self-service org creation turns into a leak. Zero and absent already answer
233
- identically (`?? 0`), so the counters drop to nothing with the run; `starts` and `refusals` are
234
- swept (a spent window and a refusal past `REFUSAL_TTL_MS` are indistinguishable from missing) and
235
- then capped at `DEFAULT_MAX_LIMIT_TENANTS`. The eviction order is `@ultimat3/http`'s
236
- `memoryRateLimitStore`'s and must stay it: the LEAST throttled window goes first, because
237
- discarding a full one hands that tenant a free rate reset. `LimitSnapshot.tracked` publishes both
238
- sizes so the bound is assertable rather than assumed.
239
- - **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`, and nowhere else.** The relay runs
240
- after commit in its own timer with no request span in scope, so a trace read there would be
241
- nobody's. A `currentSpanContext()` recovered from a `Ctx` has an empty `spanId`, which renders an
242
- all-zero parent every collector rejects — that case carries no header rather than a broken one.
243
- - `tz` is NON-OPTIONAL in `TaskDefinition`, and validated by `@ultimat3/time`'s `isValidTimeZone`
244
- — never a local `Intl` probe, which accepts `'+02:00'`, a "zone" with no DST rules at all. A
245
- non-empty string is not a timezone. A task never contains a handler body.
246
- - **One enqueue implementation.** Everything (`handle.enqueue`, `handle.as`, `task.enqueue`)
247
- goes through `jobsFacade()`; the only other `driver.enqueue(...)` call sites are the outbox
248
- relay and the scheduler's occurrence dispatch. Never add a third.
249
- - `handle.as(actor, input)` QUEUES. A job's execution surface is the queue, so it must never
250
- run the handler inline — `execute.ts`'s `executeJob` is the one execution path.
251
- - The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`) and `task.enqueue()`'s
252
- is the job's plain key. Deliberate: the first stops two schedulers double-firing a tick, the
253
- second is a manual run with no occurrence to scope to.
254
- - **TWO shutdown hooks per worker — one per PHASE — and `stop()` is what hands both back**
255
- (`As of 2026-08-23`). `accept` is `stopAccepting()`: flip the state, clear the poll timer, return.
256
- `close` is the teardown: wait out the rounds and the in-flight jobs, close the driver.
257
-
258
- It was ONE hook, at `accept`, doing all of it — so a single 10-minute job spent the entire drain
259
- deadline inside the phase whose whole purpose is to be over immediately, and every hook behind it
260
- was invoked with **0 ms** left: `@ultimat3/http`'s "stop listening" and `listenSyncNode`'s
261
- `stopAccepting` are both `accept` hooks, so the pod went on serving requests and upgrading
262
- websockets for the whole of the drain the load balancer had already been told about. Reproduced —
263
- 4 hooks started, none finished.
264
-
265
- `start()` keeps both unregisters; the teardown releases them in a `finally`, so a close that threw
266
- still gives them up. Discarding them was a hook per `start()` — the `start()` guard reads a
267
- standstill, so start -> stop -> start stacked a second registration retaining a stopped worker's
268
- driver, and the next process-wide drain ran all of them. `start()` refuses while draining for the
269
- same reason: a claim loop back on a driver the drain is about to close.
270
- - **The outbox relay drains in the same two phases, `As of 2026-09`.** It registered NO hook: on
271
- SIGTERM it went on claiming and publishing through every phase of the drain, and the only thing
272
- that ever stopped it was `RunningRoles.stop()` in `x dev`'s release path — a caller a signal can
273
- skip. Two consequences, both silent: a lease stamped on rows nothing on this pod will run (stranded
274
- for the visibility window), and a row past `driver.enqueue` but short of `markPublished` published
275
- a second time on the next boot, which the idempotency key collapses only while the first job is
276
- still live. `accept` clears the interval and returns; `close` awaits the pass in flight under
277
- `settleAllBy(…, reason.deadlineAt)`; both unregisters come back in the teardown's `finally`. The
278
- poll timer is `unref`ed for `renewal-timer.ts`'s reason — a 200ms interval refed holds the event
279
- loop open past every phase and makes SIGKILL the exit.
280
- - **A claimed job is counted with core's `beginWork()`, so the DRAIN does the waiting**
281
- (`As of 2026-08-23`). The wait for in-flight jobs belongs to the phase between `accept` and
282
- `inflight`, which exists for exactly this and is where `@ultimat3/http` already puts a request —
283
- not to a hook, where one role's work starves every other role's teardown. `/readyz`'s `inflight`
284
- becomes truthful on a worker node as a consequence.
285
- - **The teardown's wait is BOUNDED on the SIGTERM path and unbounded on a manual `stop()`**
286
- (`As of 2026-08-23`). The `close` hook is handed `ShutdownReason.deadlineAt` and passes it to
287
- `settleAllBy` (`drain-wait.ts`); a manual `stop()` passes nothing and waits as long as its
288
- jobs take, because a caller that asked has no budget to spend. Nothing in JS can kill a body that
289
- ignores `ctx.signal`, so the unbounded version was a teardown that never returned: the driver was
290
- never closed, `state` never left `'draining'`, and the memoised `stopping` promise every later
291
- `stop()` joins never settled — `x dev`'s role rollback awaits that promise. Abandoning the wait
292
- costs a lapsed lease and a job the queue delivers again, which is what at-least-once already
293
- promises; `jobs.worker.drain-abandoned` names it, with the `configureLifecycle({ deadlineMs })`
294
- raise as its fix. **A worker always REACHES `'stopped'`**, which is what makes `stop()`'s
295
- `state === 'stopped'` early return an answer rather than a wedge.
296
- - **SIGTERM reaches the job: the `accept` hook aborts every held run's `ctx.signal`**
297
- (`As of 2026-09-07`). The worker holds ONE `AbortController` (`drainSignal`), composed into every
298
- run by `worker-run.ts` as the third source beside the caller's signal and the heartbeat's;
299
- `stopAccepting(reason)` aborts it with a `JobDrainedError` — core's `X_DRAINING`, naming the
300
- worker and the signal — and a manual `stop()` passes no reason and aborts nothing, for the same
301
- line `settleAllBy` draws: a caller that asked wants its work finished. It has to be the `accept`
302
- hook and not the teardown, because core runs `accept`, then waits out in-flight work (every
303
- claimed job is `beginWork()`ed) under the same budget, then `close`: told in `close`, a body
304
- would hear it after the in-flight wait had already spent the whole budget on it. Which is what
305
- happened until this landed — the drain told nobody, so a body reading `ctx.signal` (the one
306
- documented way to stop early) ran to the deadline and was abandoned there, indistinguishable
307
- from one that ignores the signal; ai-maxxing measured it as every Ctrl-C paying the full 25s
308
- and wrote a process-wide signal of its own to get around it. The controller is replaced with a
309
- fresh one in the teardown's `finally`: aborted once stays aborted, and a restarted worker would
310
- otherwise hand every job it claimed a signal born cancelled.
311
- `executeJob` reads the CODE off the run signal's reason (`drainedBy`) and settles a drained
312
- attempt as **`interrupted`** — `nack` with `countsAsAttempt: false`, no park, no dead letter,
313
- `delayMs: 0`, the error still recorded on the row — read off the SIGNAL and not the thrown
314
- error, because what a body stops WITH is not always the reason (a killed ssh child surfaces as
315
- the app's own coded error) and an attempt burned per deploy is the "always twice" draining
316
- exists to prevent, one layer down. Not a `retried`: `attempts: 1` would dead-letter a job the
317
- process, not the job, cut short, and `jobs_total{outcome="failed"}` would spike on every
318
- rollout. The first reason on a controller wins, so a timeout that fired before the drain still
319
- reports as a timeout. `WorkerStats.interrupted` counts them; `jobs.worker.drain-signalled` and
320
- `jobs.attempt.interrupted` are the two lines. `worker-drain-signal.test.ts` holds all of it.
321
- - **One teardown, joined.** `stop()` shares the in-flight teardown promise, so a SIGTERM landing
322
- on a manual stop waits out the same in-flight jobs instead of closing the driver underneath
323
- it. The promise is cleared as it settles, so a worker that started again tears down again
324
- rather than joining one that settled a lifetime ago. A close that threw still stopped the
325
- worker — the failure goes back on the promise the caller awaited, not into a second teardown.
326
- - **The drain waits out the claim round it races, then the jobs.** `tick()` registers its round
327
- in `rounds` in the same synchronous step as its guard, so a round is either refused by a drain
328
- already under way or visible to every drain that starts after it. Guarding on entry alone was
329
- not enough: a round parked in `driver.claim()` adds to `inFlight` AFTER a drain that only
330
- snapshots `inFlight` has decided there is nothing to wait for, and `close()` then lands under a
331
- live job. The round re-reads the state before each queue — "stop claiming" means this round
332
- too — and what it already holds runs to the end.
333
- - **A fleet slot is taken INSIDE a `try`, and a renewal answering `false` CANCELS the run**
334
- (`As of 2026-08`). Two halves of one guarantee, and each was a way for `job.concurrency` to be a
335
- number the framework prints and does not hold.
336
-
337
- `fleetSlots.acquire()` is a WRITE to `x_job_leases`, so a failover, a pool timeout or a `57P01`
338
- rejects it — and it sits between `limiter.tryAcquire` and the `.finally` that releases what that
339
- returned. The in-process slot was burned permanently: four rejections on a concurrency-4 worker
340
- and the role claims nothing again for the life of the process, with `jobs.worker.tick-failed` and
341
- a climbing `queue_depth` as the only symptoms. The `catch` releases the lease and rethrows; the
342
- claimed row goes back to the queue by its visibility timeout, as it does for any round that dies.
343
-
344
- `fleetSlots.release(jobId)` is AWAITED in the settle's `.finally`, never `void`ed: the slot is a
345
- DELETE in `x_job_leases`, so a fire-and-forget one was still on the wire when the teardown's
346
- `allSettled` returned and `driver.close()` took the connection out from under it — the row then
347
- held its slot for a full TTL, and a `concurrency: 1` job was unclaimable by the pod replacing this
348
- one for a whole visibility window after every deploy. `release` swallows its own failures, so
349
- awaiting it cannot turn a finished job into a rejected one.
350
-
351
- `LeaseStore.renew` answering `false` means the row is another holder's — two runs live under a cap
352
- of one — and it was discarded by `.catch(noop)`. It now stops the timer, logs
353
- `jobs.worker.slot-lost` and aborts the run through `X_JOB_SLOT_LOST`. Its own code, not
354
- `X_JOB_LEASE_LOST`: those are different rows on different clocks, and the queue can still consider
355
- this worker the owner of the JOB while another one is running under the same cap. Read as
356
- `renewed !== false` for the reason `heartbeat` reads `held === false` — a store from before the
357
- return value resolves `undefined`, and treating that as a loss would cancel every job on every
358
- renewal. **The heartbeat cannot cover for this**: it renews `x_jobs.visible_at`, a different row,
359
- and knows nothing about `x_job_leases`.
360
- - **The run's signal is a controller this worker owns, never `AbortSignal.any`** (`As of 2026-08`).
361
- `run-signal.ts` composes the caller's `Ctx.signal` and the heartbeat's into one controller, and
362
- `worker-run.ts` disposes it in the same `finally` that stops the timers. `AbortSignal.any` cannot
363
- be undone, which cost twice: an app whose `WorkerOptions.context()` carries a process-lifetime
364
- signal accumulated one composite per job run, and nothing could abort the result — so a lost fleet
365
- slot had no way to reach the body running under it.
366
- - **A lease is HELD, not owned, and losing one is said out loud.** `heartbeat.ts` renews the
367
- window `claim()` bought and decides between two facts: one failed renewal is not a lost lease
368
- (`jobs.heartbeat.failed`, warn — the window has room for the next), a window that passes with
369
- nothing landing is (`jobs.lease.lost`, error, plus `recordLeaseLost(queue)`). `.catch(() =>
370
- undefined)` made both look like a healthy run while the queue re-delivered the job. The window is
371
- measured on THIS process's clock from the last renewal that LANDED, never against
372
- `claimed.visibleAt` — that is the driver's clock, and comparing the two makes every lease
373
- decision a function of skew. Expiry is asked BOTH SIDES of the call: before, because a heartbeat
374
- hung on a dead connection never rejects, and after, because a renewal that SUCCEEDS past its own
375
- window would otherwise restart the clock on a lease the queue already re-delivered — the loss
376
- hidden inside the call meant to prevent it. A renewal that lands after the loss does not revive
377
- it either, because that window is somebody else's now.
378
- - **Settlement is not part of the retry decision.** `executeJob`'s `driver.ack` sits AFTER the
379
- `try/catch/finally`, so only the body's own rejection can reach the retry branch. An `ack` that
380
- fails — a pool timeout, a reset on that one statement — is a job that finished and could not say
381
- so, not a job that failed: nacking it re-runs completed work and records `retried` in
382
- `jobs_total{outcome}` for a failure that never happened. It propagates instead, and the worker's
383
- `jobs.worker.settle-failed` plus the lapsing lease are the honest answer.
384
- - **The retry decision reads the ERROR as well as the attempt count — added 2026-08.**
385
- `executeJob` decided a retry from `nextRetry(handle.retry, attempt)` alone, so every `terminal`
386
- classification in the framework was decorative on the job path: an `X_SCRAPE_AUTH_FAILED` from a
387
- rotated password burned the whole policy, which at a site that locks an account after three wrong
388
- passwords makes the framework's retry the thing that destroys the account.
389
- `retry-classification.ts` composes AROUND `nextRetry` — the backoff arithmetic stays in one
390
- place — and `nextRetryForError` is the only caller `execute.ts` has.
391
-
392
- **`classifyThrown` must never read `error.retry` on its own.** That field is
393
- `init.retry ?? retryFor(code)` and `retryFor` FAILS CLOSED, so every unclassified `UltimateError`
394
- already carries `terminal`; reading it would dead-letter the first attempt of every job in every
395
- app whose codes nobody has classified. Hence core's `declaredErrorRetry(code)`, which answers
396
- `undefined` where `retryFor` answers the default — and hence the one case that is knowingly
397
- under-read: an instance `retry: 'terminal'` on an UNREGISTERED code is indistinguishable from the
398
- default and is treated as unclassified. Register the code; that is the one way.
399
- `retry-after` reuses the delay the nack already takes (`meta.retryAfterSeconds`, clamped by the
400
- policy's `maxDelay`) rather than a second suspension mechanism — `StepSuspension` stays the only
401
- way to park a run, and unlike a suspension a retry-after DOES burn an attempt, because the work
402
- failed. The ceiling outranks every classification but `terminal`.
403
-
404
- **The verdict is published, not inferred**: `jobs.attempt.failed` and `reportError` carry
405
- `stop`, `JobExecution` carries `stopReason`, and `recordedFailure` appends the terminal verdict to
406
- the nack's `error` — `lastError` is the ONE failure field a row has, so without it `x jobs show`
407
- renders a dead letter at attempt 1 of 5 as a silent early stop.
408
- - **The backoff arithmetic is `@ultimat3/core`'s, and `retry.ts` is the option names over it**
409
- (`As of 2026-08-23`). `backoffDelayMs` is `backoffDelay({ curve, jitter, base, max, attempt })`
410
- with this package's spellings applied on the way in — `DurationInput` through `finiteDurationMs`, the
411
- `DEFAULT_RETRY` fallbacks, and `jitter: boolean` mapped to `'equal' | 'none'`. **EQUAL, never
412
- `full`**: `jitter: true` here has meant half-fixed-half-random since it shipped, and `full`
413
- would hand a job that already failed twice a near-zero wait. The public `RetryPolicy`,
414
- `DEFAULT_RETRY` and `retrySchedule()` are unchanged — a declared `retry: { attempts: 5 }` is
415
- durable API — and `BackoffStrategy` is now an ALIAS of core's `BackoffCurve` rather than a second
416
- spelling of the same three names. `retry-core-parity.test.ts` is the pin: 13,824 comparisons
417
- across every curve, base, cap, attempt and roll, plus the 1-based attempt and the clamp-before-
418
- jitter. Never re-derive a delay here — four packages shipped four curves, which is why core has
419
- one — `bun run flight-copies` is the guard, and it refuses a second curve-and-jitter function
420
- anywhere in `packages/*/src`, matched on the literal shape rather than the name.
421
- - **`classifyThrown` / `statedDelayMs` are core's, RE-EXPORTED, not copied** (`As of 2026-08-23`).
422
- They moved down to `packages/core/src/error-retry.ts` beside the table they read.
423
- `retry-classification.test.ts` pins them by IDENTITY (`toBe`), not by agreement: two functions
424
- that answer alike today are two that can drift, and the rule that must never drift is the one
425
- above — an unregistered code carrying `terminal` reads as UNCLASSIFIED.
426
- - **The claim loop re-arms on the PASS, never on the jobs.** A slot belongs to its own job and is
427
- free the moment it settles, so `claimRound` starts what it claimed and returns the promises —
428
- ending the pass on `Promise.allSettled([...inFlight])` made the pool as slow as its slowest
429
- member and left every OTHER queue unasked behind it. `tick()` still resolves with the executions
430
- THAT pass started (never the whole pool), which is what tests drive. Nothing awaits a job promise
431
- in the timer path, so the loop observes each one itself: unobserved is an unhandled rejection,
432
- and `jobs.worker.settle-failed` is where a job that could not be settled with the driver becomes
433
- visible.
434
- - **`state` is set in the teardown's `finally`, never after the close.** A `driver.close()` that
435
- threw pinned it at `'draining'`: `stats()` reported a drain that had finished and `start()`,
436
- which leaves only `'idle'` or `'stopped'`, refused that worker for the life of the process.
437
- - **A deadline CANCELS, then fails the attempt — never the other way round.** `executeJob` owns
438
- an `AbortController` per attempt and hands its signal to the body as `ctx.signal` (composed
439
- with the caller's, never replacing it) and to the step runner. `raceTimeout` aborts before it
440
- rejects because the caller nacks on that rejection and the queue hands the job straight to
441
- another worker: rejecting first is one job running twice. Same order in `withStepTimeout` for
442
- `stepTimeoutMs`. The attempt is also cancelled in `executeJob`'s `finally`, so a step a handler
443
- left in flight cannot write into the run its successor owns.
444
- - **A cancelled runner writes NOTHING, and `put()` in `steps.ts` is the one place that is
445
- enforced.** Every `store.put` goes through it; aborted means `X_ABORTED` instead of the write,
446
- and refusing the write is what unwinds a body that ignores the signal. The failure branch of
447
- `step.run` checks the same flag directly rather than writing through `put`: the caller has to
448
- see the original error, never one raised by the bookkeeping. Nothing in JS can kill an
449
- uncooperative body — `jobs.timeout.abandoned` (warn) names it instead.
450
- - `Ctx.signal` is non-optional in the type, and `executeJob` still reads it defensively:
451
- `@ultimat3/http`'s `asCtx` casts a request context across the seam without one, and a job that
452
- crashed on a missing field is worse than a job with no caller to follow.
453
- - **One dispatch round at a time, and `stop()` is what waits for it.** The scheduler's timer
454
- re-arms on the round it just finished (never a fixed `setInterval` period) and every other
455
- caller of `tick()` JOINS the round in flight rather than opening a second one. Two rounds read
456
- the same `lastFiredAt`, walk the same occurrences and dispatch them both: the occurrence key
457
- deduped the jobs, so all that showed was a re-marked watermark, occurrences reported as
458
- dispatched that were never enqueued, and a `run-all` catch-up interleaved with itself. `stop()`
459
- waits that round out BEFORE `leader.release()` — a lock handed back mid-dispatch promotes a
460
- standby onto an occurrence this node is still enqueueing for, which is the double-fire leader
461
- election exists to prevent — and the round re-reads the drain state before each task, so "stop
462
- dispatching" means this round too. Same hook rules as the worker, `As of 2026-08-23`: **TWO**
463
- `onShutdown` registrations — `accept` stops dispatching and returns, `close` waits the round out
464
- and releases the lease under the deadline it is handed (`settleAllBy`) — both handed back in the
465
- teardown's `finally` so a `release()` that threw still gives them up, `isLeader` cleared there
466
- too because a lock this process could not hand back is never treated as still held. **An
467
- ABANDONED round does not release**: it is still enqueueing, so handing the lock over is that same
468
- double-fire delivered by the shutdown, and a lease row expiring on its own is the safe end —
469
- `jobs.scheduler.drain-abandoned` is the line that says so.
470
- - **`exportRows()` is a FACTORY over `job()`, and the part KEY is what makes at-least-once safe**
471
- (`As of 2026-08-24`). Same rule `backfill()`, `purge()` and `webhook()` follow. The paging is
472
- `backfill-pass.ts`'s and never a second idiom — `inBatches()`, one page per `step.run`, and what
473
- a step persists is the CURSOR and two counters, never the page.
474
-
475
- **One object per PAGE, not one per export**, and that is the whole design rather than a
476
- simplification: `StorageDriver.put()` buffers by construction — `packages/storage/src/driver.ts`
477
- says the server-side path "is for objects that FIT IN MEMORY" — so a single-object export holds
478
- the entire dataset, which is the failure this factory exists to prevent. Because a part is named
479
- by its page INDEX, a page that runs twice rewrites the same object with the same bytes: there is
480
- no idempotency argument to make about the app's rows, and no append anywhere, because a duplicate
481
- part cannot be expressed. `export-pass.test.ts` proves it by deleting a mid-run checkpoint and
482
- re-running, which is exactly the window at-least-once opens.
483
-
484
- **The interleaving assertion is the memory guard, not `maxPartBytes`.** `maxPartBytes` bounds one
485
- page; it cannot see a rewrite that accumulates every page and writes at the end. The test records
486
- how many checkpoints had LANDED at each `put` — `0,1,2,…` for a streaming pass and a flat run of
487
- the final count for a buffering one — and it is the only thing in the suite that fails on that
488
- rewrite.
489
-
490
- **An export gets NO cross-tenant escape, and `backfill()` does.** The asymmetry is deliberate and
491
- is the direction of the data: a sweep that rewrites every tenant's rows is an operator action
492
- with an audit trail, while an export READS every tenant's rows into one object somebody can
493
- download. `tenant: 'none'` therefore fails closed on a tenant-scoped entity
494
- (`X_TENANCY_ACTOR_ORG_REQUIRED`) and is for tables with no tenant column at all.
495
-
496
- **The format is the framework's and the columns are the app's.** The csv formula guard (a cell
497
- leading `=`, `+`, `-`, `@`, TAB or CR is evaluated by every spreadsheet) and RFC-4180 quoting are
498
- identical for a bank and a blog; which columns leave the building never is. The guard is on
499
- STRINGS only — prefixing every leading `-` would turn a refund column into text.
500
- - **`webhook()` is a FACTORY over `job()` and delivers ONE event to ONE endpoint** (`As of
501
- 2026-08-24`). Same rule `backfill()` and `purge()` follow. The unit is the ENDPOINT and not the
502
- event, because retry, backoff and disable-after-N are all per-endpoint facts: a job that fanned
503
- out inside one body would retry every subscriber because one of them was down, which is the
504
- defect `15-adding-a-feature.md` already names for a mail loop under a single step. **Which
505
- endpoints exist is the app's** (axiom 8) — the fan-out is the app's own loop, one `enqueue` per
506
- endpoint, and the idempotency key is `<name>:<endpointId>:<eventId>` for the same reason: a key
507
- on the event alone would dedupe every subscriber's delivery into the first one's row.
508
-
509
- Four things are load-bearing and none is an implementation detail. **The endpoint is never
510
- checkpointed**: it carries the secret, and a `step.run` output is written to `x_job_steps` — so
511
- this body uses NO steps and re-reads both seams per attempt. **The signature's timestamp is SEND
512
- time**, so a delivery retried three days later is signed again now and a receiver's freshness
513
- window measures the request rather than the age of the fact behind it. **The endpoint row's own
514
- `headers` merge UNDER the framework's**, because a row that could set
515
- `x-ultimate-webhook-signature` is a row that can forge one. **`redirect: 'manual'`**, because
516
- following a 3xx re-POSTs a body signed for one host to whatever the receiver named, signature
517
- and all.
518
-
519
- The ledger is a SEAM (`WebhookLedger`), the shape `PurgeTarget` already has: retention is seven
520
- years for one business and thirty days for the next, so shipping a schema would be shipping one
521
- of those answers. `record()` answers the consecutive-failure count rather than that being a
522
- second method — the number the mechanism disables on must not be readable from before the row it
523
- is deciding about. Every attempt is recorded BEFORE the throw: a failure the ledger cannot see is
524
- an endpoint that never gets disabled. Re-enabling is always the app's.
525
-
526
- **The wire format is `@ultimat3/core`'s, RE-EXPORTED and never re-declared** (`As of
527
- 2026-08-24`). `packages/core/src/webhook-signature.ts` states
528
- `v1:<timestampSeconds>:<eventId>:<topic>:<body>` once — the canonical string, the mac, the three
529
- header names and the parse a receiver reads back — because this package signs a delivery,
530
- `@ultimat3/http` verifies one, and neither may import the other. Exactly the argument
531
- `timing-safe-equal.ts` makes for itself. It shipped for one release as two implementations held
532
- together by a hex literal asserted in two test files; that works and is not a single source of
533
- truth. Core's own suite now pins the thing neither package could: that the SENDING form
534
- (`body: string`) and the RECEIVING form (`body: Uint8Array`) are the same function over the same
535
- bytes. **Never re-declare the canonical string here** — a second spelling is what the move
536
- deleted.
537
- - **`backfill()` is a FACTORY over `job()`, never a ninth primitive.** Same rule `llm()` follows
538
- in `@ultimat3/ai`: a new capability arrives as a factory over an existing primitive, so a
539
- backfill inherits `.enqueue()`, the retry policy, the cancellation, the dead-letter path and
540
- its manifest row instead of re-declaring them. Three things in `backfill.ts` are load-bearing
541
- and none is an implementation detail: a step's persisted output is the CURSOR and a row count
542
- and never the page, because `steps.ts` retains a completed step's output for the whole run and
543
- a page there is every processed row held until the job ends; step names are positional
544
- (`batch:<index>`) so the next attempt mints the same key the last one wrote, which is also why
545
- `handle` is handed no `step` — a name minted inside a per-batch body collides with itself on
546
- batch 2; and the iteration is rebuilt whenever `batches.cursor` disagrees with the checkpoint,
547
- because `retryFromStep` re-opens ONE step in the middle of a finished run and the cursor then
548
- jumps over a page the live iteration never read. A checkpoint READ back is checked rather than
549
- trusted — `step.run` replays it through an unchecked `as T`, and an absent cursor is not `null`,
550
- so the pass would silently reopen the source at the top and walk the whole table again.
551
- - **`purge()` is a FACTORY over `job()` too, and it is the ONE caller every `purgeExpired()` in
552
- the framework was missing** (`As of 2026-08-22`). Three stores shipped one —
553
- `postgresIdempotencyStore` (`x_idempotency`), `postgresRateLimitStore` (`x_rate_limit`) and
554
- `postgresAuthLimiter` (`x_auth_failures`/`x_auth_lockouts`) — each documented as "an app runs
555
- this from a `task`", and a task only ENQUEUES, so there was no job for one to enqueue and every
556
- row written was a row kept. `x_rate_limit` takes one upsert per HTTP request the web role serves,
557
- assets included.
558
-
559
- `PurgeTarget` is STRUCTURAL (`{ name, purgeExpired(nowMs) }`) for the reason `PgExecutor` is: two
560
- of those three packages are below this one and one is beside it, and a sweep that needed their
561
- types would put the HTTP pipeline on this package's import graph. `targets()` is a THUNK, read
562
- once per attempt: a host declares the sweep at boot and the auth limiter does not exist yet —
563
- `defineAuth` builds it when the app's modules import. One table per `step.run`, so a killed
564
- attempt resumes at the table it stopped on; a purge is idempotent by nature, so the replay that
565
- at-least-once guarantees deletes rows that are already gone. **One clock reading for the whole
566
- pass**, handed to every target: `postgresRateLimitStore.purgeExpired(nowMs)` requires the
567
- CALLER's clock, and reading the server's computed a 20,000,000-second refill against a frozen
568
- test clock and deleted a bucket holding 0 of 4 tokens — a free limit reset, handed out by the
569
- cleanup. Two targets under one name are refused (`X_INVARIANT`) before the first delete rather
570
- than discovered as `X_STEP_DUPLICATE` after one table is already empty.
571
-
572
- It declares no schedule of its own: `DEFAULT_PURGE_CRON` is the hourly cron a host's `task()`
573
- uses, and `@ultimat3/cli`'s `dev-purge.ts` is the one that declares both halves at boot.
574
- - **`handle` is AT LEAST ONCE, and the ordering that makes it so is deliberate.** The body runs
575
- inside the step and the record is written after it returns, so an attempt killed, cancelled or
576
- lease-expired between the two hands that page to the next attempt — which is why the doc comment,
577
- the CHANGELOG, `CLAUDE.md` and `x g backfill`'s generated source all say the handler must be
578
- idempotent. Never invert it: checkpointing first would report a page as swept that nobody wrote,
579
- and a lost page is unrecoverable where a repeated one is the handler's problem.
580
- - **A REPLAYED batch writes no ledger row** (`As of 2026-08-23`). `ledger.progress` sits outside
581
- `step.run` — it has to, the ledger is not transactional with the steps — so a resumed pass
582
- re-issued one `x_backfills` UPDATE per already-completed batch before it read a single new row:
583
- 4,800 statements on a 5M-row sweep killed at batch 4,800, on every attempt, inside the visibility
584
- lease the heartbeat is renewing. The flag is set INSIDE the step body, because that is the only
585
- thing that can tell a replay from a run — `step.run` answers the same shape either way. Nothing is
586
- lost: the value is ABSOLUTE, so the first batch that does run reports every replayed one behind
587
- it, and `finish` writes the total regardless.
588
- - **`x_backfills` is what has already been SWEPT, and the step checkpoints are where a pass
589
- resumes. Never the other way round.** The ledger row (`backfill-ledger.ts`) is a report an
590
- operator reads and the record that a completed name is done; the checkpoints are written in step
591
- with the work. A resume driven off `last_cursor` would be a second answer to "where were we"
592
- against a row that is not transactional with the rows it describes. Keyed by RUN, not by name,
593
- because `force` writes a NEW row rather than editing the one it reruns — history, not an edit of
594
- it. Only `completed` blocks: a `running` row is this pass resuming or one holding the single live
595
- idempotency key, a `failed` one is an attempt the queue is about to retry, and `start()` puts an
596
- adopted row back to `running` keeping the `started_at` the PASS began at and CLEARING
597
- `completed_at` — `finish` stamps that column for `failed` as well as for `completed`, and all
598
- four surfaces project it, so a retried run that kept it renders as running with a completion time
599
- in the past. A moved checksum WARNS
600
- and still does not run — it hashes `source` and `handle`'s source text with a `\u0000` between
601
- them (raw concatenation would hash a boundary, not a pair), and a bundler moves that text without
602
- a line of behaviour changing, which is why `@ultimat3/db`'s `auditLedger` may throw on the same
603
- fact and this may not. `force` is the only override, and it rides the input rather than the
604
- idempotency key: one live pass per name, forced or not, or "kick it again" becomes a second
605
- writer on one table.
606
- - **The throttle is spent INSIDE the batch's own `step.run`, and that is the whole of it.**
607
- `backfill-rate.ts` owns one pacer per declaration (`rate`, batches/sec, default 5); the wait is
608
- the first statement of the step body, never around it. Outside the step, an attempt resuming at
609
- batch 500 would replay 500 completed checkpoints — which run no body and touch no database — and
610
- still pay the full throttle of everything it had already done before reading a new row. The
611
- pacer takes the step's `signal` and rejects `X_ABORTED` both before the wait and after it, for
612
- the reason `execute.ts` cancels before it rejects: the wait's sleeper settles early on abort, so
613
- what it means is "the wait is over", never "the slot arrived". Built once at declaration, like
614
- the checksum: the interval belongs to the table and the pool, not to whichever attempt holds the
615
- run. `rate` is deliberately NOT in the checksum, for the reason `batch` is not. `createPacer` is
616
- exported, so it asserts the rate itself rather than trusting `backfill()` to have done it: `0`
617
- makes the interval `Infinity`, which the timer clamps to about a millisecond — an unvalidated
618
- zero is "no throttle at all", the one setting this module exists to make unreachable.
619
- - **A backfill STAMPS its own handle; an app registers nothing** (`As of 2026-08`). `backfill-registry.ts` is the
620
- `origin` WeakMap `task.ts` already uses, not a second mechanism — `stampBackfill` is called by
621
- `backfill()` and is deliberately **not** in `src/index.ts`, for the reason `registerJob` is not:
622
- a second way to claim backfill-hood would let a plain `job()` inherit the pending diff, the gate
623
- and the deploy trigger it was never declared for. `registeredBackfills()` is derived from
624
- `registeredJobs()` and never from a registry of its own — a backfill IS a job, and two registries
625
- disagreeing about one name would be two answers to "does this pass exist". Requiring an app to
626
- call `registerBackfill()` was rejected outright: that coupling is what axiom 8's extension model
627
- refuses, and a declaration an app has to declare twice is one half the apps will forget.
628
- - **The ledger says what RAN; the registry says what EXISTS; `backfill-pending.ts` is the diff**
629
- (`As of 2026-08`).
630
- That diff is the whole defect this subsystem had: `inspectBackfills` and `x db backfill --list`
631
- both read `x_backfills`, so a sweep merged and never enqueued had no row and was invisible on
632
- every surface. `pendingBackfills()` takes `BackfillProgress` rows — the one projection, never the
633
- driver's own `BackfillRun` — and `pending` deliberately excludes `running` (a pass in flight is
634
- progress, and a check red for the duration of every sweep gets muted within a week) and
635
- `excluded` (a declaration this environment may not run is not drift). Which states count is
636
- `isPendingBackfillState`, exported and read by `x db backfill --all` too: that selection used to
637
- be `report.pending.includes(row)`, an object-identity test that held only because the diff
638
- filters the array it returns — one `map` away from `--all` finding zero targets and exiting 0.
639
- - **`requires` / `environments` / `count` are DATA on the declaration, and each is enforced where
640
- the fact it needs is readable** (`As of 2026-08`). Three fields, three boundaries, and only
641
- `environments` is checked twice — deliberately:
642
-
643
- | Field | Checked in | Checked again in | Why there |
644
- |---|---|---|---|
645
- | `environments` | `backfillPass()`, ahead of the ledger open | `gateBackfill()`, for the CLI | the pass is the RAIL — app code calling `.enqueue()` never passes through a command, and a check only the CLI holds is a convention, not a build error (axiom 3). The gate's copy is a PRE-CHECK, so `x db backfill` refuses before it queues work that would only dead-letter |
646
- | `requires` | `gateBackfill()`, CLI only | — | `x_migrations` is `@ultimat3/db`'s, and this package holds no dependency on it; growing one so a tier-3 queue could read a migration ledger is the import this file refuses |
647
- | `count` | `backfillPass()`, once, after the last batch | — | it needs the pass's own `ctx` and only means anything once the source is exhausted |
648
-
649
- Refusing in the pass leaves NO ledger row — the check is ahead of `ledger.start` — so a
650
- wrong-environment enqueue never claims a sweep started. `count` is the same predicate `source`
651
- selects on, so a source that ran out while the count still matches rows means the two disagree:
652
- `X_BACKFILL_STALLED`, thrown inside the `try` so the ledger records `failed` and no completed row
653
- stops the next deploy re-running it. Its RESULT is parsed, never trusted — `NaN > 0` and `-1 > 0`
654
- are both false, so an unchecked bad number reads as "converged" and writes exactly the completed
655
- row the detector exists to prevent. Never a hardcoded "cleanups are production" and never a
656
- per-cleanup `dependsOn` graph: the real dependency is "after code tolerating both shapes is
657
- serving", which the framework cannot observe.
658
- - **`gateBackfill()` RETURNS its refusal, it does not throw one.** `x db backfill --all` isolates
659
- per name and continues past a failure — a thrown verdict would let one wedged cleanup block every
660
- later one forever. Order is the order an operator can act in: environment, then the migration it
661
- waits on, then whether it already ran. A live pass is judged by the ENQUEUE and not here, because
662
- only the enqueue can see the one live idempotency key without racing it.
663
- - **`inspectBackfills()` is the ONE projection of the ledger, and there is no second reader.**
664
- `backfill-inspect.ts` maps a `BackfillRun` to a plain JSON object (epochs to ISO, absent to
665
- `null`) for `x db backfill --list`, `x jobs ls`, `x jobs show` and `/_x`'s jobs panel — four
666
- surfaces that must not disagree about how many rows a pass has swept. It reads no clock: a
667
- running row's elapsed time is a different number in every process that asks, so `durationMs` is
668
- the pass's own completed span or `null`. It answers `[]` — never a throw — for a driver with no
669
- ledger, because `x jobs ls` is asked about the queue and must not fail over a fact nobody asked
670
- for; the surface that IS asking (`x db backfill --list`) says so in its own summary. `JobTrace`
671
- carries `backfill` for the same reason `steps` is on it: a step trace says which batch is next,
672
- the ledger row says what is behind it.
673
- - **The ledger hangs off the queue driver (`driver.backfills`), optional like `introspect`.**
674
- `x_backfills` ships in `SQL_JOBS_TABLE`, so a ledger a pass cannot write is a queue it could not
675
- have been claimed from — and `dev-queue.ts` applies that one constant, so `x dev` and production
676
- create the same table. A driver that ships none (`driver-redis`, `driver-nats`, a hand-rolled
677
- one) runs backfills with NO bookkeeping rather than refusing them: nothing blocks a completed
678
- name there, and that degradation is the price of one install point.
679
- - **`run-once` fires ONE catch-up, and the watermark is what makes it one.** `dispatch()` marks
680
- the occurrence it ran; under `run-once` that is the EARLIEST missed one, so the next round found
681
- the rest still due and fired the second, then the third — 24 nightly digests a second apart after
682
- a day down. The branch now marks `at` after dispatching, because dropping an occurrence means
683
- moving past it, and `at` rather than the last element of `due`, which `maxCatchUp` truncates.
684
- `skip` still fires the latest occurrence WITHIN the cap rather than the true latest missed —
685
- named in the README, unchanged here.
686
- - **`relay.stop()` JOINS the pass in flight, and answers a promise for it** (`As of 2026-08`). It
687
- cleared the interval and returned — the one loop in this package whose `stop()` did not wait out
688
- its own work, where `worker.stop()` waits for its rounds and `scheduler.stop()` for its dispatch.
689
- A SIGTERM landing between `driver.enqueue` and `markPublished` returned to a caller that then
690
- closed the database under the row it was about to mark. `OutboxRelay.stop(): Promise<void>` — a
691
- caller that does not await gets what it always got (the chain carries its own `catch`), so the
692
- join is only as good as the `await`. Both of `packages/cli/src/dev-roles.ts`'s paths take it
693
- today (`:306` returns `() => relay.stop()` so the rollback awaits it, `:356` is
694
- `await relay?.stop()`); this file claimed the opposite until 2026-08.
695
- - **The outbox claim is a LEASE, and one statement is what makes it one** (`As of 2026-08`).
696
- `SQL_OUTBOX_CLAIM` was a bare `select ... for update skip locked` run on the POOLED executor, and
697
- those row locks last only for their own statement — under autocommit they are gone before
698
- `claim()` resolves, and `x_outbox` had no claimed column, so nothing fenced the batch at all. Two
699
- relays 200ms apart read the identical rows and both published them. **The idempotency key does
700
- not collapse that in general**: `SQL_ENQUEUE`'s conflict target is the PARTIAL index over
701
- `('ready','delayed','running','suspended')`, so a repeat landing after the first job reached a
702
- terminal state inserts a second row and the handler runs twice. The mechanism is Postgres
703
- semantics; the second-publish-after-terminal ordering was argued, never reproduced — say it that
704
- way, in a comment or a changelog. The claim now stamps `claimed_at`/`claimed_by` in the same
705
- statement that locks (the CTE shape `SQL_CLAIM` already used), and the outer
706
- `select ... order by staged_at, id` is load-bearing: `update ... returning` has no defined row
707
- order and the relay publishes in the order it is handed rows. `claimed_at` is a LEASE and not a
708
- flag — without a reclaim window a relay that died mid-batch strands its rows forever — and
709
- `OutboxStore.release` (optional, so a store written before this still compiles) hands back the
710
- batch a failed publish stopped, or one pool blip would park committed work for a whole lease
711
- window instead of one poll interval. `createMemoryOutboxStore` answers the SAME question, on an
712
- injected clock, and `outbox-claim.test.ts` pins the two side by side.
713
- - **The lease is fenced on EVERY outbox mutation, and the sort key is TOTAL** (`As of 2026-08`).
714
- Two holes the first version of the lease left open, both reachable without a second relay
715
- process. `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED` matched on `id` alone, so a relay
716
- that stalled past its own lease still spoke for rows another relay had reclaimed: its late
717
- `release` unclaimed a batch mid-publish (a third relay claims it, publishes it again — the
718
- duplicate the lease exists to prevent, reached the long way round) and its late `markPublished`
719
- retired a row nobody had published, losing the job with nothing to notice. Both now carry
720
- `and claimed_by = $n`, `claim()` hands the token back as `OutboxRecord.claimedBy`, and the relay
721
- passes it to both calls. `markPublished` also gained `published_at is null`, so the stamp is
722
- first-writer-wins rather than a rewrite of an audit timestamp. The memory store fences the same
723
- way — per CLAIM there rather than per relay, because two relays there are two `claim()` calls on
724
- ONE store, and a per-store id could not tell them apart. **An absent token is NOT one rule in
725
- both**: `createMemoryOutboxStore`'s `owns(id, undefined)` answers `true` unconditionally, so a
726
- caller with no token really is unfenced there — while `createPgOutboxStore` substitutes
727
- `claimant ?? relayId` into `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED`, both of which
728
- carry `and claimed_by = $n`, so a token-less call fences on THIS store's relay id and no-ops
729
- against a row some other relay holds. `outbox-pg.ts:159-165` is the honest comment. Neither store
730
- refuses such a caller, which is the shared half: a caller holding no token is one written before
731
- the fence.
732
- **`order by staged_at` was not a total order**: every row staged in one transaction shares a
733
- `staged_at`, so the tie was the planner's to break — which rows the `limit` takes, and in which
734
- order they publish, differed between two relays and between two runs of one. `, id` fixes it in
735
- the CTE and in the projection, and needed NO DDL: `id` is a UUIDv7 minted by `uuid()`, monotonic
736
- and already the primary key, so the tiebreak IS stage order. `byClaimOrder` in `outbox.ts` is the
737
- memory store's copy of that key.
738
- - **`claimLeaseMs` is normalised in ONE place — `outbox-lease.ts`** (`As of 2026-08`). Both stores
739
- call `resolveClaimLeaseMs`, which owns `DEFAULT_OUTBOX_CLAIM_LEASE_MS` and refuses anything that
740
- is not a positive whole number of ms with `X_INVARIANT` (the generic, no new code: same borrow
741
- `@ultimat3/db` makes). A memory default and a pg default that could drift are two answers to
742
- "how long is a claim mine for", and the shorter one publishes a row twice. `0` expires before
743
- `claim()` resolves and `Infinity` never expires, so both are refused at CONSTRUCTION, not at the
744
- first tick where the only trace is a log line.
745
- - **A renewal is decided against `stopped()`, not only against the interval** (`As of 2026-08`).
746
- `renewal-timer.ts` is the one shape, read by `heartbeat.ts` and `worker-fleet-slots.ts`, and it
747
- exists because both files reported a LOSS for a job that had finished cleanly: `stop()` cleared
748
- the interval, which does nothing to the request already on the wire, so the fenced statement came
749
- back `false` — the row left `running` when `executeJob` acked it — and that answer took the loss
750
- branch. `jobs.lease.lost` at error plus `recordLeaseLost(queue)` is the one signal meaning the
751
- queue re-delivered a job this process was still running, so a false one is a page for a
752
- non-event, and the window widens exactly when the pool is slow. Re-read the flag AFTER every
753
- await and inside the reporter, the way `settleWithin`'s `decided` does in core. **The interval is
754
- `unref`ed**, like all three of `sync-node.ts`'s: it is armed from inside a job run, so a drain
755
- that ABANDONS the worker's hook leaves the run — and this timer — with nobody left to call
756
- `stop()`, and a refed interval is then the one thing holding a drained process open until the
757
- kubelet's SIGKILL.
758
- - **`stepTimeout` and `eventPoll` are DECLARED on the job, and `execute.ts` is the only place they
759
- are forwarded** (`As of 2026-08`). `StepRunnerOptions` carried both, `withStepTimeout`
760
- implemented the ceiling and `steps.test.ts` exercised it by building a runner BY HAND — while the
761
- only production construction passed neither and `JobDefinition` had no field that could. Same
762
- verdict as `job.concurrency`: a documented guarantee that silently does nothing is the worst of
763
- the three options, so it is threaded rather than deleted. Both are refused at declaration when
764
- non-positive, because `withStepTimeout` reads `<= 0` as "no ceiling at all".
765
- - **`registeredJobs()`/`registeredTasks()` sort by CODE UNITS, never `localeCompare`.** The list is
766
- projected by `describeJobs()` into `x.manifest.json`, which both tracked apps commit and
767
- `x verify`'s `drift` step diffs byte for byte; `localeCompare` with no locale argument answers
768
- from the runtime's ICU default and collation version. Same rule as `@ultimat3/http`'s
769
- `describeRoutes`, restated locally rather than imported — `http` is not below this package.
770
- - **A driver's semantics are pinned in ONE test with the pg statement beside them.**
771
- `driver-parity.test.ts` asserts the memory driver's behaviour and the SQL that has to mean the
772
- same thing in a single test, so neither side can move alone. `introspect.list` answered
773
- `createdAt` ASCENDING in memory and `created_at desc` in pg — one call, two answers, and because
774
- the limit lands after the sort, `x jobs ls` against `x dev` paged the hundred OLDEST rows. The
775
- `attempt` floor is the same shape: `greatest(attempt - 1, 0)` in pg, `Math.max(0, …)` in memory,
776
- with the settle fence in front of both. Two more closed `As of 2026-08`, and in BOTH the memory
777
- driver was the correct side:
778
- - **`SQL_LEASE_RENEW` fences on `expires_at > now()` as well as on `holder`.** The holder fence
779
- answers "another worker has this slot"; only the expiry fence answers "nobody has it YET". A
780
- lapsed slot is free for anyone's next `acquire`, so reviving it made the TTL an expiry other
781
- processes observed and the holder did not — and since `worker-fleet-slots.ts` reads `renew()
782
- === false` as `X_JOB_SLOT_LOST`, the same lapsed slot cancelled the run under `x dev` and
783
- silently continued in production. `createMemoryLeaseStore` has always purged on read.
784
- - **`SQL_STATS` puts a row in exactly ONE bucket.** `count(*) filter (where state = 'delayed' or
785
- run_at > now())` was every state's future row, so a `step.sleep` job counted as `suspended`
786
- AND `delayed` — the five buckets summed past the number of rows in the table. The memory
787
- driver's `if/else if` chain was always exclusive; the filter is now
788
- `state = 'delayed' or (state = 'ready' and run_at > now())`, which is that chain in SQL.
789
- - **Every timer body catches before it finalises.** `worker.ts`, `scheduler.ts` and the outbox
790
- relay all spell `void work().catch(log).finally(...)`. The relay's missing `.catch` made a
791
- rejected `store.claim()` an unhandled rejection, and Bun ends the process on one — with every
792
- staged, unpublished row still staged.
793
- - **The memory outbox store DELETES a published row.** `markPublished` rewriting it in place held
794
- every payload ever enqueued for the process's lifetime and made `claim()`/`pendingCount()` walk
795
- all of them each tick. `retained()` is the seam that makes the bound assertable; `published_at`
796
- stays a pg-only audit column.
797
- - **`claimName` reads a Set; `usedNames()` is a bounded window.** `Array.includes` per claim made
798
- a `backfill()` over a million rows quadratic — 20,000 steps, ~200M string compares — and carried
799
- a 20,000-entry array to the end of the run. The trace keeps `MAX_TRACE_NAMES` (200), most recent,
800
- and duplicate detection never reads it: a name that scrolled out is still refused.
801
- - **A `-fixture.ts` file is test material and does not ship** (`!src/**/*-fixture.ts` in `files`).
802
- `backfill-pass-fixture.ts` raises `BackfillHandleFailure`, a plain `Error` subclass on purpose:
803
- a backfill `handle` is app code and the pass propagates what it threw, so a framework code there
804
- would exercise a path no app takes.
805
- - **Suspension is control flow, and a SHED is not a suspension** (`As of 2026-08`).
806
- `StepSuspension` -> `nack({ countsAsAttempt: false, park: true })`; never log it as an error,
807
- never let it burn an attempt. The two facts were ONE flag until 2026-08: a limiter shed and a
808
- `job.concurrency` shed both handed the job back with `countsAsAttempt: false`, and both drivers
809
- derived `deadLetter ? 'dead' : counts ? 'ready' : 'suspended'` — so a job that is merely WAITING
810
- was filed beside a 3-day sleep. `SQL_STATS` and the memory `stats()` then counted it out of
811
- `ready` and out of `oldest_ready_ms`, which `worker.ts` publishes as `queue_depth` and
812
- `queue_oldest_ready_seconds`: 20 jobs at `concurrency: 10` behind `createLimiter({ global: 1 })`
813
- read as a depth of 10 with 19 waiting, and under sustained overload the shed fraction approaches
814
- 100%, so the HPA signal and the "oldest job older than 5 minutes" page both go quiet exactly
815
- when the queue is saturated. `park` is now the state and `countsAsAttempt` is the counter, only.
816
- The shed also wrote `last_error = 'limited: …'`, so `x jobs show` reported a failure for a job
817
- that never ran — it is a `jobs.worker.shed` log field now, and `worker.ts`'s one `shed()` is
818
- where both sheds go. `driver-parity.test.ts` pins which bucket each lands in, in both drivers.
819
- - **No read returns a WHOLE row, and `driver-pg-sql.test.ts` is what enforces it** (`As of
820
- 2026-08-22`). `PgExecutor` is an injected seam over any client that speaks `(text, values)`, and a
821
- client with no type map decodes `timestamptz` as TEXT — so `toJobRecord`/`toStepRecord` read
822
- `Number('2026-01-01 00:00:00+00')` and answered `NaN`. Six statements were `select *` /
823
- `returning *` and shipped that way: `pgStepStore.list`, `introspect.job`, `introspect.list`,
824
- `introspect.deadLetters`, `introspect.requeue` and `SQL_CANCEL`. Every one of them feeds a
825
- surface an operator reads — `x jobs ls`, `x jobs show`, `x jobs cancel` — and `SQL_CLAIM` had
826
- projected epoch ms all along, so the driver disagreed with itself. The guard is a scan of every
827
- production file in this directory, DISCOVERED rather than listed and with comments stripped: a
828
- `select *` anywhere in them, in either case, is a failing test rather than a review note. It
829
- reads the directory because the hand-kept list it replaces had already missed one
830
- (`driver-pg-ddl.ts`) — a registry a new SQL source opts out of by simply not joining it is a
831
- guard with the shape of a rule and none of the force.
832
- - **A run's acquisitions are handed back even when the wiring throws.** `worker-run.ts` starts the
833
- lease heartbeat first, so every line between it and the `try` can leak an interval renewing the
834
- lease of a job that never ran, with nothing left holding a reference to stop it. `context()` was
835
- moved ABOVE the heartbeat for that reason; `createRunSignal` and `fleetSlots.startRenewal` — the
836
- second an injected seam whose production implementation reaches a lease store — are INSIDE the
837
- `try`, with `undefined` meaning "never taken" and both handbacks idempotent.
838
- - Step results are persisted BEFORE the step returns. Keep it that way or replay breaks.
839
- - All time is epoch ms from an injected `Clock`, read via `nowMs()` in `clock.ts`.
840
- - Drivers implement exactly the six `JobDriver` methods plus optional `introspect`, `backfills`
841
- and `leases`. New capabilities go behind the interface, never as a driver-specific export.
842
- - `inspect.ts` returns plain JSON-serialisable objects — CLI, `/_x` and MCP share them.
843
- - **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which REFUSES rather than answering
844
- a silent no-op.** An operator cancelling a 40M-row sweep has to know whether they stopped it or
845
- missed it, so a finished job is `X_JOB_NOT_CANCELLABLE` and a driver with no `cancel` is too.
846
- - **The scheduler asks `leader.acquire()` EVERY round AND before EVERY task in it, not only while
847
- it thinks it is not the leader.** A lease-backed election expires, so `acquire()` is its renewal
848
- and a cached `isLeader = true` would keep dispatching past a lease another node already took.
849
- Per-task as well as per-round `As of 2026-08-23`: a round walks its tasks serially with an enqueue
850
- per job, so a 30s lease and a slow queue leave the tail of the walk running under a lease node B
851
- already holds — and **the occurrence key does not absorb that**, because `SQL_ENQUEUE`'s conflict
852
- target is the PARTIAL index over the live states, so a second dispatch landing after that
853
- occurrence's job completed or dead-lettered inserts a new row and the handler runs twice. Argued
854
- from the index definition, not reproduced — `stillLeading()` is the one place the question is
855
- asked and `break` is the answer, the same shape the drain's `dispatching()` check already had.
856
- `soleLeader` answers true every time and `createPgLeader` holds its grant behind an internal flag,
857
- so the extra calls are a no-op for both — the flag also stops Postgres refcounting a second advisory
858
- grant that `release()`'s single unlock would never hand back.
859
- - **`createPgLeader` is correct only on a DEDICATED connection and boot has a pool.** A
860
- session-level `pg_try_advisory_lock` is released when its connection returns to the pool, so
861
- every node reads itself as leader. `@ultimat3/realtime`'s `PgAdvisoryLock` owns its connection
862
- and is the shape that gets this right; this package holds no wire protocol, so it answers with a
863
- row that has an expiry instead — `createPgLeaseLeader`. Use that one.
864
- - **`step.run` hydrates the run's steps from ONE `store.list(runId)` and consults it before
865
- `store.get`.** A `backfill()` over 5M rows at `batch: 1000` killed at batch 4,800 used to issue
866
- 4,800 sequential `SQL_STEP_GET`s before reading a new row, re-paid on every retry and, on a slow
867
- pool, outrunning its own visibility timeout while the heartbeat was still renewing. Sound only
868
- because the `put` fence is: one attempt owns the run, so an absent name stays absent unless this
869
- runner writes it — every write updates the view, including the failure branch's direct
870
- `store.put`. `put` itself is UNCHANGED; the fence at `steps.ts` is the correctness boundary.
871
- - **`sleep` and `waitForEvent` record a replay through `trace()`, never `replayed.push`.** `trace`
872
- is what enforces `MAX_TRACE_NAMES`; the raw push let a long run's replayed-name array grow
873
- unbounded, which is the exact leak the bound exists to prevent.
874
- - `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim**, so a job/task file imports
875
- one package. Never wrap, spread or re-declare it: `t` delegates to `schemaProvider()` on every
876
- access, and a copy would freeze the provider at import time. `index.test.ts` asserts identity.
877
-
878
- ## Known coupling to other packages
879
-
880
- `clock.ts` calls `parseDuration(str)` and `scheduler.ts` calls
881
- `nextCronOccurrence(cron, { tz, from })`. Both are normalised in one place each
882
- (`finiteDurationMs`, `defaultCronResolver`) and the scheduler's resolver is injectable, so a
883
- signature change in `@ultimat3/time` is a one-line fix, not a sweep.
884
-
885
- **`clock.ts`'s conversion is `finiteDurationMs(duration, subject, option)`, never a bare `toMs`**
886
- (`As of 2026-08-26`). It was `toMs` — a THIRD implementation of duration→ms with no finiteness
887
- screen at all, and the one every scheduling decision in this package goes through:
888
- `step.sleep(Number(process.env.DELAY))` on an unset variable made `wakeAt = at + NaN`, and every
889
- `wakeAt <= now` against a `NaN` is false forever — a sleep that never ends, on a row `x jobs show`
890
- prints as `sleeping`. Partial screening is what hid it: `events.ts`, `events-pg.ts` and `steps.ts`
891
- wrapped the RESULT in `finiteOption` while `retry.ts` and `step.sleep` did not, so a reader saw
892
- `finiteOption` in the file and concluded the package was screened.
893
-
894
- Three things about the shape are load-bearing. The floor is `finiteOption` and NOT `finiteCount`,
895
- measured rather than assumed: `retry-core-parity.test.ts` pins `maxDelay: -5` at `0` and four
896
- `.job.test.ts` suites configure `retry: { delay: 0 }`, so a negative and a zero duration are
897
- shipped behaviour here — only a non-finite one is refused, and a caller wanting a positive whole
898
- number narrows on top, which is what `job()` does for `stepTimeout` and `eventPoll`. `subject` and
899
- `option` are REQUIRED, so a call site that does not name the app author's own key
900
- (`retry.delay`, `job("x") timeout`, `step.sleep`'s argument) is `TS2554` at the call rather than a
901
- review note — `@ultimat3/time`'s `toMs` screens under the subject `toMs`, which names a framework
902
- internal. And the callee CARRIES `Finite`, because `bun run finite-bounds` reads a repair off the
903
- callee's name: that is what lets `finiteDurationMs(options.defaultTtl ?? 604_800_000, …)` be
904
- recognised as screened with no second wrapper around it. `duration-bounds.test.ts` is the
905
- enforcement `finite-bounds` cannot be — a `typeof duration === 'number'` arm has no `??` in it.
906
-
907
- `driver-pg.ts`'s `PgExecutor` (`:62-64`) is a one-method duck-typed interface — this package still
908
- has no `@ultimat3/db` dependency, and nothing here knows what an observer, a span or
909
- `expectedQueryLoop` is. But `packages/cli/src/dev-queue.ts` is the only boot code in the repo that
910
- ever builds one, and it wraps a real `@ultimat3/db` client's `.query()` — reached by every role in
911
- production, not just `x dev`. So a queue statement issued through the framework's own wiring does
912
- pass through `@ultimat3/db`'s statement observer today, with no `attribution` — and `As of 2026-08`
913
- that **is** a narrow gap here, where it was none before: `@ultimat3/entity`'s `postgresRepo` now
914
- declares the `{entity, op}` pair through `withStatementAttribution`, so an entity read sharing the
915
- process carries it while a claim, an ack, a nack, an enqueue or a heartbeat does not. Nothing is
916
- wrong in this package for it — `driver-pg.ts` compiles its SQL straight against `PgExecutor` rather
917
- than through a repository, so no frame above a queue statement has an entity to name, and a
918
- detector reading `attribution` sees a claim loop as anonymous SQL until this package threads a pair
919
- of its own. That is future work, deliberately not reached from the `db` side. The visibility itself
920
- is a
921
- property of what `dev-queue.ts` happened to hand over, never of this package's own contract — an
922
- executor backed by something else (a raw driver, `driver-redis`, `driver-nats`) is invisible to it,
923
- same as before this seam existed. See `packages/db/CLAUDE.md`'s `observe.ts` section for the full
924
- picture from the other side.
22
+ `src/index.ts`. `register.ts` announces the registrars in core's table at import — never remove it
23
+ (`X_REGISTRAR_MISSING`). `isJobHandle`/`isTaskHandle` are structural plus proof `job()`/`task()`
24
+ built it.
25
+ - `idempotencyKey` is NON-OPTIONAL. **The namespace is `(name, coalesce(tenant_id, ''),
26
+ idempotency_key)`** — the index, the conflict target (spelling the index EXPRESSION exactly) and the
27
+ live-row lookup agree, pinned by a test; `driver-memory.ts` mirrors it. The DDL drops both
28
+ superseded indexes.
29
+ - **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — do not re-litigate.**
30
+ `tenant` is REQUIRED (`(input) => input.orgId` or `'none'`; `X_JOB_TENANT_REQUIRED` backstop).
31
+ `executeJob` puts the derived org on the ctx's actor (`jobRunActor`, org only) and runs the body in
32
+ `runWithContext`; `'none'` STRIPS the org (fail-closed). A boot-supplied service actor was rejected.
33
+ `tenantFor` is a METHOD on `JobHandle` (variance). The queue row's `tenantId` stays the ENQUEUER's
34
+ (the limiter's bucket). `tenancy-cross-surface.test.ts` asserts HTTP and job verdicts are EQUAL.
35
+ - **`enqueuedBy` is ATTRIBUTION, never authority** — do not re-litigate. A job acting FOR a user takes
36
+ the user's id in its input and re-authorises it in the body.
37
+ - `tz` is NON-OPTIONAL in `TaskDefinition`, validated by `@ultimat3/time`'s `isValidTimeZone`. A task
38
+ never contains a handler body.
39
+ - **`stepTimeout` and `eventPoll` are declared on the job**, forwarded only by `execute.ts`, refused at
40
+ declaration when non-positive.
41
+ - `registeredJobs()`/`registeredTasks()` sort by CODE UNITS (they reach `x.manifest.json`).
42
+ - `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim** (`index.test.ts`).
43
+
44
+ ## Rules — the driver and SQL
45
+
46
+ - **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_steps`,
47
+ `x_backfills`, `x_outbox`, `x_scheduler_state`, `x_scheduler_leader`, `x_job_leases`,
48
+ `x_job_events`). A shipped table grows by `alter table ... add column if not exists`. Its comments
49
+ carry NO apostrophes and NO semicolons (`dev-queue.ts` splits on `;`; `driver-pg-sql.test.ts` checks
50
+ quote parity).
51
+ - **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
52
+ `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's `claim`/`list`/`deadLetters` are `async` so a
53
+ refusal rejects on both.
54
+ - **`ack` and `nack` are FENCED on `state = 'running'`** — that is what makes cancellation possible.
55
+ `heartbeat` answers a boolean, read as `held === false` (never `!held`).
56
+ - **A driver's semantics are pinned in ONE test beside the pg statement** (`driver-parity.test.ts`):
57
+ list order, the `attempt` floor, `SQL_LEASE_RENEW` fencing on `expires_at > now()` and `holder`,
58
+ `SQL_STATS` one bucket per row.
59
+ - **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
60
+ (discovered, comments stripped) for `select *`/`returning *`.
61
+ - Drivers implement the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`. New
62
+ capabilities go behind the interface. `inspect.ts` returns plain JSON for CLI, `/_x` and MCP.
63
+ - Step results are persisted BEFORE the step returns. All time is epoch ms from an injected `Clock`
64
+ (`nowMs()` in `clock.ts`).
65
+ - **`createPgLeader` is correct only on a DEDICATED connection**; boot uses `createPgLeaseLeader`.
66
+ - `driver-redis.ts` / `driver-nats.ts` are honest `X_NOT_IMPLEMENTED` stubs.
67
+
68
+ ## Rules — numbers and limits
69
+
70
+ - **Every numeric knob is refused when not FINITE** — core's `finiteOption()` (a bound) and
71
+ `finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `createWorker` reads them.
72
+ **`bun run finite-bounds` is a floor, never proof**: `createLimiter`'s five numbers (`perTenant`,
73
+ `perQueue`, `global`, `ratePerTenant.limit` / `.windowMs`) are screened with min 0 (zero is a HARD
74
+ STOP). **A row count is `finiteCount` (min 0)** for driver parity (`list`, `deadLetters`, the
75
+ backfill ledger's `list`, `assertClaimBounds`).
76
+ - **`WorkerOptions.concurrency` is read by OWN key** — a queue named `constructor`
77
+ (`worker-slots.test.ts`).
78
+ - **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
79
+ `limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
80
+ throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
81
+ - **`limits.ts`'s per-tenant state is BOUNDED**: a zero counter is deleted; `starts`/`refusals` swept
82
+ and capped at `DEFAULT_MAX_LIMIT_TENANTS`, evicting the LEAST throttled first. `LimitSnapshot.tracked`
83
+ publishes both sizes.
84
+ - **`clock.ts`'s conversion is `finiteDurationMs(duration, subject, option)`**: floor `finiteOption`
85
+ (negative and zero are shipped behaviour), `subject`/`option` REQUIRED, and `Finite` in the name for
86
+ the ratchet. `duration-bounds.test.ts`. `parseDuration` and `nextCronOccurrence` are normalised in
87
+ one place each (`finiteDurationMs`, `defaultCronResolver`).
88
+
89
+ ## Rules — the worker and the scheduler
90
+
91
+ - **One enqueue implementation**: `jobsFacade()`; the only other `driver.enqueue` sites are the outbox
92
+ relay and the scheduler. `handle.as(actor, input)` QUEUES; `executeJob` is the one execution path.
93
+ - The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
94
+ job's plain key.
95
+ - **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
96
+ the poll timer, abort held runs); `close` is the teardown. `stop()` hands both back in a `finally`;
97
+ `start()` refuses while draining.
98
+ - **A claimed job is counted with core's `beginWork()`**, so the drain's in-flight phase does the
99
+ waiting. **The teardown's wait is bounded on SIGTERM** (`settleAllBy` in `drain-wait.ts`,
100
+ `jobs.worker.drain-abandoned`) and unbounded on a manual `stop()`. A worker always reaches `'stopped'`;
101
+ `state` is set in the `finally`. **One teardown, joined.**
102
+ - **SIGTERM reaches the job**: the worker's one `AbortController` (`drainSignal`) is composed into every
103
+ run (`worker-run.ts`); `stopAccepting(reason)` aborts it with `JobDrainedError` (`X_DRAINING`). A
104
+ drained attempt settles as **`interrupted`** (`nack`, `countsAsAttempt: false`, no park, no dead
105
+ letter), read off the SIGNAL. `WorkerStats.interrupted`; `worker-drain-signal.test.ts`.
106
+ - **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
107
+ synchronously with its guard.
108
+ - **A fleet slot is taken INSIDE a `try`, released AWAITED, and a renewal answering `false` CANCELS the
109
+ run** (`X_JOB_SLOT_LOST`, read as `renewed !== false`).
110
+ - **The run's signal is a controller this worker owns** (`run-signal.ts`), never `AbortSignal.any`.
111
+ - **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal is `jobs.heartbeat.failed`; a
112
+ whole window without one landing is `jobs.lease.lost` + `recordLeaseLost(queue)`, measured on this
113
+ process's clock, asked both sides of the call.
114
+ - **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
115
+ interval is `unref`ed.
116
+ - **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
117
+ - **The retry decision reads the ERROR** (`retry-classification.ts` composes around `nextRetry`;
118
+ `nextRetryForError` is `execute.ts`'s only caller). **`classifyThrown` never reads `error.retry`
119
+ alone** — core's `declaredErrorRetry(code)`. `retry-after` reuses the nack delay
120
+ (`meta.retryAfterSeconds`, clamped). The verdict is published (`stop`, `stopReason`,
121
+ `recordedFailure`).
122
+ - **The backoff arithmetic is core's** (`backoffDelay` via `backoffDelayMs`), `jitter: true` = EQUAL.
123
+ `RetryPolicy`, `DEFAULT_RETRY`, `retrySchedule()` unchanged; `BackoffStrategy` aliases core's
124
+ `BackoffCurve`. `retry-core-parity.test.ts`. **`classifyThrown` / `statedDelayMs` are core's,
125
+ re-exported** (pinned by identity).
126
+ - **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
127
+ - **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
128
+ the attempt is cancelled in `executeJob`'s `finally`.
129
+ - **A cancelled runner writes NOTHING** — `put()` in `steps.ts` is the one fence (`X_ABORTED`);
130
+ `jobs.timeout.abandoned` names an uncooperative body. `executeJob` reads `ctx.signal` defensively.
131
+ - **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
132
+ - **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
133
+ out before `leader.release()`. Same two-hook rule; an ABANDONED round does not release
134
+ (`jobs.scheduler.drain-abandoned`). **It asks `leader.acquire()` every round AND before every task**
135
+ (`stillLeading()`).
136
+ - **`run-once` fires ONE catch-up** (marks `at`); `skip` fires the true latest missed occurrence
137
+ (`latestOccurrenceBy`, bisection).
138
+ - **Every timer body catches before it finalises** (`void work().catch(log).finally(...)`).
139
+ - **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
140
+ `nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
141
+ stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
142
+ - **A run's acquisitions are handed back even when the wiring throws** (`worker-run.ts`: `context()`
143
+ above the heartbeat; `createRunSignal` and `fleetSlots.startRenewal` inside the `try`).
144
+ - **`step.run` hydrates the run's steps from ONE `store.list(runId)`**; `sleep` and `waitForEvent`
145
+ record replays through `trace()`. **`claimName` reads a Set**; `usedNames()` keeps
146
+ `MAX_TRACE_NAMES` (200).
147
+ - **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which refuses** a finished job or a
148
+ driver with no `cancel` (`X_JOB_NOT_CANCELLABLE`).
149
+
150
+ ## Rules — the outbox
151
+
152
+ - **The relay drains in two phases** (`accept` clears the interval; `close` awaits the pass under
153
+ `settleAllBy`), its timer `unref`ed. **`relay.stop()` JOINS the pass in flight.**
154
+ - **The claim is a LEASE in one statement** (`claimed_at`/`claimed_by` stamped in the locking CTE;
155
+ outer `order by staged_at, id`). `OutboxStore.release` (optional) hands back a failed batch.
156
+ `outbox-claim.test.ts` pins both stores.
157
+ - **The lease is fenced on every mutation** (`and claimed_by = $n`; `OutboxRecord.claimedBy`);
158
+ `markPublished` also requires `published_at is null`. A token-less call is unfenced in memory and
159
+ fenced on this relay's id in pg (`outbox-pg.ts` says so). `id` (UUIDv7) is the tiebreak.
160
+ - **`claimLeaseMs` is normalised in ONE place** (`outbox-lease.ts`, `resolveClaimLeaseMs`,
161
+ `X_INVARIANT` at construction).
162
+ - **The memory outbox store DELETES a published row** (`retained()` is the seam).
163
+
164
+ ## Rules — factories over `job()`
165
+
166
+ - **`backfill()`**: a step persists the CURSOR and a count, never the page; step names are positional
167
+ (`batch:<index>`), so `handle` gets no `step`; the iteration is rebuilt when `batches.cursor`
168
+ disagrees with the checkpoint; a read-back checkpoint is checked. **`handle` is AT LEAST ONCE** (it
169
+ runs before its checkpoint) — never invert. **A REPLAYED batch writes no ledger row.**
170
+ - **A `backfill()` declaring `tenant: 'none'` gets the cross-tenant scope, and nothing else does**
171
+ (`backfill-scope.ts`, `withBackfillScope`): only on the pass's own actor, for the pass's life;
172
+ `runWithContext` outside `crossTenant`. `backfill-tenancy.test.ts` drives `executeJob`.
173
+ - **`x_backfills` is what was SWEPT; the checkpoints are where a pass resumes**. Keyed by RUN; only
174
+ `completed` blocks; `start()` clears `completed_at` on an adopted row. A moved checksum warns and
175
+ does not run; `force` rides the input.
176
+ - **The throttle is spent INSIDE the batch's `step.run`** (`backfill-rate.ts`, `createPacer` asserts
177
+ its own rate).
178
+ - **A backfill STAMPS its own handle** (`stampBackfill`, not exported); `registeredBackfills()` derives
179
+ from `registeredJobs()`.
180
+ - **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff**;
181
+ `isPendingBackfillState` is shared with `x db backfill --all`.
182
+ - **`environments` is checked in `backfillPass()` (the rail) and `gateBackfill()` (a CLI pre-check);
183
+ `requires` in `gateBackfill()` only; `count` in `backfillPass()` after the last batch**
184
+ (`X_BACKFILL_STALLED`, its result parsed). `gateBackfill()` RETURNS its refusal.
185
+ - **`inspectBackfills()` is the ONE projection of the ledger**, reads no clock, answers `[]` for a
186
+ driver with none. The ledger hangs off `driver.backfills`, optional.
187
+ - **`purge()`** is the one caller of every `purgeExpired()`: `PurgeTarget` is structural, `targets()` a
188
+ thunk, one table per `step.run`, one clock reading for the whole pass, duplicate names refused
189
+ (`X_INVARIANT`). `DEFAULT_PURGE_CRON`; `@ultimat3/cli`'s `dev-purge.ts` schedules it.
190
+ - **`exportRows()`**: one object per PAGE, named by page index (a rerun rewrites the same bytes); the
191
+ interleaving assertion in `export-pass.test.ts` is the memory guard; NO cross-tenant escape; the CSV
192
+ formula guard is on strings only.
193
+ - **`webhook()` delivers ONE event to ONE endpoint** (key `<name>:<endpointId>:<eventId>`); no steps
194
+ (the endpoint carries the secret); the timestamp is SEND time; endpoint headers merge UNDER the
195
+ framework's; `redirect: 'manual'`. `WebhookLedger` is a seam; every attempt is recorded before the
196
+ throw. **The wire format is core's** (`packages/core/src/webhook-signature.ts`), re-exported.
197
+ - A `-fixture.ts` file does not ship; `backfill-pass-fixture.ts` raises a plain `Error` subclass on
198
+ purpose (it stands in for app code).
199
+
200
+ ## Known coupling
201
+
202
+ `driver-pg.ts`'s `PgExecutor` is a one-method structural interface (no `@ultimat3/db` dependency).
203
+ `packages/cli/src/dev-queue.ts` wraps a real `@ultimat3/db` client, so queue statements pass
204
+ `@ultimat3/db`'s statement observer with no `{entity, op}` attribution — future work, see
205
+ `packages/db/CLAUDE.md`'s `observe.ts` section.
925
206
 
926
207
  ## Files
927
208
 
@@ -975,22 +256,10 @@ picture from the other side.
975
256
  | `events.ts` | stored event bus for `step.waitForEvent` |
976
257
  | `inspect.ts` | `--json` introspection |
977
258
 
978
- The pass is pinned across three files off **one** fixture. `backfill-pass-fixture.ts` owns the
979
- harness — a real `@ultimat3/entity` chain over a real `memoryRepo`, wrapped so statements,
980
- iterations and closes are counted, plus `installLedger()`; `backfill-pass.test.ts` drives the
981
- iteration and its checkpoints, `backfill-pass-ledger.test.ts` the `x_backfills` row the same pass
982
- writes. Split at the 500-line ceiling, and shared rather than copied because two harnesses that
983
- drifted would be two different passes agreeing only by construction. The throttle left earlier, as
984
- `backfill-throttle.test.ts` with a fixture of its own (it needs a clock the pacer can be watched
985
- against, which the shared harness deliberately does not have).
986
-
987
- `*.job.test.ts` is the opt-in suite the `job` verify step runs: `replay.job.test.ts`,
988
- `idempotency.job.test.ts` and `outbox-atomicity.job.test.ts` each prove one named guarantee (step
989
- replay, idempotency dedupe, outbox atomicity) through a REAL worker — `start()`/`stop()` and its
990
- own poll loop, never `tick()` driven by hand. `worker-soak.job.test.ts` runs several real workers
991
- against one shared driver with one killed mid-job (its queue connection severed, not a clean
992
- `stop()`) and asserts every job still reaches a terminal state and the work behind it runs exactly
993
- once. Collected by bare `bun test`, excluded from `bun run test`.
259
+ `backfill-pass-fixture.ts` is the one harness `backfill-pass.test.ts` and
260
+ `backfill-pass-ledger.test.ts` share. `*.job.test.ts` is the opt-in `job` step: `replay`,
261
+ `idempotency` and `outbox-atomicity` each prove one guarantee through a REAL worker, and
262
+ `worker-soak.job.test.ts` kills one worker mid-job and asserts exactly-once completion.
994
263
 
995
264
  ## Commands
996
265
 
@@ -998,3 +267,5 @@ once. Collected by bare `bun test`, excluded from `bun run test`.
998
267
  bun test packages/jobs
999
268
  bun run --filter @ultimat3/jobs typecheck
1000
269
  ```
270
+
271
+ Why each rule above is shaped the way it is: [`docs/history/jobs.md`](../../docs/history/jobs.md).