@ultimat3/jobs 20.2.1 → 22.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CLAUDE.md +198 -927
- package/README.md +56 -6
- package/package.json +6 -6
- package/src/driver-memory.ts +31 -6
- package/src/driver-pg-jobs-sql.ts +34 -3
- package/src/driver-pg-sql.ts +5 -4
- package/src/driver-pg.ts +25 -13
- package/src/driver.ts +16 -1
- package/src/errors-declaration.ts +34 -0
- package/src/errors-requeue.ts +34 -0
- package/src/errors.ts +18 -4
- package/src/index.ts +2 -0
- package/src/job.ts +22 -3
- package/src/scheduler.ts +59 -38
- package/src/steps-memory.ts +5 -1
- package/src/steps.ts +21 -2
- package/src/task.ts +6 -1
- package/src/webhook-errors.ts +5 -2
- package/src/webhook-target.ts +98 -0
- package/src/webhook.ts +57 -17
- package/src/worker-hand-back.ts +30 -0
- package/src/worker-queue-depth.ts +61 -0
- package/src/worker.ts +47 -62
package/CLAUDE.md
CHANGED
|
@@ -10,918 +10,199 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
|
|
|
10
10
|
- Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
|
|
11
11
|
- External deps: none. Postgres access goes through the injected `PgExecutor`.
|
|
12
12
|
|
|
13
|
-
## Rules
|
|
14
|
-
|
|
15
|
-
- **The export name IS the job/task name.** `registerJobs(module)` / `registerTasks(module)`
|
|
16
|
-
|
|
17
|
-
`
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
A job module no call reaches keeps the positional `anonymous-job-<n>` — a name that appears
|
|
22
|
-
in no source file, on every queue row and in every dead-letter trace.
|
|
23
|
-
- The same handle under the same name is one registration seen twice (`defineApi` and the
|
|
24
|
-
framework's module scan both reach the same declaration file), not a collision. Everything
|
|
25
|
-
else that puts two things on one durable name is `X_JOB_DUPLICATE`, refused at the earliest
|
|
26
|
-
decidable point: two definitions sharing a `name:` collide inside `job()`/`task()`, and a
|
|
27
|
-
DIFFERENT handle — or the SAME handle under a second export name — collides at registration.
|
|
28
|
-
A handle exported twice matters because the rename is in place: the last alias would silently
|
|
29
|
-
move the queue key that queued rows were written to.
|
|
13
|
+
## Rules — registration and declarations
|
|
14
|
+
|
|
15
|
+
- **The export name IS the job/task name.** `registerJobs(module)` / `registerTasks(module)` stamp
|
|
16
|
+
it onto the exported handle in place. A definition with its own `name:` keeps it (a durable queue
|
|
17
|
+
key). `defineApi({ jobs, tasks })` is where a module is handed over; nothing else registers.
|
|
18
|
+
- The same handle under the same name is one registration; anything else on one durable name is
|
|
19
|
+
`X_JOB_DUPLICATE` at the earliest decidable point (same `name:` inside `job()`/`task()`; a
|
|
20
|
+
different handle, or the same handle under a second export name, at registration).
|
|
30
21
|
- `registerJob`/`registerJobs`/`registerTask`/`registerTasks`/`nameJobs`/`nameTasks` are NOT in
|
|
31
|
-
`src/index.ts`. `
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
`
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
`
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
`
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
`
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
- **`
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
`
|
|
110
|
-
`
|
|
111
|
-
|
|
112
|
-
-
|
|
113
|
-
(`
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
- **
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
`
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
`
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
`
|
|
150
|
-
`
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
- **
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
`
|
|
166
|
-
|
|
167
|
-
`
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
`
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
`
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
`
|
|
181
|
-
`runWithContext`. `
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
`
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
`
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
(`
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
identically for every sweep in the app.
|
|
216
|
-
|
|
217
|
-
**The capability is granted on the PASS's actor, not on the worker's — and that is the decision.**
|
|
218
|
-
Minting a worker identity carrying `tenancy:cross` at boot (`packages/cli/src/dev-roles.ts`
|
|
219
|
-
builds that context) would hand it to every job that worker claims, including a plain
|
|
220
|
-
`job({ tenant: 'none' })` that declared no sweep, and would move the decision into deployment
|
|
221
|
-
config where no reviewer sees it — one identity serving every job, which is the shape this whole
|
|
222
|
-
change exists to remove. Here it is bounded four ways: only `backfill()`, only on an explicit
|
|
223
|
-
`tenant: 'none'`, only for the duration of that pass, and only on a context that dies with it.
|
|
224
|
-
**So `dev-roles.ts` needs no change**, and no framework role's actor carries `tenancy:cross`.
|
|
225
|
-
`runWithContext` goes OUTSIDE `crossTenant`, never the reverse: the capability is proved against
|
|
226
|
-
the ambient actor at the call and again for every plan built inside. Nesting is safe — an app
|
|
227
|
-
`handle` opening its own `crossTenant` replaces the reason and re-proves a capability the actor
|
|
228
|
-
already holds — so an app that wraps its own body does not fight the pass.
|
|
229
|
-
- **`limits.ts`'s per-tenant state is BOUNDED, and a counter at zero is DELETED** (`As of 2026-08`).
|
|
230
|
-
`bump(…, -1)` wrote `0` and kept the key, `refusals` cleared only on a matching acquire, and
|
|
231
|
-
`starts` kept an array per tenant — four permanent entries per org in a process that never
|
|
232
|
-
restarts, which self-service org creation turns into a leak. Zero and absent already answer
|
|
233
|
-
identically (`?? 0`), so the counters drop to nothing with the run; `starts` and `refusals` are
|
|
234
|
-
swept (a spent window and a refusal past `REFUSAL_TTL_MS` are indistinguishable from missing) and
|
|
235
|
-
then capped at `DEFAULT_MAX_LIMIT_TENANTS`. The eviction order is `@ultimat3/http`'s
|
|
236
|
-
`memoryRateLimitStore`'s and must stay it: the LEAST throttled window goes first, because
|
|
237
|
-
discarding a full one hands that tenant a free rate reset. `LimitSnapshot.tracked` publishes both
|
|
238
|
-
sizes so the bound is assertable rather than assumed.
|
|
239
|
-
- **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`, and nowhere else.** The relay runs
|
|
240
|
-
after commit in its own timer with no request span in scope, so a trace read there would be
|
|
241
|
-
nobody's. A `currentSpanContext()` recovered from a `Ctx` has an empty `spanId`, which renders an
|
|
242
|
-
all-zero parent every collector rejects — that case carries no header rather than a broken one.
|
|
243
|
-
- `tz` is NON-OPTIONAL in `TaskDefinition`, and validated by `@ultimat3/time`'s `isValidTimeZone`
|
|
244
|
-
— never a local `Intl` probe, which accepts `'+02:00'`, a "zone" with no DST rules at all. A
|
|
245
|
-
non-empty string is not a timezone. A task never contains a handler body.
|
|
246
|
-
- **One enqueue implementation.** Everything (`handle.enqueue`, `handle.as`, `task.enqueue`)
|
|
247
|
-
goes through `jobsFacade()`; the only other `driver.enqueue(...)` call sites are the outbox
|
|
248
|
-
relay and the scheduler's occurrence dispatch. Never add a third.
|
|
249
|
-
- `handle.as(actor, input)` QUEUES. A job's execution surface is the queue, so it must never
|
|
250
|
-
run the handler inline — `execute.ts`'s `executeJob` is the one execution path.
|
|
251
|
-
- The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`) and `task.enqueue()`'s
|
|
252
|
-
is the job's plain key. Deliberate: the first stops two schedulers double-firing a tick, the
|
|
253
|
-
second is a manual run with no occurrence to scope to.
|
|
254
|
-
- **TWO shutdown hooks per worker — one per PHASE — and `stop()` is what hands both back**
|
|
255
|
-
(`As of 2026-08-23`). `accept` is `stopAccepting()`: flip the state, clear the poll timer, return.
|
|
256
|
-
`close` is the teardown: wait out the rounds and the in-flight jobs, close the driver.
|
|
257
|
-
|
|
258
|
-
It was ONE hook, at `accept`, doing all of it — so a single 10-minute job spent the entire drain
|
|
259
|
-
deadline inside the phase whose whole purpose is to be over immediately, and every hook behind it
|
|
260
|
-
was invoked with **0 ms** left: `@ultimat3/http`'s "stop listening" and `listenSyncNode`'s
|
|
261
|
-
`stopAccepting` are both `accept` hooks, so the pod went on serving requests and upgrading
|
|
262
|
-
websockets for the whole of the drain the load balancer had already been told about. Reproduced —
|
|
263
|
-
4 hooks started, none finished.
|
|
264
|
-
|
|
265
|
-
`start()` keeps both unregisters; the teardown releases them in a `finally`, so a close that threw
|
|
266
|
-
still gives them up. Discarding them was a hook per `start()` — the `start()` guard reads a
|
|
267
|
-
standstill, so start -> stop -> start stacked a second registration retaining a stopped worker's
|
|
268
|
-
driver, and the next process-wide drain ran all of them. `start()` refuses while draining for the
|
|
269
|
-
same reason: a claim loop back on a driver the drain is about to close.
|
|
270
|
-
- **The outbox relay drains in the same two phases, `As of 2026-09`.** It registered NO hook: on
|
|
271
|
-
SIGTERM it went on claiming and publishing through every phase of the drain, and the only thing
|
|
272
|
-
that ever stopped it was `RunningRoles.stop()` in `x dev`'s release path — a caller a signal can
|
|
273
|
-
skip. Two consequences, both silent: a lease stamped on rows nothing on this pod will run (stranded
|
|
274
|
-
for the visibility window), and a row past `driver.enqueue` but short of `markPublished` published
|
|
275
|
-
a second time on the next boot, which the idempotency key collapses only while the first job is
|
|
276
|
-
still live. `accept` clears the interval and returns; `close` awaits the pass in flight under
|
|
277
|
-
`settleAllBy(…, reason.deadlineAt)`; both unregisters come back in the teardown's `finally`. The
|
|
278
|
-
poll timer is `unref`ed for `renewal-timer.ts`'s reason — a 200ms interval refed holds the event
|
|
279
|
-
loop open past every phase and makes SIGKILL the exit.
|
|
280
|
-
- **A claimed job is counted with core's `beginWork()`, so the DRAIN does the waiting**
|
|
281
|
-
(`As of 2026-08-23`). The wait for in-flight jobs belongs to the phase between `accept` and
|
|
282
|
-
`inflight`, which exists for exactly this and is where `@ultimat3/http` already puts a request —
|
|
283
|
-
not to a hook, where one role's work starves every other role's teardown. `/readyz`'s `inflight`
|
|
284
|
-
becomes truthful on a worker node as a consequence.
|
|
285
|
-
- **The teardown's wait is BOUNDED on the SIGTERM path and unbounded on a manual `stop()`**
|
|
286
|
-
(`As of 2026-08-23`). The `close` hook is handed `ShutdownReason.deadlineAt` and passes it to
|
|
287
|
-
`settleAllBy` (`drain-wait.ts`); a manual `stop()` passes nothing and waits as long as its
|
|
288
|
-
jobs take, because a caller that asked has no budget to spend. Nothing in JS can kill a body that
|
|
289
|
-
ignores `ctx.signal`, so the unbounded version was a teardown that never returned: the driver was
|
|
290
|
-
never closed, `state` never left `'draining'`, and the memoised `stopping` promise every later
|
|
291
|
-
`stop()` joins never settled — `x dev`'s role rollback awaits that promise. Abandoning the wait
|
|
292
|
-
costs a lapsed lease and a job the queue delivers again, which is what at-least-once already
|
|
293
|
-
promises; `jobs.worker.drain-abandoned` names it, with the `configureLifecycle({ deadlineMs })`
|
|
294
|
-
raise as its fix. **A worker always REACHES `'stopped'`**, which is what makes `stop()`'s
|
|
295
|
-
`state === 'stopped'` early return an answer rather than a wedge.
|
|
296
|
-
- **SIGTERM reaches the job: the `accept` hook aborts every held run's `ctx.signal`**
|
|
297
|
-
(`As of 2026-09-07`). The worker holds ONE `AbortController` (`drainSignal`), composed into every
|
|
298
|
-
run by `worker-run.ts` as the third source beside the caller's signal and the heartbeat's;
|
|
299
|
-
`stopAccepting(reason)` aborts it with a `JobDrainedError` — core's `X_DRAINING`, naming the
|
|
300
|
-
worker and the signal — and a manual `stop()` passes no reason and aborts nothing, for the same
|
|
301
|
-
line `settleAllBy` draws: a caller that asked wants its work finished. It has to be the `accept`
|
|
302
|
-
hook and not the teardown, because core runs `accept`, then waits out in-flight work (every
|
|
303
|
-
claimed job is `beginWork()`ed) under the same budget, then `close`: told in `close`, a body
|
|
304
|
-
would hear it after the in-flight wait had already spent the whole budget on it. Which is what
|
|
305
|
-
happened until this landed — the drain told nobody, so a body reading `ctx.signal` (the one
|
|
306
|
-
documented way to stop early) ran to the deadline and was abandoned there, indistinguishable
|
|
307
|
-
from one that ignores the signal; ai-maxxing measured it as every Ctrl-C paying the full 25s
|
|
308
|
-
and wrote a process-wide signal of its own to get around it. The controller is replaced with a
|
|
309
|
-
fresh one in the teardown's `finally`: aborted once stays aborted, and a restarted worker would
|
|
310
|
-
otherwise hand every job it claimed a signal born cancelled.
|
|
311
|
-
`executeJob` reads the CODE off the run signal's reason (`drainedBy`) and settles a drained
|
|
312
|
-
attempt as **`interrupted`** — `nack` with `countsAsAttempt: false`, no park, no dead letter,
|
|
313
|
-
`delayMs: 0`, the error still recorded on the row — read off the SIGNAL and not the thrown
|
|
314
|
-
error, because what a body stops WITH is not always the reason (a killed ssh child surfaces as
|
|
315
|
-
the app's own coded error) and an attempt burned per deploy is the "always twice" draining
|
|
316
|
-
exists to prevent, one layer down. Not a `retried`: `attempts: 1` would dead-letter a job the
|
|
317
|
-
process, not the job, cut short, and `jobs_total{outcome="failed"}` would spike on every
|
|
318
|
-
rollout. The first reason on a controller wins, so a timeout that fired before the drain still
|
|
319
|
-
reports as a timeout. `WorkerStats.interrupted` counts them; `jobs.worker.drain-signalled` and
|
|
320
|
-
`jobs.attempt.interrupted` are the two lines. `worker-drain-signal.test.ts` holds all of it.
|
|
321
|
-
- **One teardown, joined.** `stop()` shares the in-flight teardown promise, so a SIGTERM landing
|
|
322
|
-
on a manual stop waits out the same in-flight jobs instead of closing the driver underneath
|
|
323
|
-
it. The promise is cleared as it settles, so a worker that started again tears down again
|
|
324
|
-
rather than joining one that settled a lifetime ago. A close that threw still stopped the
|
|
325
|
-
worker — the failure goes back on the promise the caller awaited, not into a second teardown.
|
|
326
|
-
- **The drain waits out the claim round it races, then the jobs.** `tick()` registers its round
|
|
327
|
-
in `rounds` in the same synchronous step as its guard, so a round is either refused by a drain
|
|
328
|
-
already under way or visible to every drain that starts after it. Guarding on entry alone was
|
|
329
|
-
not enough: a round parked in `driver.claim()` adds to `inFlight` AFTER a drain that only
|
|
330
|
-
snapshots `inFlight` has decided there is nothing to wait for, and `close()` then lands under a
|
|
331
|
-
live job. The round re-reads the state before each queue — "stop claiming" means this round
|
|
332
|
-
too — and what it already holds runs to the end.
|
|
333
|
-
- **A fleet slot is taken INSIDE a `try`, and a renewal answering `false` CANCELS the run**
|
|
334
|
-
(`As of 2026-08`). Two halves of one guarantee, and each was a way for `job.concurrency` to be a
|
|
335
|
-
number the framework prints and does not hold.
|
|
336
|
-
|
|
337
|
-
`fleetSlots.acquire()` is a WRITE to `x_job_leases`, so a failover, a pool timeout or a `57P01`
|
|
338
|
-
rejects it — and it sits between `limiter.tryAcquire` and the `.finally` that releases what that
|
|
339
|
-
returned. The in-process slot was burned permanently: four rejections on a concurrency-4 worker
|
|
340
|
-
and the role claims nothing again for the life of the process, with `jobs.worker.tick-failed` and
|
|
341
|
-
a climbing `queue_depth` as the only symptoms. The `catch` releases the lease and rethrows; the
|
|
342
|
-
claimed row goes back to the queue by its visibility timeout, as it does for any round that dies.
|
|
343
|
-
|
|
344
|
-
`fleetSlots.release(jobId)` is AWAITED in the settle's `.finally`, never `void`ed: the slot is a
|
|
345
|
-
DELETE in `x_job_leases`, so a fire-and-forget one was still on the wire when the teardown's
|
|
346
|
-
`allSettled` returned and `driver.close()` took the connection out from under it — the row then
|
|
347
|
-
held its slot for a full TTL, and a `concurrency: 1` job was unclaimable by the pod replacing this
|
|
348
|
-
one for a whole visibility window after every deploy. `release` swallows its own failures, so
|
|
349
|
-
awaiting it cannot turn a finished job into a rejected one.
|
|
350
|
-
|
|
351
|
-
`LeaseStore.renew` answering `false` means the row is another holder's — two runs live under a cap
|
|
352
|
-
of one — and it was discarded by `.catch(noop)`. It now stops the timer, logs
|
|
353
|
-
`jobs.worker.slot-lost` and aborts the run through `X_JOB_SLOT_LOST`. Its own code, not
|
|
354
|
-
`X_JOB_LEASE_LOST`: those are different rows on different clocks, and the queue can still consider
|
|
355
|
-
this worker the owner of the JOB while another one is running under the same cap. Read as
|
|
356
|
-
`renewed !== false` for the reason `heartbeat` reads `held === false` — a store from before the
|
|
357
|
-
return value resolves `undefined`, and treating that as a loss would cancel every job on every
|
|
358
|
-
renewal. **The heartbeat cannot cover for this**: it renews `x_jobs.visible_at`, a different row,
|
|
359
|
-
and knows nothing about `x_job_leases`.
|
|
360
|
-
- **The run's signal is a controller this worker owns, never `AbortSignal.any`** (`As of 2026-08`).
|
|
361
|
-
`run-signal.ts` composes the caller's `Ctx.signal` and the heartbeat's into one controller, and
|
|
362
|
-
`worker-run.ts` disposes it in the same `finally` that stops the timers. `AbortSignal.any` cannot
|
|
363
|
-
be undone, which cost twice: an app whose `WorkerOptions.context()` carries a process-lifetime
|
|
364
|
-
signal accumulated one composite per job run, and nothing could abort the result — so a lost fleet
|
|
365
|
-
slot had no way to reach the body running under it.
|
|
366
|
-
- **A lease is HELD, not owned, and losing one is said out loud.** `heartbeat.ts` renews the
|
|
367
|
-
window `claim()` bought and decides between two facts: one failed renewal is not a lost lease
|
|
368
|
-
(`jobs.heartbeat.failed`, warn — the window has room for the next), a window that passes with
|
|
369
|
-
nothing landing is (`jobs.lease.lost`, error, plus `recordLeaseLost(queue)`). `.catch(() =>
|
|
370
|
-
undefined)` made both look like a healthy run while the queue re-delivered the job. The window is
|
|
371
|
-
measured on THIS process's clock from the last renewal that LANDED, never against
|
|
372
|
-
`claimed.visibleAt` — that is the driver's clock, and comparing the two makes every lease
|
|
373
|
-
decision a function of skew. Expiry is asked BOTH SIDES of the call: before, because a heartbeat
|
|
374
|
-
hung on a dead connection never rejects, and after, because a renewal that SUCCEEDS past its own
|
|
375
|
-
window would otherwise restart the clock on a lease the queue already re-delivered — the loss
|
|
376
|
-
hidden inside the call meant to prevent it. A renewal that lands after the loss does not revive
|
|
377
|
-
it either, because that window is somebody else's now.
|
|
378
|
-
- **Settlement is not part of the retry decision.** `executeJob`'s `driver.ack` sits AFTER the
|
|
379
|
-
`try/catch/finally`, so only the body's own rejection can reach the retry branch. An `ack` that
|
|
380
|
-
fails — a pool timeout, a reset on that one statement — is a job that finished and could not say
|
|
381
|
-
so, not a job that failed: nacking it re-runs completed work and records `retried` in
|
|
382
|
-
`jobs_total{outcome}` for a failure that never happened. It propagates instead, and the worker's
|
|
383
|
-
`jobs.worker.settle-failed` plus the lapsing lease are the honest answer.
|
|
384
|
-
- **The retry decision reads the ERROR as well as the attempt count — added 2026-08.**
|
|
385
|
-
`executeJob` decided a retry from `nextRetry(handle.retry, attempt)` alone, so every `terminal`
|
|
386
|
-
classification in the framework was decorative on the job path: an `X_SCRAPE_AUTH_FAILED` from a
|
|
387
|
-
rotated password burned the whole policy, which at a site that locks an account after three wrong
|
|
388
|
-
passwords makes the framework's retry the thing that destroys the account.
|
|
389
|
-
`retry-classification.ts` composes AROUND `nextRetry` — the backoff arithmetic stays in one
|
|
390
|
-
place — and `nextRetryForError` is the only caller `execute.ts` has.
|
|
391
|
-
|
|
392
|
-
**`classifyThrown` must never read `error.retry` on its own.** That field is
|
|
393
|
-
`init.retry ?? retryFor(code)` and `retryFor` FAILS CLOSED, so every unclassified `UltimateError`
|
|
394
|
-
already carries `terminal`; reading it would dead-letter the first attempt of every job in every
|
|
395
|
-
app whose codes nobody has classified. Hence core's `declaredErrorRetry(code)`, which answers
|
|
396
|
-
`undefined` where `retryFor` answers the default — and hence the one case that is knowingly
|
|
397
|
-
under-read: an instance `retry: 'terminal'` on an UNREGISTERED code is indistinguishable from the
|
|
398
|
-
default and is treated as unclassified. Register the code; that is the one way.
|
|
399
|
-
`retry-after` reuses the delay the nack already takes (`meta.retryAfterSeconds`, clamped by the
|
|
400
|
-
policy's `maxDelay`) rather than a second suspension mechanism — `StepSuspension` stays the only
|
|
401
|
-
way to park a run, and unlike a suspension a retry-after DOES burn an attempt, because the work
|
|
402
|
-
failed. The ceiling outranks every classification but `terminal`.
|
|
403
|
-
|
|
404
|
-
**The verdict is published, not inferred**: `jobs.attempt.failed` and `reportError` carry
|
|
405
|
-
`stop`, `JobExecution` carries `stopReason`, and `recordedFailure` appends the terminal verdict to
|
|
406
|
-
the nack's `error` — `lastError` is the ONE failure field a row has, so without it `x jobs show`
|
|
407
|
-
renders a dead letter at attempt 1 of 5 as a silent early stop.
|
|
408
|
-
- **The backoff arithmetic is `@ultimat3/core`'s, and `retry.ts` is the option names over it**
|
|
409
|
-
(`As of 2026-08-23`). `backoffDelayMs` is `backoffDelay({ curve, jitter, base, max, attempt })`
|
|
410
|
-
with this package's spellings applied on the way in — `DurationInput` through `finiteDurationMs`, the
|
|
411
|
-
`DEFAULT_RETRY` fallbacks, and `jitter: boolean` mapped to `'equal' | 'none'`. **EQUAL, never
|
|
412
|
-
`full`**: `jitter: true` here has meant half-fixed-half-random since it shipped, and `full`
|
|
413
|
-
would hand a job that already failed twice a near-zero wait. The public `RetryPolicy`,
|
|
414
|
-
`DEFAULT_RETRY` and `retrySchedule()` are unchanged — a declared `retry: { attempts: 5 }` is
|
|
415
|
-
durable API — and `BackoffStrategy` is now an ALIAS of core's `BackoffCurve` rather than a second
|
|
416
|
-
spelling of the same three names. `retry-core-parity.test.ts` is the pin: 13,824 comparisons
|
|
417
|
-
across every curve, base, cap, attempt and roll, plus the 1-based attempt and the clamp-before-
|
|
418
|
-
jitter. Never re-derive a delay here — four packages shipped four curves, which is why core has
|
|
419
|
-
one — `bun run flight-copies` is the guard, and it refuses a second curve-and-jitter function
|
|
420
|
-
anywhere in `packages/*/src`, matched on the literal shape rather than the name.
|
|
421
|
-
- **`classifyThrown` / `statedDelayMs` are core's, RE-EXPORTED, not copied** (`As of 2026-08-23`).
|
|
422
|
-
They moved down to `packages/core/src/error-retry.ts` beside the table they read.
|
|
423
|
-
`retry-classification.test.ts` pins them by IDENTITY (`toBe`), not by agreement: two functions
|
|
424
|
-
that answer alike today are two that can drift, and the rule that must never drift is the one
|
|
425
|
-
above — an unregistered code carrying `terminal` reads as UNCLASSIFIED.
|
|
426
|
-
- **The claim loop re-arms on the PASS, never on the jobs.** A slot belongs to its own job and is
|
|
427
|
-
free the moment it settles, so `claimRound` starts what it claimed and returns the promises —
|
|
428
|
-
ending the pass on `Promise.allSettled([...inFlight])` made the pool as slow as its slowest
|
|
429
|
-
member and left every OTHER queue unasked behind it. `tick()` still resolves with the executions
|
|
430
|
-
THAT pass started (never the whole pool), which is what tests drive. Nothing awaits a job promise
|
|
431
|
-
in the timer path, so the loop observes each one itself: unobserved is an unhandled rejection,
|
|
432
|
-
and `jobs.worker.settle-failed` is where a job that could not be settled with the driver becomes
|
|
433
|
-
visible.
|
|
434
|
-
- **`state` is set in the teardown's `finally`, never after the close.** A `driver.close()` that
|
|
435
|
-
threw pinned it at `'draining'`: `stats()` reported a drain that had finished and `start()`,
|
|
436
|
-
which leaves only `'idle'` or `'stopped'`, refused that worker for the life of the process.
|
|
437
|
-
- **A deadline CANCELS, then fails the attempt — never the other way round.** `executeJob` owns
|
|
438
|
-
an `AbortController` per attempt and hands its signal to the body as `ctx.signal` (composed
|
|
439
|
-
with the caller's, never replacing it) and to the step runner. `raceTimeout` aborts before it
|
|
440
|
-
rejects because the caller nacks on that rejection and the queue hands the job straight to
|
|
441
|
-
another worker: rejecting first is one job running twice. Same order in `withStepTimeout` for
|
|
442
|
-
`stepTimeoutMs`. The attempt is also cancelled in `executeJob`'s `finally`, so a step a handler
|
|
443
|
-
left in flight cannot write into the run its successor owns.
|
|
444
|
-
- **A cancelled runner writes NOTHING, and `put()` in `steps.ts` is the one place that is
|
|
445
|
-
enforced.** Every `store.put` goes through it; aborted means `X_ABORTED` instead of the write,
|
|
446
|
-
and refusing the write is what unwinds a body that ignores the signal. The failure branch of
|
|
447
|
-
`step.run` checks the same flag directly rather than writing through `put`: the caller has to
|
|
448
|
-
see the original error, never one raised by the bookkeeping. Nothing in JS can kill an
|
|
449
|
-
uncooperative body — `jobs.timeout.abandoned` (warn) names it instead.
|
|
450
|
-
- `Ctx.signal` is non-optional in the type, and `executeJob` still reads it defensively:
|
|
451
|
-
`@ultimat3/http`'s `asCtx` casts a request context across the seam without one, and a job that
|
|
452
|
-
crashed on a missing field is worse than a job with no caller to follow.
|
|
453
|
-
- **One dispatch round at a time, and `stop()` is what waits for it.** The scheduler's timer
|
|
454
|
-
re-arms on the round it just finished (never a fixed `setInterval` period) and every other
|
|
455
|
-
caller of `tick()` JOINS the round in flight rather than opening a second one. Two rounds read
|
|
456
|
-
the same `lastFiredAt`, walk the same occurrences and dispatch them both: the occurrence key
|
|
457
|
-
deduped the jobs, so all that showed was a re-marked watermark, occurrences reported as
|
|
458
|
-
dispatched that were never enqueued, and a `run-all` catch-up interleaved with itself. `stop()`
|
|
459
|
-
waits that round out BEFORE `leader.release()` — a lock handed back mid-dispatch promotes a
|
|
460
|
-
standby onto an occurrence this node is still enqueueing for, which is the double-fire leader
|
|
461
|
-
election exists to prevent — and the round re-reads the drain state before each task, so "stop
|
|
462
|
-
dispatching" means this round too. Same hook rules as the worker, `As of 2026-08-23`: **TWO**
|
|
463
|
-
`onShutdown` registrations — `accept` stops dispatching and returns, `close` waits the round out
|
|
464
|
-
and releases the lease under the deadline it is handed (`settleAllBy`) — both handed back in the
|
|
465
|
-
teardown's `finally` so a `release()` that threw still gives them up, `isLeader` cleared there
|
|
466
|
-
too because a lock this process could not hand back is never treated as still held. **An
|
|
467
|
-
ABANDONED round does not release**: it is still enqueueing, so handing the lock over is that same
|
|
468
|
-
double-fire delivered by the shutdown, and a lease row expiring on its own is the safe end —
|
|
469
|
-
`jobs.scheduler.drain-abandoned` is the line that says so.
|
|
470
|
-
- **`exportRows()` is a FACTORY over `job()`, and the part KEY is what makes at-least-once safe**
|
|
471
|
-
(`As of 2026-08-24`). Same rule `backfill()`, `purge()` and `webhook()` follow. The paging is
|
|
472
|
-
`backfill-pass.ts`'s and never a second idiom — `inBatches()`, one page per `step.run`, and what
|
|
473
|
-
a step persists is the CURSOR and two counters, never the page.
|
|
474
|
-
|
|
475
|
-
**One object per PAGE, not one per export**, and that is the whole design rather than a
|
|
476
|
-
simplification: `StorageDriver.put()` buffers by construction — `packages/storage/src/driver.ts`
|
|
477
|
-
says the server-side path "is for objects that FIT IN MEMORY" — so a single-object export holds
|
|
478
|
-
the entire dataset, which is the failure this factory exists to prevent. Because a part is named
|
|
479
|
-
by its page INDEX, a page that runs twice rewrites the same object with the same bytes: there is
|
|
480
|
-
no idempotency argument to make about the app's rows, and no append anywhere, because a duplicate
|
|
481
|
-
part cannot be expressed. `export-pass.test.ts` proves it by deleting a mid-run checkpoint and
|
|
482
|
-
re-running, which is exactly the window at-least-once opens.
|
|
483
|
-
|
|
484
|
-
**The interleaving assertion is the memory guard, not `maxPartBytes`.** `maxPartBytes` bounds one
|
|
485
|
-
page; it cannot see a rewrite that accumulates every page and writes at the end. The test records
|
|
486
|
-
how many checkpoints had LANDED at each `put` — `0,1,2,…` for a streaming pass and a flat run of
|
|
487
|
-
the final count for a buffering one — and it is the only thing in the suite that fails on that
|
|
488
|
-
rewrite.
|
|
489
|
-
|
|
490
|
-
**An export gets NO cross-tenant escape, and `backfill()` does.** The asymmetry is deliberate and
|
|
491
|
-
is the direction of the data: a sweep that rewrites every tenant's rows is an operator action
|
|
492
|
-
with an audit trail, while an export READS every tenant's rows into one object somebody can
|
|
493
|
-
download. `tenant: 'none'` therefore fails closed on a tenant-scoped entity
|
|
494
|
-
(`X_TENANCY_ACTOR_ORG_REQUIRED`) and is for tables with no tenant column at all.
|
|
495
|
-
|
|
496
|
-
**The format is the framework's and the columns are the app's.** The csv formula guard (a cell
|
|
497
|
-
leading `=`, `+`, `-`, `@`, TAB or CR is evaluated by every spreadsheet) and RFC-4180 quoting are
|
|
498
|
-
identical for a bank and a blog; which columns leave the building never is. The guard is on
|
|
499
|
-
STRINGS only — prefixing every leading `-` would turn a refund column into text.
|
|
500
|
-
- **`webhook()` is a FACTORY over `job()` and delivers ONE event to ONE endpoint** (`As of
|
|
501
|
-
2026-08-24`). Same rule `backfill()` and `purge()` follow. The unit is the ENDPOINT and not the
|
|
502
|
-
event, because retry, backoff and disable-after-N are all per-endpoint facts: a job that fanned
|
|
503
|
-
out inside one body would retry every subscriber because one of them was down, which is the
|
|
504
|
-
defect `15-adding-a-feature.md` already names for a mail loop under a single step. **Which
|
|
505
|
-
endpoints exist is the app's** (axiom 8) — the fan-out is the app's own loop, one `enqueue` per
|
|
506
|
-
endpoint, and the idempotency key is `<name>:<endpointId>:<eventId>` for the same reason: a key
|
|
507
|
-
on the event alone would dedupe every subscriber's delivery into the first one's row.
|
|
508
|
-
|
|
509
|
-
Four things are load-bearing and none is an implementation detail. **The endpoint is never
|
|
510
|
-
checkpointed**: it carries the secret, and a `step.run` output is written to `x_job_steps` — so
|
|
511
|
-
this body uses NO steps and re-reads both seams per attempt. **The signature's timestamp is SEND
|
|
512
|
-
time**, so a delivery retried three days later is signed again now and a receiver's freshness
|
|
513
|
-
window measures the request rather than the age of the fact behind it. **The endpoint row's own
|
|
514
|
-
`headers` merge UNDER the framework's**, because a row that could set
|
|
515
|
-
`x-ultimate-webhook-signature` is a row that can forge one. **`redirect: 'manual'`**, because
|
|
516
|
-
following a 3xx re-POSTs a body signed for one host to whatever the receiver named, signature
|
|
517
|
-
and all.
|
|
518
|
-
|
|
519
|
-
The ledger is a SEAM (`WebhookLedger`), the shape `PurgeTarget` already has: retention is seven
|
|
520
|
-
years for one business and thirty days for the next, so shipping a schema would be shipping one
|
|
521
|
-
of those answers. `record()` answers the consecutive-failure count rather than that being a
|
|
522
|
-
second method — the number the mechanism disables on must not be readable from before the row it
|
|
523
|
-
is deciding about. Every attempt is recorded BEFORE the throw: a failure the ledger cannot see is
|
|
524
|
-
an endpoint that never gets disabled. Re-enabling is always the app's.
|
|
525
|
-
|
|
526
|
-
**The wire format is `@ultimat3/core`'s, RE-EXPORTED and never re-declared** (`As of
|
|
527
|
-
2026-08-24`). `packages/core/src/webhook-signature.ts` states
|
|
528
|
-
`v1:<timestampSeconds>:<eventId>:<topic>:<body>` once — the canonical string, the mac, the three
|
|
529
|
-
header names and the parse a receiver reads back — because this package signs a delivery,
|
|
530
|
-
`@ultimat3/http` verifies one, and neither may import the other. Exactly the argument
|
|
531
|
-
`timing-safe-equal.ts` makes for itself. It shipped for one release as two implementations held
|
|
532
|
-
together by a hex literal asserted in two test files; that works and is not a single source of
|
|
533
|
-
truth. Core's own suite now pins the thing neither package could: that the SENDING form
|
|
534
|
-
(`body: string`) and the RECEIVING form (`body: Uint8Array`) are the same function over the same
|
|
535
|
-
bytes. **Never re-declare the canonical string here** — a second spelling is what the move
|
|
536
|
-
deleted.
|
|
537
|
-
- **`backfill()` is a FACTORY over `job()`, never a ninth primitive.** Same rule `llm()` follows
|
|
538
|
-
in `@ultimat3/ai`: a new capability arrives as a factory over an existing primitive, so a
|
|
539
|
-
backfill inherits `.enqueue()`, the retry policy, the cancellation, the dead-letter path and
|
|
540
|
-
its manifest row instead of re-declaring them. Three things in `backfill.ts` are load-bearing
|
|
541
|
-
and none is an implementation detail: a step's persisted output is the CURSOR and a row count
|
|
542
|
-
and never the page, because `steps.ts` retains a completed step's output for the whole run and
|
|
543
|
-
a page there is every processed row held until the job ends; step names are positional
|
|
544
|
-
(`batch:<index>`) so the next attempt mints the same key the last one wrote, which is also why
|
|
545
|
-
`handle` is handed no `step` — a name minted inside a per-batch body collides with itself on
|
|
546
|
-
batch 2; and the iteration is rebuilt whenever `batches.cursor` disagrees with the checkpoint,
|
|
547
|
-
because `retryFromStep` re-opens ONE step in the middle of a finished run and the cursor then
|
|
548
|
-
jumps over a page the live iteration never read. A checkpoint READ back is checked rather than
|
|
549
|
-
trusted — `step.run` replays it through an unchecked `as T`, and an absent cursor is not `null`,
|
|
550
|
-
so the pass would silently reopen the source at the top and walk the whole table again.
|
|
551
|
-
- **`purge()` is a FACTORY over `job()` too, and it is the ONE caller every `purgeExpired()` in
|
|
552
|
-
the framework was missing** (`As of 2026-08-22`). Three stores shipped one —
|
|
553
|
-
`postgresIdempotencyStore` (`x_idempotency`), `postgresRateLimitStore` (`x_rate_limit`) and
|
|
554
|
-
`postgresAuthLimiter` (`x_auth_failures`/`x_auth_lockouts`) — each documented as "an app runs
|
|
555
|
-
this from a `task`", and a task only ENQUEUES, so there was no job for one to enqueue and every
|
|
556
|
-
row written was a row kept. `x_rate_limit` takes one upsert per HTTP request the web role serves,
|
|
557
|
-
assets included.
|
|
558
|
-
|
|
559
|
-
`PurgeTarget` is STRUCTURAL (`{ name, purgeExpired(nowMs) }`) for the reason `PgExecutor` is: two
|
|
560
|
-
of those three packages are below this one and one is beside it, and a sweep that needed their
|
|
561
|
-
types would put the HTTP pipeline on this package's import graph. `targets()` is a THUNK, read
|
|
562
|
-
once per attempt: a host declares the sweep at boot and the auth limiter does not exist yet —
|
|
563
|
-
`defineAuth` builds it when the app's modules import. One table per `step.run`, so a killed
|
|
564
|
-
attempt resumes at the table it stopped on; a purge is idempotent by nature, so the replay that
|
|
565
|
-
at-least-once guarantees deletes rows that are already gone. **One clock reading for the whole
|
|
566
|
-
pass**, handed to every target: `postgresRateLimitStore.purgeExpired(nowMs)` requires the
|
|
567
|
-
CALLER's clock, and reading the server's computed a 20,000,000-second refill against a frozen
|
|
568
|
-
test clock and deleted a bucket holding 0 of 4 tokens — a free limit reset, handed out by the
|
|
569
|
-
cleanup. Two targets under one name are refused (`X_INVARIANT`) before the first delete rather
|
|
570
|
-
than discovered as `X_STEP_DUPLICATE` after one table is already empty.
|
|
571
|
-
|
|
572
|
-
It declares no schedule of its own: `DEFAULT_PURGE_CRON` is the hourly cron a host's `task()`
|
|
573
|
-
uses, and `@ultimat3/cli`'s `dev-purge.ts` is the one that declares both halves at boot.
|
|
574
|
-
- **`handle` is AT LEAST ONCE, and the ordering that makes it so is deliberate.** The body runs
|
|
575
|
-
inside the step and the record is written after it returns, so an attempt killed, cancelled or
|
|
576
|
-
lease-expired between the two hands that page to the next attempt — which is why the doc comment,
|
|
577
|
-
the CHANGELOG, `CLAUDE.md` and `x g backfill`'s generated source all say the handler must be
|
|
578
|
-
idempotent. Never invert it: checkpointing first would report a page as swept that nobody wrote,
|
|
579
|
-
and a lost page is unrecoverable where a repeated one is the handler's problem.
|
|
580
|
-
- **A REPLAYED batch writes no ledger row** (`As of 2026-08-23`). `ledger.progress` sits outside
|
|
581
|
-
`step.run` — it has to, the ledger is not transactional with the steps — so a resumed pass
|
|
582
|
-
re-issued one `x_backfills` UPDATE per already-completed batch before it read a single new row:
|
|
583
|
-
4,800 statements on a 5M-row sweep killed at batch 4,800, on every attempt, inside the visibility
|
|
584
|
-
lease the heartbeat is renewing. The flag is set INSIDE the step body, because that is the only
|
|
585
|
-
thing that can tell a replay from a run — `step.run` answers the same shape either way. Nothing is
|
|
586
|
-
lost: the value is ABSOLUTE, so the first batch that does run reports every replayed one behind
|
|
587
|
-
it, and `finish` writes the total regardless.
|
|
588
|
-
- **`x_backfills` is what has already been SWEPT, and the step checkpoints are where a pass
|
|
589
|
-
resumes. Never the other way round.** The ledger row (`backfill-ledger.ts`) is a report an
|
|
590
|
-
operator reads and the record that a completed name is done; the checkpoints are written in step
|
|
591
|
-
with the work. A resume driven off `last_cursor` would be a second answer to "where were we"
|
|
592
|
-
against a row that is not transactional with the rows it describes. Keyed by RUN, not by name,
|
|
593
|
-
because `force` writes a NEW row rather than editing the one it reruns — history, not an edit of
|
|
594
|
-
it. Only `completed` blocks: a `running` row is this pass resuming or one holding the single live
|
|
595
|
-
idempotency key, a `failed` one is an attempt the queue is about to retry, and `start()` puts an
|
|
596
|
-
adopted row back to `running` keeping the `started_at` the PASS began at and CLEARING
|
|
597
|
-
`completed_at` — `finish` stamps that column for `failed` as well as for `completed`, and all
|
|
598
|
-
four surfaces project it, so a retried run that kept it renders as running with a completion time
|
|
599
|
-
in the past. A moved checksum WARNS
|
|
600
|
-
and still does not run — it hashes `source` and `handle`'s source text with a `\u0000` between
|
|
601
|
-
them (raw concatenation would hash a boundary, not a pair), and a bundler moves that text without
|
|
602
|
-
a line of behaviour changing, which is why `@ultimat3/db`'s `auditLedger` may throw on the same
|
|
603
|
-
fact and this may not. `force` is the only override, and it rides the input rather than the
|
|
604
|
-
idempotency key: one live pass per name, forced or not, or "kick it again" becomes a second
|
|
605
|
-
writer on one table.
|
|
606
|
-
- **The throttle is spent INSIDE the batch's own `step.run`, and that is the whole of it.**
|
|
607
|
-
`backfill-rate.ts` owns one pacer per declaration (`rate`, batches/sec, default 5); the wait is
|
|
608
|
-
the first statement of the step body, never around it. Outside the step, an attempt resuming at
|
|
609
|
-
batch 500 would replay 500 completed checkpoints — which run no body and touch no database — and
|
|
610
|
-
still pay the full throttle of everything it had already done before reading a new row. The
|
|
611
|
-
pacer takes the step's `signal` and rejects `X_ABORTED` both before the wait and after it, for
|
|
612
|
-
the reason `execute.ts` cancels before it rejects: the wait's sleeper settles early on abort, so
|
|
613
|
-
what it means is "the wait is over", never "the slot arrived". Built once at declaration, like
|
|
614
|
-
the checksum: the interval belongs to the table and the pool, not to whichever attempt holds the
|
|
615
|
-
run. `rate` is deliberately NOT in the checksum, for the reason `batch` is not. `createPacer` is
|
|
616
|
-
exported, so it asserts the rate itself rather than trusting `backfill()` to have done it: `0`
|
|
617
|
-
makes the interval `Infinity`, which the timer clamps to about a millisecond — an unvalidated
|
|
618
|
-
zero is "no throttle at all", the one setting this module exists to make unreachable.
|
|
619
|
-
- **A backfill STAMPS its own handle; an app registers nothing** (`As of 2026-08`). `backfill-registry.ts` is the
|
|
620
|
-
`origin` WeakMap `task.ts` already uses, not a second mechanism — `stampBackfill` is called by
|
|
621
|
-
`backfill()` and is deliberately **not** in `src/index.ts`, for the reason `registerJob` is not:
|
|
622
|
-
a second way to claim backfill-hood would let a plain `job()` inherit the pending diff, the gate
|
|
623
|
-
and the deploy trigger it was never declared for. `registeredBackfills()` is derived from
|
|
624
|
-
`registeredJobs()` and never from a registry of its own — a backfill IS a job, and two registries
|
|
625
|
-
disagreeing about one name would be two answers to "does this pass exist". Requiring an app to
|
|
626
|
-
call `registerBackfill()` was rejected outright: that coupling is what axiom 8's extension model
|
|
627
|
-
refuses, and a declaration an app has to declare twice is one half the apps will forget.
|
|
628
|
-
- **The ledger says what RAN; the registry says what EXISTS; `backfill-pending.ts` is the diff**
|
|
629
|
-
(`As of 2026-08`).
|
|
630
|
-
That diff is the whole defect this subsystem had: `inspectBackfills` and `x db backfill --list`
|
|
631
|
-
both read `x_backfills`, so a sweep merged and never enqueued had no row and was invisible on
|
|
632
|
-
every surface. `pendingBackfills()` takes `BackfillProgress` rows — the one projection, never the
|
|
633
|
-
driver's own `BackfillRun` — and `pending` deliberately excludes `running` (a pass in flight is
|
|
634
|
-
progress, and a check red for the duration of every sweep gets muted within a week) and
|
|
635
|
-
`excluded` (a declaration this environment may not run is not drift). Which states count is
|
|
636
|
-
`isPendingBackfillState`, exported and read by `x db backfill --all` too: that selection used to
|
|
637
|
-
be `report.pending.includes(row)`, an object-identity test that held only because the diff
|
|
638
|
-
filters the array it returns — one `map` away from `--all` finding zero targets and exiting 0.
|
|
639
|
-
- **`requires` / `environments` / `count` are DATA on the declaration, and each is enforced where
|
|
640
|
-
the fact it needs is readable** (`As of 2026-08`). Three fields, three boundaries, and only
|
|
641
|
-
`environments` is checked twice — deliberately:
|
|
642
|
-
|
|
643
|
-
| Field | Checked in | Checked again in | Why there |
|
|
644
|
-
|---|---|---|---|
|
|
645
|
-
| `environments` | `backfillPass()`, ahead of the ledger open | `gateBackfill()`, for the CLI | the pass is the RAIL — app code calling `.enqueue()` never passes through a command, and a check only the CLI holds is a convention, not a build error (axiom 3). The gate's copy is a PRE-CHECK, so `x db backfill` refuses before it queues work that would only dead-letter |
|
|
646
|
-
| `requires` | `gateBackfill()`, CLI only | — | `x_migrations` is `@ultimat3/db`'s, and this package holds no dependency on it; growing one so a tier-3 queue could read a migration ledger is the import this file refuses |
|
|
647
|
-
| `count` | `backfillPass()`, once, after the last batch | — | it needs the pass's own `ctx` and only means anything once the source is exhausted |
|
|
648
|
-
|
|
649
|
-
Refusing in the pass leaves NO ledger row — the check is ahead of `ledger.start` — so a
|
|
650
|
-
wrong-environment enqueue never claims a sweep started. `count` is the same predicate `source`
|
|
651
|
-
selects on, so a source that ran out while the count still matches rows means the two disagree:
|
|
652
|
-
`X_BACKFILL_STALLED`, thrown inside the `try` so the ledger records `failed` and no completed row
|
|
653
|
-
stops the next deploy re-running it. Its RESULT is parsed, never trusted — `NaN > 0` and `-1 > 0`
|
|
654
|
-
are both false, so an unchecked bad number reads as "converged" and writes exactly the completed
|
|
655
|
-
row the detector exists to prevent. Never a hardcoded "cleanups are production" and never a
|
|
656
|
-
per-cleanup `dependsOn` graph: the real dependency is "after code tolerating both shapes is
|
|
657
|
-
serving", which the framework cannot observe.
|
|
658
|
-
- **`gateBackfill()` RETURNS its refusal, it does not throw one.** `x db backfill --all` isolates
|
|
659
|
-
per name and continues past a failure — a thrown verdict would let one wedged cleanup block every
|
|
660
|
-
later one forever. Order is the order an operator can act in: environment, then the migration it
|
|
661
|
-
waits on, then whether it already ran. A live pass is judged by the ENQUEUE and not here, because
|
|
662
|
-
only the enqueue can see the one live idempotency key without racing it.
|
|
663
|
-
- **`inspectBackfills()` is the ONE projection of the ledger, and there is no second reader.**
|
|
664
|
-
`backfill-inspect.ts` maps a `BackfillRun` to a plain JSON object (epochs to ISO, absent to
|
|
665
|
-
`null`) for `x db backfill --list`, `x jobs ls`, `x jobs show` and `/_x`'s jobs panel — four
|
|
666
|
-
surfaces that must not disagree about how many rows a pass has swept. It reads no clock: a
|
|
667
|
-
running row's elapsed time is a different number in every process that asks, so `durationMs` is
|
|
668
|
-
the pass's own completed span or `null`. It answers `[]` — never a throw — for a driver with no
|
|
669
|
-
ledger, because `x jobs ls` is asked about the queue and must not fail over a fact nobody asked
|
|
670
|
-
for; the surface that IS asking (`x db backfill --list`) says so in its own summary. `JobTrace`
|
|
671
|
-
carries `backfill` for the same reason `steps` is on it: a step trace says which batch is next,
|
|
672
|
-
the ledger row says what is behind it.
|
|
673
|
-
- **The ledger hangs off the queue driver (`driver.backfills`), optional like `introspect`.**
|
|
674
|
-
`x_backfills` ships in `SQL_JOBS_TABLE`, so a ledger a pass cannot write is a queue it could not
|
|
675
|
-
have been claimed from — and `dev-queue.ts` applies that one constant, so `x dev` and production
|
|
676
|
-
create the same table. A driver that ships none (`driver-redis`, `driver-nats`, a hand-rolled
|
|
677
|
-
one) runs backfills with NO bookkeeping rather than refusing them: nothing blocks a completed
|
|
678
|
-
name there, and that degradation is the price of one install point.
|
|
679
|
-
- **`run-once` fires ONE catch-up, and the watermark is what makes it one.** `dispatch()` marks
|
|
680
|
-
the occurrence it ran; under `run-once` that is the EARLIEST missed one, so the next round found
|
|
681
|
-
the rest still due and fired the second, then the third — 24 nightly digests a second apart after
|
|
682
|
-
a day down. The branch now marks `at` after dispatching, because dropping an occurrence means
|
|
683
|
-
moving past it, and `at` rather than the last element of `due`, which `maxCatchUp` truncates.
|
|
684
|
-
`skip` still fires the latest occurrence WITHIN the cap rather than the true latest missed —
|
|
685
|
-
named in the README, unchanged here.
|
|
686
|
-
- **`relay.stop()` JOINS the pass in flight, and answers a promise for it** (`As of 2026-08`). It
|
|
687
|
-
cleared the interval and returned — the one loop in this package whose `stop()` did not wait out
|
|
688
|
-
its own work, where `worker.stop()` waits for its rounds and `scheduler.stop()` for its dispatch.
|
|
689
|
-
A SIGTERM landing between `driver.enqueue` and `markPublished` returned to a caller that then
|
|
690
|
-
closed the database under the row it was about to mark. `OutboxRelay.stop(): Promise<void>` — a
|
|
691
|
-
caller that does not await gets what it always got (the chain carries its own `catch`), so the
|
|
692
|
-
join is only as good as the `await`. Both of `packages/cli/src/dev-roles.ts`'s paths take it
|
|
693
|
-
today (`:306` returns `() => relay.stop()` so the rollback awaits it, `:356` is
|
|
694
|
-
`await relay?.stop()`); this file claimed the opposite until 2026-08.
|
|
695
|
-
- **The outbox claim is a LEASE, and one statement is what makes it one** (`As of 2026-08`).
|
|
696
|
-
`SQL_OUTBOX_CLAIM` was a bare `select ... for update skip locked` run on the POOLED executor, and
|
|
697
|
-
those row locks last only for their own statement — under autocommit they are gone before
|
|
698
|
-
`claim()` resolves, and `x_outbox` had no claimed column, so nothing fenced the batch at all. Two
|
|
699
|
-
relays 200ms apart read the identical rows and both published them. **The idempotency key does
|
|
700
|
-
not collapse that in general**: `SQL_ENQUEUE`'s conflict target is the PARTIAL index over
|
|
701
|
-
`('ready','delayed','running','suspended')`, so a repeat landing after the first job reached a
|
|
702
|
-
terminal state inserts a second row and the handler runs twice. The mechanism is Postgres
|
|
703
|
-
semantics; the second-publish-after-terminal ordering was argued, never reproduced — say it that
|
|
704
|
-
way, in a comment or a changelog. The claim now stamps `claimed_at`/`claimed_by` in the same
|
|
705
|
-
statement that locks (the CTE shape `SQL_CLAIM` already used), and the outer
|
|
706
|
-
`select ... order by staged_at, id` is load-bearing: `update ... returning` has no defined row
|
|
707
|
-
order and the relay publishes in the order it is handed rows. `claimed_at` is a LEASE and not a
|
|
708
|
-
flag — without a reclaim window a relay that died mid-batch strands its rows forever — and
|
|
709
|
-
`OutboxStore.release` (optional, so a store written before this still compiles) hands back the
|
|
710
|
-
batch a failed publish stopped, or one pool blip would park committed work for a whole lease
|
|
711
|
-
window instead of one poll interval. `createMemoryOutboxStore` answers the SAME question, on an
|
|
712
|
-
injected clock, and `outbox-claim.test.ts` pins the two side by side.
|
|
713
|
-
- **The lease is fenced on EVERY outbox mutation, and the sort key is TOTAL** (`As of 2026-08`).
|
|
714
|
-
Two holes the first version of the lease left open, both reachable without a second relay
|
|
715
|
-
process. `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED` matched on `id` alone, so a relay
|
|
716
|
-
that stalled past its own lease still spoke for rows another relay had reclaimed: its late
|
|
717
|
-
`release` unclaimed a batch mid-publish (a third relay claims it, publishes it again — the
|
|
718
|
-
duplicate the lease exists to prevent, reached the long way round) and its late `markPublished`
|
|
719
|
-
retired a row nobody had published, losing the job with nothing to notice. Both now carry
|
|
720
|
-
`and claimed_by = $n`, `claim()` hands the token back as `OutboxRecord.claimedBy`, and the relay
|
|
721
|
-
passes it to both calls. `markPublished` also gained `published_at is null`, so the stamp is
|
|
722
|
-
first-writer-wins rather than a rewrite of an audit timestamp. The memory store fences the same
|
|
723
|
-
way — per CLAIM there rather than per relay, because two relays there are two `claim()` calls on
|
|
724
|
-
ONE store, and a per-store id could not tell them apart. **An absent token is NOT one rule in
|
|
725
|
-
both**: `createMemoryOutboxStore`'s `owns(id, undefined)` answers `true` unconditionally, so a
|
|
726
|
-
caller with no token really is unfenced there — while `createPgOutboxStore` substitutes
|
|
727
|
-
`claimant ?? relayId` into `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED`, both of which
|
|
728
|
-
carry `and claimed_by = $n`, so a token-less call fences on THIS store's relay id and no-ops
|
|
729
|
-
against a row some other relay holds. `outbox-pg.ts:159-165` is the honest comment. Neither store
|
|
730
|
-
refuses such a caller, which is the shared half: a caller holding no token is one written before
|
|
731
|
-
the fence.
|
|
732
|
-
**`order by staged_at` was not a total order**: every row staged in one transaction shares a
|
|
733
|
-
`staged_at`, so the tie was the planner's to break — which rows the `limit` takes, and in which
|
|
734
|
-
order they publish, differed between two relays and between two runs of one. `, id` fixes it in
|
|
735
|
-
the CTE and in the projection, and needed NO DDL: `id` is a UUIDv7 minted by `uuid()`, monotonic
|
|
736
|
-
and already the primary key, so the tiebreak IS stage order. `byClaimOrder` in `outbox.ts` is the
|
|
737
|
-
memory store's copy of that key.
|
|
738
|
-
- **`claimLeaseMs` is normalised in ONE place — `outbox-lease.ts`** (`As of 2026-08`). Both stores
|
|
739
|
-
call `resolveClaimLeaseMs`, which owns `DEFAULT_OUTBOX_CLAIM_LEASE_MS` and refuses anything that
|
|
740
|
-
is not a positive whole number of ms with `X_INVARIANT` (the generic, no new code: same borrow
|
|
741
|
-
`@ultimat3/db` makes). A memory default and a pg default that could drift are two answers to
|
|
742
|
-
"how long is a claim mine for", and the shorter one publishes a row twice. `0` expires before
|
|
743
|
-
`claim()` resolves and `Infinity` never expires, so both are refused at CONSTRUCTION, not at the
|
|
744
|
-
first tick where the only trace is a log line.
|
|
745
|
-
- **A renewal is decided against `stopped()`, not only against the interval** (`As of 2026-08`).
|
|
746
|
-
`renewal-timer.ts` is the one shape, read by `heartbeat.ts` and `worker-fleet-slots.ts`, and it
|
|
747
|
-
exists because both files reported a LOSS for a job that had finished cleanly: `stop()` cleared
|
|
748
|
-
the interval, which does nothing to the request already on the wire, so the fenced statement came
|
|
749
|
-
back `false` — the row left `running` when `executeJob` acked it — and that answer took the loss
|
|
750
|
-
branch. `jobs.lease.lost` at error plus `recordLeaseLost(queue)` is the one signal meaning the
|
|
751
|
-
queue re-delivered a job this process was still running, so a false one is a page for a
|
|
752
|
-
non-event, and the window widens exactly when the pool is slow. Re-read the flag AFTER every
|
|
753
|
-
await and inside the reporter, the way `settleWithin`'s `decided` does in core. **The interval is
|
|
754
|
-
`unref`ed**, like all three of `sync-node.ts`'s: it is armed from inside a job run, so a drain
|
|
755
|
-
that ABANDONS the worker's hook leaves the run — and this timer — with nobody left to call
|
|
756
|
-
`stop()`, and a refed interval is then the one thing holding a drained process open until the
|
|
757
|
-
kubelet's SIGKILL.
|
|
758
|
-
- **`stepTimeout` and `eventPoll` are DECLARED on the job, and `execute.ts` is the only place they
|
|
759
|
-
are forwarded** (`As of 2026-08`). `StepRunnerOptions` carried both, `withStepTimeout`
|
|
760
|
-
implemented the ceiling and `steps.test.ts` exercised it by building a runner BY HAND — while the
|
|
761
|
-
only production construction passed neither and `JobDefinition` had no field that could. Same
|
|
762
|
-
verdict as `job.concurrency`: a documented guarantee that silently does nothing is the worst of
|
|
763
|
-
the three options, so it is threaded rather than deleted. Both are refused at declaration when
|
|
764
|
-
non-positive, because `withStepTimeout` reads `<= 0` as "no ceiling at all".
|
|
765
|
-
- **`registeredJobs()`/`registeredTasks()` sort by CODE UNITS, never `localeCompare`.** The list is
|
|
766
|
-
projected by `describeJobs()` into `x.manifest.json`, which both tracked apps commit and
|
|
767
|
-
`x verify`'s `drift` step diffs byte for byte; `localeCompare` with no locale argument answers
|
|
768
|
-
from the runtime's ICU default and collation version. Same rule as `@ultimat3/http`'s
|
|
769
|
-
`describeRoutes`, restated locally rather than imported — `http` is not below this package.
|
|
770
|
-
- **A driver's semantics are pinned in ONE test with the pg statement beside them.**
|
|
771
|
-
`driver-parity.test.ts` asserts the memory driver's behaviour and the SQL that has to mean the
|
|
772
|
-
same thing in a single test, so neither side can move alone. `introspect.list` answered
|
|
773
|
-
`createdAt` ASCENDING in memory and `created_at desc` in pg — one call, two answers, and because
|
|
774
|
-
the limit lands after the sort, `x jobs ls` against `x dev` paged the hundred OLDEST rows. The
|
|
775
|
-
`attempt` floor is the same shape: `greatest(attempt - 1, 0)` in pg, `Math.max(0, …)` in memory,
|
|
776
|
-
with the settle fence in front of both. Two more closed `As of 2026-08`, and in BOTH the memory
|
|
777
|
-
driver was the correct side:
|
|
778
|
-
- **`SQL_LEASE_RENEW` fences on `expires_at > now()` as well as on `holder`.** The holder fence
|
|
779
|
-
answers "another worker has this slot"; only the expiry fence answers "nobody has it YET". A
|
|
780
|
-
lapsed slot is free for anyone's next `acquire`, so reviving it made the TTL an expiry other
|
|
781
|
-
processes observed and the holder did not — and since `worker-fleet-slots.ts` reads `renew()
|
|
782
|
-
=== false` as `X_JOB_SLOT_LOST`, the same lapsed slot cancelled the run under `x dev` and
|
|
783
|
-
silently continued in production. `createMemoryLeaseStore` has always purged on read.
|
|
784
|
-
- **`SQL_STATS` puts a row in exactly ONE bucket.** `count(*) filter (where state = 'delayed' or
|
|
785
|
-
run_at > now())` was every state's future row, so a `step.sleep` job counted as `suspended`
|
|
786
|
-
AND `delayed` — the five buckets summed past the number of rows in the table. The memory
|
|
787
|
-
driver's `if/else if` chain was always exclusive; the filter is now
|
|
788
|
-
`state = 'delayed' or (state = 'ready' and run_at > now())`, which is that chain in SQL.
|
|
789
|
-
- **Every timer body catches before it finalises.** `worker.ts`, `scheduler.ts` and the outbox
|
|
790
|
-
relay all spell `void work().catch(log).finally(...)`. The relay's missing `.catch` made a
|
|
791
|
-
rejected `store.claim()` an unhandled rejection, and Bun ends the process on one — with every
|
|
792
|
-
staged, unpublished row still staged.
|
|
793
|
-
- **The memory outbox store DELETES a published row.** `markPublished` rewriting it in place held
|
|
794
|
-
every payload ever enqueued for the process's lifetime and made `claim()`/`pendingCount()` walk
|
|
795
|
-
all of them each tick. `retained()` is the seam that makes the bound assertable; `published_at`
|
|
796
|
-
stays a pg-only audit column.
|
|
797
|
-
- **`claimName` reads a Set; `usedNames()` is a bounded window.** `Array.includes` per claim made
|
|
798
|
-
a `backfill()` over a million rows quadratic — 20,000 steps, ~200M string compares — and carried
|
|
799
|
-
a 20,000-entry array to the end of the run. The trace keeps `MAX_TRACE_NAMES` (200), most recent,
|
|
800
|
-
and duplicate detection never reads it: a name that scrolled out is still refused.
|
|
801
|
-
- **A `-fixture.ts` file is test material and does not ship** (`!src/**/*-fixture.ts` in `files`).
|
|
802
|
-
`backfill-pass-fixture.ts` raises `BackfillHandleFailure`, a plain `Error` subclass on purpose:
|
|
803
|
-
a backfill `handle` is app code and the pass propagates what it threw, so a framework code there
|
|
804
|
-
would exercise a path no app takes.
|
|
805
|
-
- **Suspension is control flow, and a SHED is not a suspension** (`As of 2026-08`).
|
|
806
|
-
`StepSuspension` -> `nack({ countsAsAttempt: false, park: true })`; never log it as an error,
|
|
807
|
-
never let it burn an attempt. The two facts were ONE flag until 2026-08: a limiter shed and a
|
|
808
|
-
`job.concurrency` shed both handed the job back with `countsAsAttempt: false`, and both drivers
|
|
809
|
-
derived `deadLetter ? 'dead' : counts ? 'ready' : 'suspended'` — so a job that is merely WAITING
|
|
810
|
-
was filed beside a 3-day sleep. `SQL_STATS` and the memory `stats()` then counted it out of
|
|
811
|
-
`ready` and out of `oldest_ready_ms`, which `worker.ts` publishes as `queue_depth` and
|
|
812
|
-
`queue_oldest_ready_seconds`: 20 jobs at `concurrency: 10` behind `createLimiter({ global: 1 })`
|
|
813
|
-
read as a depth of 10 with 19 waiting, and under sustained overload the shed fraction approaches
|
|
814
|
-
100%, so the HPA signal and the "oldest job older than 5 minutes" page both go quiet exactly
|
|
815
|
-
when the queue is saturated. `park` is now the state and `countsAsAttempt` is the counter, only.
|
|
816
|
-
The shed also wrote `last_error = 'limited: …'`, so `x jobs show` reported a failure for a job
|
|
817
|
-
that never ran — it is a `jobs.worker.shed` log field now, and `worker.ts`'s one `shed()` is
|
|
818
|
-
where both sheds go. `driver-parity.test.ts` pins which bucket each lands in, in both drivers.
|
|
819
|
-
- **No read returns a WHOLE row, and `driver-pg-sql.test.ts` is what enforces it** (`As of
|
|
820
|
-
2026-08-22`). `PgExecutor` is an injected seam over any client that speaks `(text, values)`, and a
|
|
821
|
-
client with no type map decodes `timestamptz` as TEXT — so `toJobRecord`/`toStepRecord` read
|
|
822
|
-
`Number('2026-01-01 00:00:00+00')` and answered `NaN`. Six statements were `select *` /
|
|
823
|
-
`returning *` and shipped that way: `pgStepStore.list`, `introspect.job`, `introspect.list`,
|
|
824
|
-
`introspect.deadLetters`, `introspect.requeue` and `SQL_CANCEL`. Every one of them feeds a
|
|
825
|
-
surface an operator reads — `x jobs ls`, `x jobs show`, `x jobs cancel` — and `SQL_CLAIM` had
|
|
826
|
-
projected epoch ms all along, so the driver disagreed with itself. The guard is a scan of every
|
|
827
|
-
production file in this directory, DISCOVERED rather than listed and with comments stripped: a
|
|
828
|
-
`select *` anywhere in them, in either case, is a failing test rather than a review note. It
|
|
829
|
-
reads the directory because the hand-kept list it replaces had already missed one
|
|
830
|
-
(`driver-pg-ddl.ts`) — a registry a new SQL source opts out of by simply not joining it is a
|
|
831
|
-
guard with the shape of a rule and none of the force.
|
|
832
|
-
- **A run's acquisitions are handed back even when the wiring throws.** `worker-run.ts` starts the
|
|
833
|
-
lease heartbeat first, so every line between it and the `try` can leak an interval renewing the
|
|
834
|
-
lease of a job that never ran, with nothing left holding a reference to stop it. `context()` was
|
|
835
|
-
moved ABOVE the heartbeat for that reason; `createRunSignal` and `fleetSlots.startRenewal` — the
|
|
836
|
-
second an injected seam whose production implementation reaches a lease store — are INSIDE the
|
|
837
|
-
`try`, with `undefined` meaning "never taken" and both handbacks idempotent.
|
|
838
|
-
- Step results are persisted BEFORE the step returns. Keep it that way or replay breaks.
|
|
839
|
-
- All time is epoch ms from an injected `Clock`, read via `nowMs()` in `clock.ts`.
|
|
840
|
-
- Drivers implement exactly the six `JobDriver` methods plus optional `introspect`, `backfills`
|
|
841
|
-
and `leases`. New capabilities go behind the interface, never as a driver-specific export.
|
|
842
|
-
- `inspect.ts` returns plain JSON-serialisable objects — CLI, `/_x` and MCP share them.
|
|
843
|
-
- **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which REFUSES rather than answering
|
|
844
|
-
a silent no-op.** An operator cancelling a 40M-row sweep has to know whether they stopped it or
|
|
845
|
-
missed it, so a finished job is `X_JOB_NOT_CANCELLABLE` and a driver with no `cancel` is too.
|
|
846
|
-
- **The scheduler asks `leader.acquire()` EVERY round AND before EVERY task in it, not only while
|
|
847
|
-
it thinks it is not the leader.** A lease-backed election expires, so `acquire()` is its renewal
|
|
848
|
-
and a cached `isLeader = true` would keep dispatching past a lease another node already took.
|
|
849
|
-
Per-task as well as per-round `As of 2026-08-23`: a round walks its tasks serially with an enqueue
|
|
850
|
-
per job, so a 30s lease and a slow queue leave the tail of the walk running under a lease node B
|
|
851
|
-
already holds — and **the occurrence key does not absorb that**, because `SQL_ENQUEUE`'s conflict
|
|
852
|
-
target is the PARTIAL index over the live states, so a second dispatch landing after that
|
|
853
|
-
occurrence's job completed or dead-lettered inserts a new row and the handler runs twice. Argued
|
|
854
|
-
from the index definition, not reproduced — `stillLeading()` is the one place the question is
|
|
855
|
-
asked and `break` is the answer, the same shape the drain's `dispatching()` check already had.
|
|
856
|
-
`soleLeader` answers true every time and `createPgLeader` holds its grant behind an internal flag,
|
|
857
|
-
so the extra calls are a no-op for both — the flag also stops Postgres refcounting a second advisory
|
|
858
|
-
grant that `release()`'s single unlock would never hand back.
|
|
859
|
-
- **`createPgLeader` is correct only on a DEDICATED connection and boot has a pool.** A
|
|
860
|
-
session-level `pg_try_advisory_lock` is released when its connection returns to the pool, so
|
|
861
|
-
every node reads itself as leader. `@ultimat3/realtime`'s `PgAdvisoryLock` owns its connection
|
|
862
|
-
and is the shape that gets this right; this package holds no wire protocol, so it answers with a
|
|
863
|
-
row that has an expiry instead — `createPgLeaseLeader`. Use that one.
|
|
864
|
-
- **`step.run` hydrates the run's steps from ONE `store.list(runId)` and consults it before
|
|
865
|
-
`store.get`.** A `backfill()` over 5M rows at `batch: 1000` killed at batch 4,800 used to issue
|
|
866
|
-
4,800 sequential `SQL_STEP_GET`s before reading a new row, re-paid on every retry and, on a slow
|
|
867
|
-
pool, outrunning its own visibility timeout while the heartbeat was still renewing. Sound only
|
|
868
|
-
because the `put` fence is: one attempt owns the run, so an absent name stays absent unless this
|
|
869
|
-
runner writes it — every write updates the view, including the failure branch's direct
|
|
870
|
-
`store.put`. `put` itself is UNCHANGED; the fence at `steps.ts` is the correctness boundary.
|
|
871
|
-
- **`sleep` and `waitForEvent` record a replay through `trace()`, never `replayed.push`.** `trace`
|
|
872
|
-
is what enforces `MAX_TRACE_NAMES`; the raw push let a long run's replayed-name array grow
|
|
873
|
-
unbounded, which is the exact leak the bound exists to prevent.
|
|
874
|
-
- `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim**, so a job/task file imports
|
|
875
|
-
one package. Never wrap, spread or re-declare it: `t` delegates to `schemaProvider()` on every
|
|
876
|
-
access, and a copy would freeze the provider at import time. `index.test.ts` asserts identity.
|
|
877
|
-
|
|
878
|
-
## Known coupling to other packages
|
|
879
|
-
|
|
880
|
-
`clock.ts` calls `parseDuration(str)` and `scheduler.ts` calls
|
|
881
|
-
`nextCronOccurrence(cron, { tz, from })`. Both are normalised in one place each
|
|
882
|
-
(`finiteDurationMs`, `defaultCronResolver`) and the scheduler's resolver is injectable, so a
|
|
883
|
-
signature change in `@ultimat3/time` is a one-line fix, not a sweep.
|
|
884
|
-
|
|
885
|
-
**`clock.ts`'s conversion is `finiteDurationMs(duration, subject, option)`, never a bare `toMs`**
|
|
886
|
-
(`As of 2026-08-26`). It was `toMs` — a THIRD implementation of duration→ms with no finiteness
|
|
887
|
-
screen at all, and the one every scheduling decision in this package goes through:
|
|
888
|
-
`step.sleep(Number(process.env.DELAY))` on an unset variable made `wakeAt = at + NaN`, and every
|
|
889
|
-
`wakeAt <= now` against a `NaN` is false forever — a sleep that never ends, on a row `x jobs show`
|
|
890
|
-
prints as `sleeping`. Partial screening is what hid it: `events.ts`, `events-pg.ts` and `steps.ts`
|
|
891
|
-
wrapped the RESULT in `finiteOption` while `retry.ts` and `step.sleep` did not, so a reader saw
|
|
892
|
-
`finiteOption` in the file and concluded the package was screened.
|
|
893
|
-
|
|
894
|
-
Three things about the shape are load-bearing. The floor is `finiteOption` and NOT `finiteCount`,
|
|
895
|
-
measured rather than assumed: `retry-core-parity.test.ts` pins `maxDelay: -5` at `0` and four
|
|
896
|
-
`.job.test.ts` suites configure `retry: { delay: 0 }`, so a negative and a zero duration are
|
|
897
|
-
shipped behaviour here — only a non-finite one is refused, and a caller wanting a positive whole
|
|
898
|
-
number narrows on top, which is what `job()` does for `stepTimeout` and `eventPoll`. `subject` and
|
|
899
|
-
`option` are REQUIRED, so a call site that does not name the app author's own key
|
|
900
|
-
(`retry.delay`, `job("x") timeout`, `step.sleep`'s argument) is `TS2554` at the call rather than a
|
|
901
|
-
review note — `@ultimat3/time`'s `toMs` screens under the subject `toMs`, which names a framework
|
|
902
|
-
internal. And the callee CARRIES `Finite`, because `bun run finite-bounds` reads a repair off the
|
|
903
|
-
callee's name: that is what lets `finiteDurationMs(options.defaultTtl ?? 604_800_000, …)` be
|
|
904
|
-
recognised as screened with no second wrapper around it. `duration-bounds.test.ts` is the
|
|
905
|
-
enforcement `finite-bounds` cannot be — a `typeof duration === 'number'` arm has no `??` in it.
|
|
906
|
-
|
|
907
|
-
`driver-pg.ts`'s `PgExecutor` (`:62-64`) is a one-method duck-typed interface — this package still
|
|
908
|
-
has no `@ultimat3/db` dependency, and nothing here knows what an observer, a span or
|
|
909
|
-
`expectedQueryLoop` is. But `packages/cli/src/dev-queue.ts` is the only boot code in the repo that
|
|
910
|
-
ever builds one, and it wraps a real `@ultimat3/db` client's `.query()` — reached by every role in
|
|
911
|
-
production, not just `x dev`. So a queue statement issued through the framework's own wiring does
|
|
912
|
-
pass through `@ultimat3/db`'s statement observer today, with no `attribution` — and `As of 2026-08`
|
|
913
|
-
that **is** a narrow gap here, where it was none before: `@ultimat3/entity`'s `postgresRepo` now
|
|
914
|
-
declares the `{entity, op}` pair through `withStatementAttribution`, so an entity read sharing the
|
|
915
|
-
process carries it while a claim, an ack, a nack, an enqueue or a heartbeat does not. Nothing is
|
|
916
|
-
wrong in this package for it — `driver-pg.ts` compiles its SQL straight against `PgExecutor` rather
|
|
917
|
-
than through a repository, so no frame above a queue statement has an entity to name, and a
|
|
918
|
-
detector reading `attribution` sees a claim loop as anonymous SQL until this package threads a pair
|
|
919
|
-
of its own. That is future work, deliberately not reached from the `db` side. The visibility itself
|
|
920
|
-
is a
|
|
921
|
-
property of what `dev-queue.ts` happened to hand over, never of this package's own contract — an
|
|
922
|
-
executor backed by something else (a raw driver, `driver-redis`, `driver-nats`) is invisible to it,
|
|
923
|
-
same as before this seam existed. See `packages/db/CLAUDE.md`'s `observe.ts` section for the full
|
|
924
|
-
picture from the other side.
|
|
22
|
+
`src/index.ts`. `register.ts` announces the registrars in core's table at import — never remove it
|
|
23
|
+
(`X_REGISTRAR_MISSING`). `isJobHandle`/`isTaskHandle` are structural plus proof `job()`/`task()`
|
|
24
|
+
built it.
|
|
25
|
+
- `idempotencyKey` is NON-OPTIONAL. **The namespace is `(name, coalesce(tenant_id, ''),
|
|
26
|
+
idempotency_key)`** — the index, the conflict target (spelling the index EXPRESSION exactly) and the
|
|
27
|
+
live-row lookup agree, pinned by a test; `driver-memory.ts` mirrors it. The DDL drops both
|
|
28
|
+
superseded indexes.
|
|
29
|
+
- **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — do not re-litigate.**
|
|
30
|
+
`tenant` is REQUIRED (`(input) => input.orgId` or `'none'`; `X_JOB_TENANT_REQUIRED` backstop).
|
|
31
|
+
`executeJob` puts the derived org on the ctx's actor (`jobRunActor`, org only) and runs the body in
|
|
32
|
+
`runWithContext`; `'none'` STRIPS the org (fail-closed). A boot-supplied service actor was rejected.
|
|
33
|
+
`tenantFor` is a METHOD on `JobHandle` (variance). The queue row's `tenantId` stays the ENQUEUER's
|
|
34
|
+
(the limiter's bucket). `tenancy-cross-surface.test.ts` asserts HTTP and job verdicts are EQUAL.
|
|
35
|
+
- **`enqueuedBy` is ATTRIBUTION, never authority** — do not re-litigate. A job acting FOR a user takes
|
|
36
|
+
the user's id in its input and re-authorises it in the body.
|
|
37
|
+
- `tz` is NON-OPTIONAL in `TaskDefinition`, validated by `@ultimat3/time`'s `isValidTimeZone`. A task
|
|
38
|
+
never contains a handler body.
|
|
39
|
+
- **`stepTimeout` and `eventPoll` are declared on the job**, forwarded only by `execute.ts`, refused at
|
|
40
|
+
declaration when non-positive.
|
|
41
|
+
- `registeredJobs()`/`registeredTasks()` sort by CODE UNITS (they reach `x.manifest.json`).
|
|
42
|
+
- `src/index.ts` re-exports `t` from `@ultimat3/schema` **verbatim** (`index.test.ts`).
|
|
43
|
+
|
|
44
|
+
## Rules — the driver and SQL
|
|
45
|
+
|
|
46
|
+
- **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_steps`,
|
|
47
|
+
`x_backfills`, `x_outbox`, `x_scheduler_state`, `x_scheduler_leader`, `x_job_leases`,
|
|
48
|
+
`x_job_events`). A shipped table grows by `alter table ... add column if not exists`. Its comments
|
|
49
|
+
carry NO apostrophes and NO semicolons (`dev-queue.ts` splits on `;`; `driver-pg-sql.test.ts` checks
|
|
50
|
+
quote parity).
|
|
51
|
+
- **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
|
|
52
|
+
`X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's `claim`/`list`/`deadLetters` are `async` so a
|
|
53
|
+
refusal rejects on both.
|
|
54
|
+
- **`ack` and `nack` are FENCED on `state = 'running'`** — that is what makes cancellation possible.
|
|
55
|
+
`heartbeat` answers a boolean, read as `held === false` (never `!held`).
|
|
56
|
+
- **A driver's semantics are pinned in ONE test beside the pg statement** (`driver-parity.test.ts`):
|
|
57
|
+
list order, the `attempt` floor, `SQL_LEASE_RENEW` fencing on `expires_at > now()` and `holder`,
|
|
58
|
+
`SQL_STATS` one bucket per row.
|
|
59
|
+
- **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
|
|
60
|
+
(discovered, comments stripped) for `select *`/`returning *`.
|
|
61
|
+
- Drivers implement the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`. New
|
|
62
|
+
capabilities go behind the interface. `inspect.ts` returns plain JSON for CLI, `/_x` and MCP.
|
|
63
|
+
- Step results are persisted BEFORE the step returns. All time is epoch ms from an injected `Clock`
|
|
64
|
+
(`nowMs()` in `clock.ts`).
|
|
65
|
+
- **`createPgLeader` is correct only on a DEDICATED connection**; boot uses `createPgLeaseLeader`.
|
|
66
|
+
- `driver-redis.ts` / `driver-nats.ts` are honest `X_NOT_IMPLEMENTED` stubs.
|
|
67
|
+
|
|
68
|
+
## Rules — numbers and limits
|
|
69
|
+
|
|
70
|
+
- **Every numeric knob is refused when not FINITE** — core's `finiteOption()` (a bound) and
|
|
71
|
+
`finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `createWorker` reads them.
|
|
72
|
+
**`bun run finite-bounds` is a floor, never proof**: `createLimiter`'s five numbers (`perTenant`,
|
|
73
|
+
`perQueue`, `global`, `ratePerTenant.limit` / `.windowMs`) are screened with min 0 (zero is a HARD
|
|
74
|
+
STOP). **A row count is `finiteCount` (min 0)** for driver parity (`list`, `deadLetters`, the
|
|
75
|
+
backfill ledger's `list`, `assertClaimBounds`).
|
|
76
|
+
- **`WorkerOptions.concurrency` is read by OWN key** — a queue named `constructor`
|
|
77
|
+
(`worker-slots.test.ts`).
|
|
78
|
+
- **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
|
|
79
|
+
`limits.ts` is the per-process fast path. No lease store + a declared `concurrency` makes `start()`
|
|
80
|
+
throw `X_JOB_CONCURRENCY_UNENFORCEABLE`.
|
|
81
|
+
- **`limits.ts`'s per-tenant state is BOUNDED**: a zero counter is deleted; `starts`/`refusals` swept
|
|
82
|
+
and capped at `DEFAULT_MAX_LIMIT_TENANTS`, evicting the LEAST throttled first. `LimitSnapshot.tracked`
|
|
83
|
+
publishes both sizes.
|
|
84
|
+
- **`clock.ts`'s conversion is `finiteDurationMs(duration, subject, option)`**: floor `finiteOption`
|
|
85
|
+
(negative and zero are shipped behaviour), `subject`/`option` REQUIRED, and `Finite` in the name for
|
|
86
|
+
the ratchet. `duration-bounds.test.ts`. `parseDuration` and `nextCronOccurrence` are normalised in
|
|
87
|
+
one place each (`finiteDurationMs`, `defaultCronResolver`).
|
|
88
|
+
|
|
89
|
+
## Rules — the worker and the scheduler
|
|
90
|
+
|
|
91
|
+
- **One enqueue implementation**: `jobsFacade()`; the only other `driver.enqueue` sites are the outbox
|
|
92
|
+
relay and the scheduler. `handle.as(actor, input)` QUEUES; `executeJob` is the one execution path.
|
|
93
|
+
- The scheduler's key is occurrence-scoped (`task:occurrenceMs:jobKey`); `task.enqueue()` uses the
|
|
94
|
+
job's plain key.
|
|
95
|
+
- **TWO shutdown hooks per worker, one per PHASE**: `accept` is `stopAccepting()` (flip state, clear
|
|
96
|
+
the poll timer, abort held runs); `close` is the teardown. `stop()` hands both back in a `finally`;
|
|
97
|
+
`start()` refuses while draining.
|
|
98
|
+
- **A claimed job is counted with core's `beginWork()`**, so the drain's in-flight phase does the
|
|
99
|
+
waiting. **The teardown's wait is bounded on SIGTERM** (`settleAllBy` in `drain-wait.ts`,
|
|
100
|
+
`jobs.worker.drain-abandoned`) and unbounded on a manual `stop()`. A worker always reaches `'stopped'`;
|
|
101
|
+
`state` is set in the `finally`. **One teardown, joined.**
|
|
102
|
+
- **SIGTERM reaches the job**: the worker's one `AbortController` (`drainSignal`) is composed into every
|
|
103
|
+
run (`worker-run.ts`); `stopAccepting(reason)` aborts it with `JobDrainedError` (`X_DRAINING`). A
|
|
104
|
+
drained attempt settles as **`interrupted`** (`nack`, `countsAsAttempt: false`, no park, no dead
|
|
105
|
+
letter), read off the SIGNAL. `WorkerStats.interrupted`; `worker-drain-signal.test.ts`.
|
|
106
|
+
- **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
|
|
107
|
+
synchronously with its guard.
|
|
108
|
+
- **A fleet slot is taken INSIDE a `try`, released AWAITED, and a renewal answering `false` CANCELS the
|
|
109
|
+
run** (`X_JOB_SLOT_LOST`, read as `renewed !== false`).
|
|
110
|
+
- **The run's signal is a controller this worker owns** (`run-signal.ts`), never `AbortSignal.any`.
|
|
111
|
+
- **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal is `jobs.heartbeat.failed`; a
|
|
112
|
+
whole window without one landing is `jobs.lease.lost` + `recordLeaseLost(queue)`, measured on this
|
|
113
|
+
process's clock, asked both sides of the call.
|
|
114
|
+
- **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
|
|
115
|
+
interval is `unref`ed.
|
|
116
|
+
- **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
|
|
117
|
+
- **The retry decision reads the ERROR** (`retry-classification.ts` composes around `nextRetry`;
|
|
118
|
+
`nextRetryForError` is `execute.ts`'s only caller). **`classifyThrown` never reads `error.retry`
|
|
119
|
+
alone** — core's `declaredErrorRetry(code)`. `retry-after` reuses the nack delay
|
|
120
|
+
(`meta.retryAfterSeconds`, clamped). The verdict is published (`stop`, `stopReason`,
|
|
121
|
+
`recordedFailure`).
|
|
122
|
+
- **The backoff arithmetic is core's** (`backoffDelay` via `backoffDelayMs`), `jitter: true` = EQUAL.
|
|
123
|
+
`RetryPolicy`, `DEFAULT_RETRY`, `retrySchedule()` unchanged; `BackoffStrategy` aliases core's
|
|
124
|
+
`BackoffCurve`. `retry-core-parity.test.ts`. **`classifyThrown` / `statedDelayMs` are core's,
|
|
125
|
+
re-exported** (pinned by identity).
|
|
126
|
+
- **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
|
|
127
|
+
- **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
|
|
128
|
+
the attempt is cancelled in `executeJob`'s `finally`.
|
|
129
|
+
- **A cancelled runner writes NOTHING** — `put()` in `steps.ts` is the one fence (`X_ABORTED`);
|
|
130
|
+
`jobs.timeout.abandoned` names an uncooperative body. `executeJob` reads `ctx.signal` defensively.
|
|
131
|
+
- **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
|
|
132
|
+
- **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
|
|
133
|
+
out before `leader.release()`. Same two-hook rule; an ABANDONED round does not release
|
|
134
|
+
(`jobs.scheduler.drain-abandoned`). **It asks `leader.acquire()` every round AND before every task**
|
|
135
|
+
(`stillLeading()`).
|
|
136
|
+
- **`run-once` fires ONE catch-up** (marks `at`); `skip` fires the true latest missed occurrence
|
|
137
|
+
(`latestOccurrenceBy`, bisection).
|
|
138
|
+
- **Every timer body catches before it finalises** (`void work().catch(log).finally(...)`).
|
|
139
|
+
- **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
|
|
140
|
+
`nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
|
|
141
|
+
stays `ready`, logs `jobs.worker.shed` (no `last_error`). `driver-parity.test.ts`.
|
|
142
|
+
- **A run's acquisitions are handed back even when the wiring throws** (`worker-run.ts`: `context()`
|
|
143
|
+
above the heartbeat; `createRunSignal` and `fleetSlots.startRenewal` inside the `try`).
|
|
144
|
+
- **`step.run` hydrates the run's steps from ONE `store.list(runId)`**; `sleep` and `waitForEvent`
|
|
145
|
+
record replays through `trace()`. **`claimName` reads a Set**; `usedNames()` keeps
|
|
146
|
+
`MAX_TRACE_NAMES` (200).
|
|
147
|
+
- **`x jobs cancel` binds to `cancelJob(driver, id, reason?)`, which refuses** a finished job or a
|
|
148
|
+
driver with no `cancel` (`X_JOB_NOT_CANCELLABLE`).
|
|
149
|
+
|
|
150
|
+
## Rules — the outbox
|
|
151
|
+
|
|
152
|
+
- **The relay drains in two phases** (`accept` clears the interval; `close` awaits the pass under
|
|
153
|
+
`settleAllBy`), its timer `unref`ed. **`relay.stop()` JOINS the pass in flight.**
|
|
154
|
+
- **The claim is a LEASE in one statement** (`claimed_at`/`claimed_by` stamped in the locking CTE;
|
|
155
|
+
outer `order by staged_at, id`). `OutboxStore.release` (optional) hands back a failed batch.
|
|
156
|
+
`outbox-claim.test.ts` pins both stores.
|
|
157
|
+
- **The lease is fenced on every mutation** (`and claimed_by = $n`; `OutboxRecord.claimedBy`);
|
|
158
|
+
`markPublished` also requires `published_at is null`. A token-less call is unfenced in memory and
|
|
159
|
+
fenced on this relay's id in pg (`outbox-pg.ts` says so). `id` (UUIDv7) is the tiebreak.
|
|
160
|
+
- **`claimLeaseMs` is normalised in ONE place** (`outbox-lease.ts`, `resolveClaimLeaseMs`,
|
|
161
|
+
`X_INVARIANT` at construction).
|
|
162
|
+
- **The memory outbox store DELETES a published row** (`retained()` is the seam).
|
|
163
|
+
|
|
164
|
+
## Rules — factories over `job()`
|
|
165
|
+
|
|
166
|
+
- **`backfill()`**: a step persists the CURSOR and a count, never the page; step names are positional
|
|
167
|
+
(`batch:<index>`), so `handle` gets no `step`; the iteration is rebuilt when `batches.cursor`
|
|
168
|
+
disagrees with the checkpoint; a read-back checkpoint is checked. **`handle` is AT LEAST ONCE** (it
|
|
169
|
+
runs before its checkpoint) — never invert. **A REPLAYED batch writes no ledger row.**
|
|
170
|
+
- **A `backfill()` declaring `tenant: 'none'` gets the cross-tenant scope, and nothing else does**
|
|
171
|
+
(`backfill-scope.ts`, `withBackfillScope`): only on the pass's own actor, for the pass's life;
|
|
172
|
+
`runWithContext` outside `crossTenant`. `backfill-tenancy.test.ts` drives `executeJob`.
|
|
173
|
+
- **`x_backfills` is what was SWEPT; the checkpoints are where a pass resumes**. Keyed by RUN; only
|
|
174
|
+
`completed` blocks; `start()` clears `completed_at` on an adopted row. A moved checksum warns and
|
|
175
|
+
does not run; `force` rides the input.
|
|
176
|
+
- **The throttle is spent INSIDE the batch's `step.run`** (`backfill-rate.ts`, `createPacer` asserts
|
|
177
|
+
its own rate).
|
|
178
|
+
- **A backfill STAMPS its own handle** (`stampBackfill`, not exported); `registeredBackfills()` derives
|
|
179
|
+
from `registeredJobs()`.
|
|
180
|
+
- **The ledger says what RAN, the registry what EXISTS, `backfill-pending.ts` is the diff**;
|
|
181
|
+
`isPendingBackfillState` is shared with `x db backfill --all`.
|
|
182
|
+
- **`environments` is checked in `backfillPass()` (the rail) and `gateBackfill()` (a CLI pre-check);
|
|
183
|
+
`requires` in `gateBackfill()` only; `count` in `backfillPass()` after the last batch**
|
|
184
|
+
(`X_BACKFILL_STALLED`, its result parsed). `gateBackfill()` RETURNS its refusal.
|
|
185
|
+
- **`inspectBackfills()` is the ONE projection of the ledger**, reads no clock, answers `[]` for a
|
|
186
|
+
driver with none. The ledger hangs off `driver.backfills`, optional.
|
|
187
|
+
- **`purge()`** is the one caller of every `purgeExpired()`: `PurgeTarget` is structural, `targets()` a
|
|
188
|
+
thunk, one table per `step.run`, one clock reading for the whole pass, duplicate names refused
|
|
189
|
+
(`X_INVARIANT`). `DEFAULT_PURGE_CRON`; `@ultimat3/cli`'s `dev-purge.ts` schedules it.
|
|
190
|
+
- **`exportRows()`**: one object per PAGE, named by page index (a rerun rewrites the same bytes); the
|
|
191
|
+
interleaving assertion in `export-pass.test.ts` is the memory guard; NO cross-tenant escape; the CSV
|
|
192
|
+
formula guard is on strings only.
|
|
193
|
+
- **`webhook()` delivers ONE event to ONE endpoint** (key `<name>:<endpointId>:<eventId>`); no steps
|
|
194
|
+
(the endpoint carries the secret); the timestamp is SEND time; endpoint headers merge UNDER the
|
|
195
|
+
framework's; `redirect: 'manual'`. `WebhookLedger` is a seam; every attempt is recorded before the
|
|
196
|
+
throw. **The wire format is core's** (`packages/core/src/webhook-signature.ts`), re-exported.
|
|
197
|
+
- A `-fixture.ts` file does not ship; `backfill-pass-fixture.ts` raises a plain `Error` subclass on
|
|
198
|
+
purpose (it stands in for app code).
|
|
199
|
+
|
|
200
|
+
## Known coupling
|
|
201
|
+
|
|
202
|
+
`driver-pg.ts`'s `PgExecutor` is a one-method structural interface (no `@ultimat3/db` dependency).
|
|
203
|
+
`packages/cli/src/dev-queue.ts` wraps a real `@ultimat3/db` client, so queue statements pass
|
|
204
|
+
`@ultimat3/db`'s statement observer with no `{entity, op}` attribution — future work, see
|
|
205
|
+
`packages/db/CLAUDE.md`'s `observe.ts` section.
|
|
925
206
|
|
|
926
207
|
## Files
|
|
927
208
|
|
|
@@ -975,22 +256,10 @@ picture from the other side.
|
|
|
975
256
|
| `events.ts` | stored event bus for `step.waitForEvent` |
|
|
976
257
|
| `inspect.ts` | `--json` introspection |
|
|
977
258
|
|
|
978
|
-
|
|
979
|
-
|
|
980
|
-
|
|
981
|
-
|
|
982
|
-
writes. Split at the 500-line ceiling, and shared rather than copied because two harnesses that
|
|
983
|
-
drifted would be two different passes agreeing only by construction. The throttle left earlier, as
|
|
984
|
-
`backfill-throttle.test.ts` with a fixture of its own (it needs a clock the pacer can be watched
|
|
985
|
-
against, which the shared harness deliberately does not have).
|
|
986
|
-
|
|
987
|
-
`*.job.test.ts` is the opt-in suite the `job` verify step runs: `replay.job.test.ts`,
|
|
988
|
-
`idempotency.job.test.ts` and `outbox-atomicity.job.test.ts` each prove one named guarantee (step
|
|
989
|
-
replay, idempotency dedupe, outbox atomicity) through a REAL worker — `start()`/`stop()` and its
|
|
990
|
-
own poll loop, never `tick()` driven by hand. `worker-soak.job.test.ts` runs several real workers
|
|
991
|
-
against one shared driver with one killed mid-job (its queue connection severed, not a clean
|
|
992
|
-
`stop()`) and asserts every job still reaches a terminal state and the work behind it runs exactly
|
|
993
|
-
once. Collected by bare `bun test`, excluded from `bun run test`.
|
|
259
|
+
`backfill-pass-fixture.ts` is the one harness `backfill-pass.test.ts` and
|
|
260
|
+
`backfill-pass-ledger.test.ts` share. `*.job.test.ts` is the opt-in `job` step: `replay`,
|
|
261
|
+
`idempotency` and `outbox-atomicity` each prove one guarantee through a REAL worker, and
|
|
262
|
+
`worker-soak.job.test.ts` kills one worker mid-job and asserts exactly-once completion.
|
|
994
263
|
|
|
995
264
|
## Commands
|
|
996
265
|
|
|
@@ -998,3 +267,5 @@ once. Collected by bare `bun test`, excluded from `bun run test`.
|
|
|
998
267
|
bun test packages/jobs
|
|
999
268
|
bun run --filter @ultimat3/jobs typecheck
|
|
1000
269
|
```
|
|
270
|
+
|
|
271
|
+
Why each rule above is shaped the way it is: [`docs/history/jobs.md`](../../docs/history/jobs.md).
|