@ultimat3/jobs 23.0.0 → 24.0.0

package/CLAUDE.md

@@ -8,7 +8,8 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   `expectedQueryLoop` ONLY: `steps.ts` declares the per-step write one-per-step to the N+1
   detector there, and no client is ever taken from it. Never `http`, `render`, `ui`.
 - Consumers: `action` (`<job>.enqueue`, via the ambient jobs facade), `cli`, `mcp`, `admin`.
-- External deps: none. Postgres access goes through the injected `PgExecutor`.
+- External deps: none. Postgres access goes through the injected `PgExecutor` (`@ultimat3/core`'s,
+  re-exported from `index.ts`).
 
 ## Rules — registration and declarations
 
@@ -29,7 +30,7 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 - **JOBS DECLARE THEIR TENANT, and `executeJob` INSTALLS the context — do not re-litigate.**
   `tenant` is REQUIRED (`(input) => input.orgId` or `'none'`; `X_JOB_TENANT_REQUIRED` backstop).
   `executeJob` puts the derived org on the ctx's actor (`jobRunActor`, org only) and runs the body in
-  `runWithContext`; `'none'` STRIPS the org (fail-closed). A boot-supplied service actor was rejected.
+  `runWithContext`; `'none'` STRIPS the org (fail-closed).
   `tenantFor` is a METHOD on `JobHandle` (variance). The queue row's `tenantId` stays the ENQUEUER's
   (the limiter's bucket). `tenancy-cross-surface.test.ts` asserts HTTP and job verdicts are EQUAL.
 - **`enqueuedBy` is ATTRIBUTION, never authority** — do not re-litigate. A job acting FOR a user takes
@@ -43,39 +44,37 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 
 ## Rules — the driver and SQL
 
-- **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_steps`,
-  `x_backfills`, `x_outbox`, `x_scheduler_state`, `x_scheduler_leader`, `x_job_leases`,
-  `x_job_events`, `x_job_pauses`, `x_job_workers`, `x_job_counters`). A NEW table is also a name in
+- **`SQL_JOBS_TABLE` is the ONE install point** for every durable table (`x_jobs`, `x_job_*`, `x_backfills`,
+  `x_outbox`, `x_scheduler_*`). A NEW table is also a name in
   `packages/cli/src/framework-schema.ts`. A shipped table grows by `alter table ... add column if
   not exists`. Its comments carry NO apostrophes and NO semicolons (`dev-queue.ts` splits on `;`; `driver-pg-sql.test.ts` checks
   quote parity).
 - **`claim({ queues: [] })` is REFUSED by every driver** (`assertClaimQueues`,
-  `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's `claim`/`list`/`deadLetters` are `async` so a
-  refusal rejects on both.
+  `X_JOB_CLAIM_QUEUES_EMPTY`); the memory driver's reads are `async` so a refusal rejects on both.
 - **`ack`, `nack`, `heartbeat` and `recordProgress` are FENCED on `state = 'running'` AND the
   CLAIM** — `{ workerId, claim }`, `claimOf(claimed)`; `x_jobs.claims` is moved by the claim and
-  never reset. A worker id alone let a worker's own earlier body settle its re-claim. `heartbeat`
-  answers a boolean, read as `held === false` (never `!held`).
-- **A driver's semantics are pinned in ONE test beside the pg statement** (`driver-parity.test.ts`):
-  list order, the `attempt` floor, `SQL_LEASE_RENEW` fencing on `expires_at > now()` and `holder`,
-  `SQL_STATS` one bucket per row.
+  never reset. `heartbeat` answers a boolean, read as `held === false` (never `!held`).
+- **A driver's semantics are pinned beside the pg statement** (`driver-parity*.test.ts`);
+  `driver-*-fixture.ts` scenarios run on memory AND Postgres.
+- **The claim BURIES a lease that lapsed on the final attempt** (`attempt >= max_attempts`): `dead`
+  in `SQL_CLAIM` — `failed` for a name in `ClaimOptions.dropExhausted` (`retry.deadLetter: false`)
+  — counted, never returned; the worker's round hears `onExhausted` and calls `announceExhausted`.
+- **`cancel` is fenced on `LIVE_STATES`.** **`requeue`** refuses (`X_JOB_NOT_FOUND`, not requeueable,
+  duplicate), then ONE statement. **`enqueue` inserts twice at most.** **`BulkResult.remaining` is
+  what a second call would move.**
 - **No read returns a WHOLE row** — `driver-pg-sql.test.ts` scans every production file here
   (discovered, comments stripped) for `select *`/`returning *`.
 - Drivers implement the six `JobDriver` methods plus optional `introspect`, `backfills`, `leases`. New
   capabilities go behind the interface. `inspect.ts` returns plain JSON for CLI, `/_x` and MCP.
-- Step results are persisted BEFORE the step returns. All time is epoch ms from an injected `Clock`
-  (`nowMs()` in `clock.ts`).
+- Step results are persisted BEFORE the step returns. All time is epoch ms (`nowMs()`, `clock.ts`).
 - **`createPgLeader` is correct only on a DEDICATED connection**; boot uses `createPgLeaseLeader`.
-- `driver-redis.ts` / `driver-nats.ts` are honest `X_NOT_IMPLEMENTED` stubs.
 
 ## Rules — numbers and limits
 
 - **Every numeric knob is refused when not FINITE** — core's `finiteOption()` (a bound) and
   `finiteCount()` (a count, caller's minimum); `worker-options.ts` is where `createWorker` reads them.
-  **`bun run finite-bounds` is a floor, never proof**: `createLimiter`'s five numbers (`perTenant`,
-  `perQueue`, `global`, `ratePerTenant.limit` / `.windowMs`) are screened with min 0 (zero is a HARD
-  STOP). **A row count is `finiteCount` (min 0)** for driver parity (`list`, `deadLetters`, the
-  backfill ledger's `list`, `assertClaimBounds`).
+  **`bun run finite-bounds` is a floor, never proof**: `createLimiter`'s five numbers are screened
+  with min 0 (zero is a HARD STOP). **A row count is `finiteCount` (min 0)**; `retry.attempts` is one with min 1.
 - **`WorkerOptions.concurrency` is read by OWN key** — a queue named `constructor`
   (`worker-slots.test.ts`).
 - **`job.concurrency` is enforced by `JobDriver.leases`** (one row per held slot in `x_job_leases`);
@@ -111,12 +110,11 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   letter), read off the SIGNAL. `WorkerStats.interrupted`; `worker-drain-signal.test.ts`.
 - **The drain waits out the claim round it races** — `tick()` registers its round in `rounds`
   synchronously with its guard.
-- **A fleet slot is taken INSIDE a `try`, released AWAITED, and a renewal answering `false` CANCELS the
-  run** (`X_JOB_SLOT_LOST`, read as `renewed !== false`).
+- **A fleet slot is taken INSIDE a `try`, released AWAITED, and HELD, not owned**: `false`, or a TTL
+  with no renewal landing, CANCELS the run (`X_JOB_SLOT_LOST`); held per job id as a LIST.
 - **The run's signal is a controller this worker owns** (`run-signal.ts`), never `AbortSignal.any`.
-- **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal is `jobs.heartbeat.failed`; a
-  whole window without one landing is `jobs.lease.lost` + `recordLeaseLost(queue)`, measured on this
-  process's clock, asked both sides of the call.
+- **A lease is HELD, not owned** (`heartbeat.ts`): one failed renewal warns; a whole window without
+  one landing is `jobs.lease.lost`, on this process's clock, asked both sides of the call.
 - **A renewal is decided against `stopped()`** (`renewal-timer.ts`, re-read after every await); the
   interval is `unref`ed; every renewal is armed through `WorkerOptions.schedule`.
 - **Settlement is not part of the retry decision**: `driver.ack` sits after the `try`.
@@ -125,15 +123,19 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 - **The claim loop re-arms on the PASS, never the jobs**; `tick()` resolves with that pass's executions.
 - **A deadline CANCELS, then fails the attempt**, never the reverse (`raceTimeout`, `withStepTimeout`);
   the attempt is cancelled in `executeJob`'s `finally`.
-- **A cancelled runner writes NOTHING** — `put()` in `steps.ts` is the one fence (`X_ABORTED`);
-  `jobs.timeout.abandoned` names an uncooperative body. `executeJob` reads `ctx.signal` defensively.
+- **A cancelled runner writes NOTHING** — `put()` in `steps.ts` (`X_ABORTED`), and the STORE fences
+  the claim (`StepFence`, `SQL_STEP_PUT` as `SQL_ACK`: `X_JOB_LEASE_LOST`). `executeJob` passes it.
 - **`traceparent` is stamped at ENQUEUE time, in `outbox.ts`**, and an empty `spanId` sends none.
 - **The scheduler runs one dispatch round at a time**; every other `tick()` joins it; `stop()` waits it
   out before `leader.release()`. Same two-hook rule; an ABANDONED round does not release
   (`jobs.scheduler.drain-abandoned`). `stillLeading()` is asked every round and before every task,
   and answers from memory inside `renewEveryMs`.
-- **`run-once` fires ONE catch-up** (marks `at`); `skip` fires the true latest missed occurrence
-  (`latestOccurrenceBy`, bisection).
+- **`run-once` fires ONE catch-up**: the watermark rides the fire (`ScheduledFire.watermarkMs`,
+  `SQL_SCHEDULER_FIRE` `$4`), never a `markFired` behind it; `skip` fires the true latest missed
+  occurrence (`latestOccurrenceBy`, bisection).
+- **A lease for a run that never STARTED is `abandon()`ed, never `release()`d** — the rate stamp
+  goes with it (`worker-admit.ts`). **A failed pass re-arms at the floor** (`worker-loop.ts`
+  catch); `timers` is the test seam.
 - **Every timer body catches before it finalises** (`void work().catch(log).finally(...)`).
 - **Suspension is control flow, and a SHED is not a suspension**: `StepSuspension` →
   `nack({ countsAsAttempt: false, park: true })`; a shed is `countsAsAttempt: false` without `park`,
@@ -166,10 +168,9 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
 - **An occurrence fires in ONE statement** (`SchedulerState.fire`, `SQL_SCHEDULER_FIRE`): the
   watermark is the fence, jobs insert only `where exists (select 1 from moved)`.
 - **`progress()` is throttled in `progress.ts`** and flushed before every settle. **`onSettled`
-  (`settled.ts`) is the ONE ending hook** — `completed` (with `result`), `dead-lettered`,
-  `dropped`, `refused` — AFTER a settle that LANDED, in the body's tenant scope,
-  `ON_SETTLED_ATTEMPTS` tries, never rethrown (`X_JOB_ON_SETTLED_FAILED`), at most once across a
-  crash. Never for a cancel, a retry or a suspension; no second hook beside it.
+  (`settled.ts`) is the ONE ending hook** — `completed`, `dead-lettered`, `dropped`, `refused` —
+  AFTER a settle that LANDED, in the body's tenant scope, `ON_SETTLED_ATTEMPTS` tries, never
+  rethrown, at most once across a crash. Never for a cancel, a retry or a suspension.
 - **Failed for good + `deadLetter: false` is DROPPED**: `nack({ fail })`, outcome `dropped`.
 - **Worker registry rows expire** (`worker-registry.ts`, TTL = visibility timeout); `stop()` forgets.
 
@@ -196,7 +197,9 @@ Long form: `docs/history/jobs.md`, "the cross-process wake".
 - **`idle-cost.test.ts` / `queue-wake.test.ts` count statements per idle minute** — scheduler 6,
   worker and relay 30 with no wake, 12 with. Raising one is a regression.
 - **The memory driver stores a payload's JSON form**, as pg binds it (`driver-settle-parity.test.ts`).
-- **The event bus has ONE clock**: `createPgEventBus` takes no `clock`; `bus.now()` is "asked at".
+- **The event bus has ONE clock**: `createPgEventBus` takes no `clock`; `EventLookup.now()` is
+  REQUIRED, stamps a NEW wait, and calls the TIMEOUT (asked once the runner thinks time is up).
+  `purgeExpired()` is awaited and counted; `eventsPurgeTarget(bus)` is its `PurgeTarget`.
 
 ## Rules — the outbox
 
@@ -238,7 +241,7 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
 | `tenant.ts` | what tenant a run acts under: `JobTenant`, the declaration's backstop, the actor `executeJob` installs |
 | `backfill.ts` | `backfill()` — a factory over `job()`: the declaration, its checksum and its input |
 | `backfill-pass.ts` | one pass: the batched iteration, its cursor checkpoints and its ledger row |
-| `backfill-scope.ts` | which sweeps run across tenants — `tenant: 'none'` only, on the pass's own actor, for the pass's own life |
+| `backfill-scope.ts` | which sweeps run across tenants (`tenant: 'none'` only) |
 | `backfill-ledger.ts` | `x_backfills` — the contract, `BACKFILL_STATUSES`, the checksum, the verdict, the memory ledger |
 | `backfill-registry.ts` | what was DECLARED: the `origin` stamp, `isBackfill`, `registeredBackfills` |
 | `backfill-gate.ts` | may this sweep run here and now — environment, `requires`, already-applied |
@@ -249,38 +252,39 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
 | `register.ts` | `registerJobs`/`registerTasks` over a module namespace + the registrar announcements; an `action-job` projection is `X_ACTION_JOB_UNBRIDGED` |
 | `describe.ts` | the JSON projection one handle emits; `describeJobs()` is a map over it |
 | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
+| `steps-timeout.ts` / `steps-suspension.ts` | a step's ceiling; `StepSuspension` and the timed-out marker |
 | `outbox.ts` | staging in a `Tx`, the store seam, the ambient `JobsFacade` slot |
 | `outbox-relay.ts` | the relay: the poll timer, one pass, and its TWO shutdown hooks |
 | `outbox-pg.ts` | `createPgOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
 | `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
-| `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one (drops a key with its last slot), `jobLeaseKey` |
+| `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
 | `concurrency.ts` | `job.concurrency` as declared, resolved once: `KeyedConcurrency`, `WhenBusy`, the declaration and key refusals |
-| `errors-concurrency.ts` | every concurrency refusal's class — declaration, key, `X_JOB_KEY_BUSY`, the boot refusal. Codes stay declared in `errors.ts` |
+| `errors-concurrency.ts` | every concurrency refusal's class; codes stay in `errors.ts` |
 | `worker-key-busy.ts` | one run refused by its key: settled `failed`, body never run |
 | `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
 | `scheduler-pg.ts` | `pgSchedulerState` (the durable watermark, the atomic fire) + `createPgLeaseLeader` |
 | `events-pg.ts` | `createPgEventBus` — `step.waitForEvent` across processes |
 | `driver.ts` | `JobDriver` contract + wire records |
-| `driver-pg.ts` | default driver, and `createPgLeader` — the advisory-lock election a scheduler does NOT use |
+| `driver-pg.ts` | default driver, and `createPgLeader` |
 | `driver-pg-ddl.ts` | `SQL_JOBS_TABLE` — the ONE install point. Its comments carry no `;` and no `'` |
 | `driver-pg-jobs-sql.ts` | every statement returning a whole `x_jobs` row, and `JOB_ROW_COLUMNS`; re-exported from `driver-pg-sql.ts` |
 | `driver-pg-rows.ts` | a Postgres row → a wire record: `JobRow`/`StepRow`/`BackfillRow` and their mappings |
-| `driver-memory.ts` | `x dev` / tests |
+| `driver-memory.ts` / `claim-exhausted.ts` | `x dev` / tests; a buried row's log line and `onSettled` |
 | `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
 | `retry.ts` | the dead-letter decision, and this package's option names over core's `backoffDelay` — no curve of its own |
-| `retry-classification.ts` | the OTHER half of that decision: what the thrown error says, and the stop reason the row and the log carry |
+| `retry-classification.ts` | the other half: what the thrown error says, and the stop reason |
 | `execute.ts` / `run-deadline.ts` | `executeJob` — one claimed job run and settled; the run's deadline: cancel, then fail |
 | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
-| `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
-| `worker.ts` | `worker` role, claim loop, drain — and the one `AbortController` SIGTERM reaches every held run through |
+| `renewal-timer.ts` | the renewal interval and its `stopped()` latch |
+| `worker.ts` | `worker` role, claim round, drain |
 | `worker-admit.ts` | may this claimed job start: the limiter, the fleet slot, and the shed / refusal |
 | `worker-registry.ts` | one worker's registry row: announce, heartbeat, forget |
 | `worker-types.ts` | the worker's public contract: `WorkerOptions`, `WorkerStats`, `Worker` |
-| `drain-wait.ts` | the drain's wait, shared by both roles: settled, or abandoned at the `close` hook's budget |
-| `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span — handed back in one `finally` |
-| `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |
+| `drain-wait.ts` | the drain's wait, shared by both roles |
+| `worker-run.ts` | one claimed job, wired: heartbeat, slot renewal, run signal, span |
+| `run-signal.ts` | the signal ONE run is cancelled by |
 | `worker-fleet-slots.ts` | the fleet slot an in-flight job holds — take, renew, hand back — and the boot refusal |
-| `purge.ts` | `purge()` — a factory over `job()`: the retention sweep, its structural target seam and the hourly cron a host schedules it on |
+| `purge.ts` | `purge()` — a factory over `job()`: the retention sweep and its target seam |
 | `task.ts` | the `task()` primitive + registry + the handle's surface + `registerTask` |
 | `scheduler.ts` | `scheduler` role: the dispatch round, catch-up, leader election, the drain |
 | `limits.ts` | per-tenant / per-queue / global concurrency + rate |
@@ -298,16 +302,11 @@ Long form: [`docs/history/jobs.md`](../../docs/history/jobs.md), "Moved 2026-10-
 | `queue-wake.ts` / `driver-pg-wake-sql.ts` | the `LISTEN` session and its proof; which statement notifies |
 | `worker-loop.ts` | the claim loop's timer: wake, `again`, the kicks |
 | `driver-pg-outbox-sql.ts` | the `x_outbox` statements |
-| `errors-operator.ts` | `X_JOB_NOT_REMOVABLE`, `X_JOB_NOT_PROMOTABLE`, `X_JOB_ON_SETTLED_FAILED` classes |
+| `errors-operator.ts` / `errors-requeue.ts` | the operator verbs' refusals; `requeue`'s |
 
 Which `-fixture.ts` harness each suite shares, and what each `.job.` suite proves:
 `docs/history/jobs.md`, "Moved 2026-10-01 — test harnesses".
 
-## Commands
-
-```
-bun test packages/jobs
-bun run --filter @ultimat3/jobs typecheck
-```
+Commands: `bun test packages/jobs` · `bun run --filter @ultimat3/jobs typecheck`.
 
 Why each rule above is shaped the way it is: [`docs/history/jobs.md`](../../docs/history/jobs.md).

package/README.md

@@ -196,7 +196,13 @@ The event bus has ONE clock. `createPgEventBus({ executor })` stamps `publishedA
 with the database's `now()` — it takes no `clock` — and `bus.now()` answers that same clock. A
 consumer that wants "published after I asked" (`@ultimat3/scraping`'s `eventPrompt`) reads its
 "asked at" from `bus.now()`, never from its own process: a skew between two pods must not decide
-which answers count.
+which answers count. `step.waitForEvent` stamps a new wait from the same clock
+(`EventLookup.now()`), so a worker running ahead of the database still matches an event published
+after it asked.
+`step.waitForEvent` stamps a new wait from that clock (`EventLookup.now()`, required) and asks it
+again before declaring a timeout, so a worker running ahead of the database still matches an event
+published after it asked and does not give up early. `eventsPurgeTarget(bus)` is the bus as a
+`purge()` target: `purgeExpired()` is awaited and answers the rows it removed.
 
 Step names are the replay key, so they must be deterministic and unique in a run — a
 duplicate is `X_STEP_DUPLICATE`, not a silent overwrite. Suspension is control flow
@@ -405,6 +411,11 @@ export const hourly = task({
 | at least once is safe here | a replayed delete removes rows that are already gone, and a row a purge deleted answers exactly as one that was never there |
 | two targets under one name | `X_INVARIANT`, before the first delete — `step.run` would raise `X_STEP_DUPLICATE` after one table was already empty |
 
+The stored event bus is a target too: `eventsPurgeTarget(bus)` answers
+`{ name: EVENTS_PURGE_TARGET, purgeExpired }` over `bus.purgeExpired()`, which is awaited and
+answers the rows it removed — `x_job_events` only grew while nothing called it. Add it to
+`targets`: `targets: () => [eventsPurgeTarget(eventBus())]`.
+
 `@ultimat3/cli`'s boot declares both halves over the three tables it owns, so an app gets the sweep
 without writing any of the above. It needs a `worker` to run it and a `scheduler` to fire it: a
 deployment with neither has no background work at all, and this is one more thing it does not do.
@@ -731,6 +742,20 @@ between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driv
 The pg SQL is exported verbatim (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …) so an agent
 debugging a stuck queue can read and run the exact statement.
 
+**A lease that lapses on a row's final attempt is buried by the claim.** `claim` re-takes a
+`running` row whose lease lapsed — the worker died — but one already at `attempt >= maxAttempts`
+is settled `dead` in the claim statement itself (`SQL_CLAIM`'s second arm; `isFinalAttempt` on the
+memory driver), with `lastError` set to `LEASE_LAPSED_FINAL_ATTEMPT`, counted `dead`, and never
+handed out: a job that kills its worker reaches the dead-letter queue after `retry.attempts`
+claims instead of taking a worker per visibility timeout, forever. The pass is told through
+`ClaimOptions.onExhausted(rows)` — at most once per pass, once per row — and `announceExhausted`
+(`claim-exhausted.ts`) is what a claim round does with them: `jobs.claim.exhausted` at `error`, and
+the job's `onSettled`, and the worker's claim round does exactly that. The row carries
+`maxAttempts` and not the policy, so the CALLER names the jobs declaring `retry.deadLetter: false`
+(`ClaimOptions.dropExhausted` — the worker reads them off its registry): those rows are buried
+`failed` and announced `dropped`, every other one `dead` / `dead-lettered`. A caller that names
+none (`x jobs drain`) buries every such row `dead`.
+
 A driver you write reads a nack's target state through `nackState(options)` — `dead`
 (`deadLetter`), `failed` (`fail`, terminal and out of the dead-letter queue), `suspended` (`park`)
 or `ready` — so a new branch cannot land in one driver and not another. Its lease store implements
@@ -1071,7 +1096,9 @@ the pg and the memory driver alike. `x jobs`, `/_x` and a dashboard read nothing
 |---|---|---|
 | `list(filter)` | one page, newest first; `filter.after = jobCursor(lastRow)` for the next, `filter.before = jobCursor(firstRow)` for the previous (the rows nearest the cursor, still newest first; never both). Filters: `queue`, `name`, `state`, `tenantId`, `idPrefix`, `createdFrom`, `createdTo` | `MAX_JOB_PAGE` = 200 rows; `DEFAULT_JOB_PAGE` = 100. A page shorter than its limit is the last. A larger `limit`, or a cursor no page produced, is `X_JOB_PAGE_INVALID` |
 | `remove(id)` | delete one job and its steps. A running one is `X_JOB_NOT_REMOVABLE` | — |
-| `requeueMany({ state, queue?, name?, tenantId? })` | re-queue finished rows; a row whose key a live job holds stays | `MAX_BULK_ROWS` = 1,000 per call; answers `{ affected, remaining }` |
+| `requeue(id, { fromStep? })` | re-queue ONE finished job (`REQUEUEABLE_STATES`). A live one is `X_JOB_NOT_REQUEUEABLE`, a key a live job holds `X_JOB_DUPLICATE`, an id nobody queued `X_JOB_NOT_FOUND` — on both drivers. `fromStep` drops that step and every later one in the statement that moves the row | — |
+| `cancel(id, reason?)` | stop a LIVE job (`LIVE_STATES`: ready, delayed, running, suspended). A done, failed, dead or cancelled row is left exactly as it ended and answers `undefined` — `cancelJob` turns that into `X_JOB_NOT_CANCELLABLE` | — |
+| `requeueMany({ state, queue?, name?, tenantId? })` | re-queue finished rows; a row whose key a live job holds stays, and of several finished rows sharing one key the oldest is requeued | `MAX_BULK_ROWS` = 1,000 per call, spent only on rows it can move; answers `{ affected, remaining }`, where `remaining` is what a second call could still move — a held row is not counted, so "call until zero" ends |
 | `removeMany({ state, queue?, name?, tenantId? })` | delete rows in any state but `running` | the same |
 | `promoteMany({ state, queue?, name?, tenantId? })` | `promote` over a set: every row still waiting on its `runAt` is due now. `state` is `delayed` or `ready` (`PROMOTABLE_STATES`) | the same; `remaining` is the rows still waiting |
 | `promote(id)` | a job waiting on its `runAt` — delayed, or backing off — is due now | — |
@@ -1133,6 +1160,7 @@ export const syncAccountWatched = job({
 |---|---|---|---|
 | the body returned and the ack landed | `completed` | `input`, `result` — what `run` returned, typed from it | `done` |
 | failed for good, parked in the dead-letter queue | `dead-lettered` | `input` (or `undefined`: the stored payload no longer parses), `error`, `code` | `dead` |
+| its lease lapsed on the final attempt and the claim buried it — every worker that took it died | `dead-lettered` | `input`, `error` (`LEASE_LAPSED_FINAL_ATTEMPT`), `code: undefined` — nothing was thrown | `dead` |
 | failed for good on `retry.deadLetter: false` | `dropped` | the same | `failed` |
 | `whenBusy: 'fail'` over a busy key — the body never ran | `refused` | the same, `code: 'X_JOB_KEY_BUSY'` | `failed` |
 
@@ -1198,13 +1226,14 @@ FOR a user takes that user's id in its input and re-authorises it in the body.
 | `X_ABORTED` | a cancelled attempt tried to write a step — core's code, not a second name for it |
 | `X_JOB_LEASE_LOST` | the job was cancelled, or its lease lapsed and the queue re-delivered it, while this worker was still running it |
 | `X_JOB_SLOT_LOST` | the fleet `concurrency` slot this run held was taken by another worker — a different row on a different clock from the lease above |
-| `X_JOB_NOT_CANCELLABLE` | `cancelJob` reached a job that already finished, or a driver with no `cancel` |
+| `X_JOB_NOT_CANCELLABLE` | `cancelJob` reached a job that is not live — done, failed, dead or already cancelled — an id no queue holds, or a driver with no `cancel` |
+| `X_JOB_NOT_FOUND` | `requeue` was asked for an id the queue does not hold — one answer on both drivers. Fix: `x jobs ls --json` |
 | `X_JOB_CONCURRENCY_UNENFORCEABLE` | a registered job declares `concurrency`, plain or keyed, and the driver has no lease store |
 | `X_JOB_KEY_BUSY` | a run claimed under `whenBusy: 'fail'` while its key already held `limit` runs. Classified `terminal`. Fix: `x jobs ls --name <job> --state running --json` |
 | `X_JOB_DECLARATION_INVALID` | `job()` is missing a required field, declares a `concurrency` no worker can honour (plain or keyed), or its `concurrency.key` answered an empty, non-string or over-long key at enqueue |
 | `X_JOB_NOT_REMOVABLE` | `remove` / `removeMany` reached a running job. Fix: `x jobs cancel <id> --json` |
 | `X_JOB_NOT_PROMOTABLE` | `promote` reached a job not waiting on its run time |
-| `X_JOB_PAGE_INVALID` | `list()` was asked for more than `MAX_JOB_PAGE` rows, or handed a cursor no page produced. Fix: `list({ limit: 200, after: jobCursor(lastRow) })` |
+| `X_JOB_PAGE_INVALID` | `list()` was asked for more than `MAX_JOB_PAGE` rows, or handed a cursor no page produced. The cause names which cursor (`after` / `before`). Fix: `x jobs ls --limit 200 --json` |
 | `X_JOB_ON_SETTLED_FAILED` | a declared `onSettled` threw on every one of its tries; logged and reported, never thrown |
 | `X_NOT_IMPLEMENTED` | redis / nats driver |
 
@@ -1240,6 +1269,7 @@ a job boundary the class is gone and the `code` is what survives — match on th
 | `JobMaxAttemptsError` | `X_JOB_MAX_ATTEMPTS` | `src/errors.ts` |
 | `JobNameTakenError` | `X_JOB_DUPLICATE` | `src/errors.ts` |
 | `JobNotCancellableError` | `X_JOB_NOT_CANCELLABLE` | `src/errors.ts` |
+| `JobNotFoundError` | `X_JOB_NOT_FOUND` | `src/errors-requeue.ts` |
 | `JobNotPromotableError` | `X_JOB_NOT_PROMOTABLE` | `src/errors-operator.ts` |
 | `JobNotRemovableError` | `X_JOB_NOT_REMOVABLE` | `src/errors-operator.ts` |
 | `JobNotRequeueableError` | `X_JOB_NOT_REQUEUEABLE` | `src/errors-requeue.ts` |
@@ -1247,7 +1277,6 @@ a job boundary the class is gone and the `code` is what survives — match on th
 | `JobPageInvalidError` | `X_JOB_PAGE_INVALID` | `src/errors-operator.ts` |
 | `JobRowStatusUnknownError` | `X_JOB_ROW_STATUS_UNKNOWN` | `src/errors.ts` |
 | `JobSlotLostError` | `X_JOB_SLOT_LOST` | `src/errors.ts` |
-| `JobsNotImplementedError` | `X_NOT_IMPLEMENTED` | `src/errors.ts` |
 | `JobTenantRequiredError` | `X_JOB_TENANT_REQUIRED` | `src/errors.ts` |
 | `JobTimeoutError` | `X_JOB_TIMEOUT` | `src/errors.ts` |
 | `LeaseLostError` | `X_JOB_LEASE_LOST` | `src/errors.ts` |

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "23.0.0",
+  "version": "24.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,10 +32,10 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "23.0.0",
-    "@ultimat3/db": "23.0.0",
-    "@ultimat3/entity": "23.0.0",
-    "@ultimat3/schema": "23.0.0",
-    "@ultimat3/time": "23.0.0"
+    "@ultimat3/core": "24.0.0",
+    "@ultimat3/db": "24.0.0",
+    "@ultimat3/entity": "24.0.0",
+    "@ultimat3/schema": "24.0.0",
+    "@ultimat3/time": "24.0.0"
   }
 }

package/src/backfill-ledger.ts

@@ -10,7 +10,7 @@
 // become — the checkpoints are transactional with the work, and this row is not.
 
 import type { Clock } from '@ultimat3/core';
-import { finiteCount, systemClock } from '@ultimat3/core';
+import { describeValue, finiteCount, systemClock, UltimateError } from '@ultimat3/core';
 import { nowMs } from './clock';
 
 /**
@@ -56,6 +56,27 @@ export interface BackfillFilter {
   readonly limit?: number | undefined;
 }
 
+/** What Postgres prints for a `uuid` — the one spelling both ledgers store identically. */
+const RUN_ID_SHAPE = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/;
+
+/**
+ * `BackfillFilter.runId`, refused by BOTH ledgers' `list` when it is not a uuid. `run_id` is a
+ * `uuid` column: Postgres raised a raw `22P02` where the memory ledger answered an empty page, so
+ * a mistyped id read as "no such pass" under `x dev` and as a database fault in production. The
+ * shape `outbox.ts` holds an enqueue's `runId` to — lowercase, so the two never disagree on case.
+ */
+export function listedRunId(subject: string, runId: unknown): string | undefined {
+  if (runId === undefined) return undefined;
+  if (typeof runId === 'string' && RUN_ID_SHAPE.test(runId)) return runId;
+  throw new UltimateError({
+    code: 'X_ID_INVALID',
+    // `describeValue`, never the value: what arrives here wrong is as often a key as a typo.
+    cause: `${subject} backfills.list was filtered by runId ${describeValue(runId)}, and a run id is a lowercase uuid (8-4-4-4-12 hex) — x_backfills.run_id is a uuid column`,
+    fix: 'x db backfill --list --json',
+    meta: { option: 'runId' },
+  });
+}
+
 /**
  * Durable, and therefore behind the queue driver: `x_backfills` ships in the same DDL as `x_jobs`
  * and `x_job_steps` (`driver-pg-sql.ts`), so a ledger a pass cannot write is a queue it could not
@@ -169,6 +190,7 @@ export function createMemoryBackfillLedger(clock: Clock = systemClock): Backfill
     },
     // `async`, so a refused limit REJECTS here as it does on the pg ledger — see `driver-memory.ts`.
     async list(filter = {}) {
+      const runId = listedRunId('the memory ledger', filter.runId);
       // Reversed BEFORE the sort: the test clock is frozen, so two rows share a `startedAt` and a
       // stable sort would hand back the oldest of them first under a "newest first" contract.
       const rows = [...runs.values()]
@@ -176,7 +198,7 @@ export function createMemoryBackfillLedger(clock: Clock = systemClock): Backfill
         .sort((a, b) => b.startedAt - a.startedAt)
         .filter((run) => filter.name === undefined || run.name === filter.name)
         .filter((run) => filter.status === undefined || run.status === filter.status)
-        .filter((run) => filter.runId === undefined || run.runId === filter.runId)
+        .filter((run) => runId === undefined || run.runId === runId)
         .slice(0, finiteCount('the backfill ledger list', 'limit', filter.limit ?? 100));
       return rows;
     },

package/src/backfill-pass.ts

@@ -164,7 +164,13 @@ export async function backfillPass<Row>(
   // refusing it — the same degradation `introspect` already has.
   const ledger = jobDriver()?.backfills;
 
-  const previous = await completedRun(ledger, name);
+  const completed = await completedRun(ledger, name);
+  // Its OWN row is not a previous pass: a worker that died between `ledger.finish` and the ack is
+  // handed the same run again, and skipping there reported `{ skipped: true, previousRunId:
+  // <itself> }` for a pass that swept. It falls through and replays — every batch is served from
+  // its checkpoint, nothing is read or handled twice, and the report is the one it first made.
+  const redelivered = completed?.runId === runId;
+  const previous = redelivered ? undefined : completed;
   const verdict = decideBackfill(previous, checksum, args.input.force === true);
   if (!verdict.run) {
     if (verdict.changed) {
@@ -196,7 +202,14 @@ export async function backfillPass<Row>(
     definition.tenant,
     args.ctx,
     async (ctx): Promise<BackfillReport> => {
-      await ledger?.start({ runId, name, checksum, appVersion: appVersion() });
+      // A redelivered pass leaves its completed row ALONE until a batch body actually runs (an
+      // operator's `retryFromStep` re-opens one): replaying checkpoints sweeps nothing, so there is
+      // nothing to record and no convergence to re-ask — rows that started matching since would
+      // otherwise turn a finished pass into a failed one.
+      let recording = !redelivered;
+      const open = (): Promise<void> | undefined =>
+        ledger?.start({ runId, name, checksum, appVersion: appVersion() });
+      if (recording) await open();
 
       let cursor: string | null = null;
       let rows = 0;
@@ -242,6 +255,10 @@ export async function backfillPass<Row>(
               const iteration = await iterate();
               const next = await iteration.pull.next();
               if (next.done === true) return { cursor: null, rows: 0 };
+              if (!recording) {
+                recording = true;
+                await open();
+              }
               await definition.handle({ rows: next.value, ctx, signal, index });
               swept = true;
               return { cursor: iteration.batches.cursor, rows: next.value.length };
@@ -271,10 +288,10 @@ export async function backfillPass<Row>(
         // this converge", asked once, where the answer is finally decidable. Inside the `try` so the
         // ledger records the attempt as `failed`: a pass that left rows behind must not write the
         // completed row that stops the next deploy re-running it.
-        await assertConverged(definition, ctx, rows);
+        if (recording) await assertConverged(definition, ctx, rows);
       } catch (error) {
         // Control flow, not a failure: a suspended run is parked and will be back on this step.
-        if (!isStepSuspension(error)) await markFailed(ledger, runId, rows);
+        if (!isStepSuspension(error) && recording) await markFailed(ledger, runId, rows);
         throw error;
       } finally {
         // Whatever the iteration holds belongs to this attempt, and an attempt that failed, was
@@ -282,7 +299,7 @@ export async function backfillPass<Row>(
         await live?.batches.close();
       }
 
-      await ledger?.finish(runId, { status: 'completed', rows });
+      if (recording) await ledger?.finish(runId, { status: 'completed', rows });
       return { name, batches, rows, skipped: false };
     },
   );

package/src/claim-exhausted.ts

@@ -0,0 +1,76 @@
+// The claim round's half of a burial: a row the claim settled `dead` (`ClaimOptions.onExhausted`)
+// has no body to end it and no settle to announce it, so the pass that buried it says so — one log
+// line an operator can alert on, and the job's `onSettled`, exactly as a dead letter a worker
+// settled itself would get.
+
+import type { Ctx } from '@ultimat3/core';
+import { logger, renderThrowable } from '@ultimat3/core';
+import type { JobRecord } from './driver';
+import { getJob } from './job';
+import { announceSettled } from './settled';
+
+export interface AnnounceExhaustedOptions {
+  /** What `onExhausted` handed this pass — every row the claim buried, as it now stands. */
+  readonly exhausted: readonly JobRecord[];
+  readonly workerId: string;
+  /** The worker's own context: what the job's `onSettled` is scoped from, as a body would be. */
+  readonly context: () => Ctx;
+}
+
+/** What one pass buried, by how each row ended — what the worker's own counters take. */
+export interface ExhaustedCounts {
+  readonly deadLettered: number;
+  readonly dropped: number;
+}
+
+/**
+ * Announce every row one claim pass buried, and answer how each ended. Once per row, because the
+ * claim buries a row once: the statement that settled it is the only one that hands it to
+ * `onExhausted`. A row buried `failed` is a job declaring `retry.deadLetter: false`
+ * (`ClaimOptions.dropExhausted`) — `dropped`, as its own nack would have ended it. Never rejects:
+ * `announceSettled` owns a hook's failure, and a `context()` that throws is logged.
+ */
+export async function announceExhausted(
+  options: AnnounceExhaustedOptions,
+): Promise<ExhaustedCounts> {
+  let dropped = 0;
+  for (const record of options.exhausted) {
+    const error = record.lastError ?? '';
+    const outcome = record.state === 'failed' ? 'dropped' : 'dead-lettered';
+    if (outcome === 'dropped') dropped += 1;
+    // `error`, not `warn`: nobody recovered from this one, and a job that outlives every worker
+    // that takes it is a crash loop somebody has to look at.
+    logger.error('jobs.claim.exhausted', {
+      workerId: options.workerId,
+      job: record.name,
+      jobId: record.id,
+      attempt: record.attempt,
+      maxAttempts: record.maxAttempts,
+      outcome,
+      error,
+    });
+    const handle = getJob(record.name);
+    if (handle === undefined) continue;
+    let ctx: Ctx;
+    try {
+      ctx = options.context();
+    } catch (raised) {
+      // The app's own function, asked AFTER the row was settled: thrown, it failed the round.
+      logger.error('jobs.claim.exhausted-unannounced', {
+        workerId: options.workerId,
+        job: record.name,
+        jobId: record.id,
+        error: renderThrowable(raised),
+      });
+      continue;
+    }
+    await announceSettled({
+      handle,
+      claimed: record,
+      ctx,
+      // No code: nothing was thrown, and `JobFailed.code` is never invented.
+      settlement: { outcome, error, code: undefined },
+    });
+  }
+  return { deadLettered: options.exhausted.length - dropped, dropped };
+}

package/src/driver-memory-operator.ts

@@ -134,7 +134,7 @@ export function createMemoryOperator(state: MemoryQueueState): MemoryOperator {
       const before =
         filter.before === undefined
           ? undefined
-          : parseJobCursor(filter.before, 'the memory driver list');
+          : parseJobCursor(filter.before, 'the memory driver list', 'before');
       const { idPrefix, createdFrom, createdTo, tenantId } = filter;
       const matching = [...jobs.values()]
         .filter((record) => filter.queue === undefined || record.queue === filter.queue)
@@ -168,16 +168,38 @@ export function createMemoryOperator(state: MemoryQueueState): MemoryOperator {
         `requeueMany was asked for ${filter.state} jobs, and only a finished job can be requeued`,
         "pass state: 'dead', 'failed', 'cancelled' or 'done' to requeueMany",
       );
-      const result = await bulk(filter, (record) => {
-        // A key a live job holds stays where it is — the single requeue's `X_JOB_DUPLICATE`,
-        // answered as "not this one" so one conflict does not fail a thousand rows.
-        if (state.liveHolder(record) !== undefined) return false;
-        state.settle(record.id, { state: 'ready', attempt: 0, runAt: nowMs(clock) });
-        return true;
-      });
+      // A key a live job holds stays where it is — the single requeue's `X_JOB_DUPLICATE`,
+      // answered as "not this one" so one conflict does not fail a thousand rows.
+      const free = (record: JobRecord): boolean => state.liveHolder(record) === undefined;
+      // `SQL_JOB_REQUEUE_MANY`'s candidates: no live holder, and ONE row per idempotency namespace
+      // — the oldest — both decided BEFORE the bound. Decided inside `apply`, a held row spent a
+      // slot of `MAX_BULK_ROWS` and was then skipped: a thousand held rows ahead of one free row
+      // answered `affected: 0` here and `affected: 1` on Postgres.
+      const oldest = new Map<string, JobRecord>();
+      for (const record of jobs.values()) {
+        if (!inBulk(record, filter) || !free(record)) continue;
+        const key = JSON.stringify([record.name, record.tenantId ?? '', record.idempotencyKey]);
+        const seen = oldest.get(key);
+        if (seen === undefined || newerThan(seen, record)) oldest.set(key, record);
+      }
+      const candidates = new Set([...oldest.values()].map((record) => record.id));
+      const result = await bulk(
+        filter,
+        (record) => {
+          if (!free(record)) return false;
+          state.settle(record.id, { state: 'ready', attempt: 0, runAt: nowMs(clock) });
+          return true;
+        },
+        (record) => candidates.has(record.id),
+      );
       // The pg driver's `SQL_WAKE`, in-process: an operator's repair starts now.
       if (result.affected > 0) signalEnqueued(filter.queue);
-      return result;
+      // What a second call could still move: rows of the filter whose key no live job holds. A
+      // row requeued just now holds its key, so its dead siblings left with it.
+      const remaining = [...jobs.values()].filter(
+        (record) => inBulk(record, filter) && free(record),
+      ).length;
+      return { affected: result.affected, remaining };
     },
 
     async promoteMany(filter) {