@ultimat3/jobs 2.0.0 → 3.0.0

package/CLAUDE.md

@@ -237,6 +237,30 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   so, not a job that failed: nacking it re-runs completed work and records `retried` in
   `jobs_total{outcome}` for a failure that never happened. It propagates instead, and the worker's
   `jobs.worker.settle-failed` plus the lapsing lease are the honest answer.
+- **The retry decision reads the ERROR as well as the attempt count — added 2026-08.**
+  `executeJob` decided a retry from `nextRetry(handle.retry, attempt)` alone, so every `terminal`
+  classification in the framework was decorative on the job path: an `X_SCRAPE_AUTH_FAILED` from a
+  rotated password burned the whole policy, which at a site that locks an account after three wrong
+  passwords makes the framework's retry the thing that destroys the account.
+  `retry-classification.ts` composes AROUND `nextRetry` — the backoff arithmetic stays in one
+  place — and `nextRetryForError` is the only caller `execute.ts` has.
+
+  **`classifyThrown` must never read `error.retry` on its own.** That field is
+  `init.retry ?? retryFor(code)` and `retryFor` FAILS CLOSED, so every unclassified `UltimateError`
+  already carries `terminal`; reading it would dead-letter the first attempt of every job in every
+  app whose codes nobody has classified. Hence core's `declaredErrorRetry(code)`, which answers
+  `undefined` where `retryFor` answers the default — and hence the one case that is knowingly
+  under-read: an instance `retry: 'terminal'` on an UNREGISTERED code is indistinguishable from the
+  default and is treated as unclassified. Register the code; that is the one way.
+  `retry-after` reuses the delay the nack already takes (`meta.retryAfterSeconds`, clamped by the
+  policy's `maxDelay`) rather than a second suspension mechanism — `StepSuspension` stays the only
+  way to park a run, and unlike a suspension a retry-after DOES burn an attempt, because the work
+  failed. The ceiling outranks every classification but `terminal`.
+
+  **The verdict is published, not inferred**: `jobs.attempt.failed` and `reportError` carry
+  `stop`, `JobExecution` carries `stopReason`, and `recordedFailure` appends the terminal verdict to
+  the nack's `error` — `lastError` is the ONE failure field a row has, so without it `x jobs show`
+  renders a dead letter at attempt 1 of 5 as a silent early stop.
 - **The claim loop re-arms on the PASS, never on the jobs.** A slot belongs to its own job and is
   free the moment it settles, so `claimRound` starts what it claimed and returns the promises —
   ending the pass on `Promise.allSettled([...inFlight])` made the pool as slow as its slowest
@@ -401,8 +425,74 @@ Tier 3. The `job` + `task` primitives, durable steps, transactional outbox, queu
   A SIGTERM landing between `driver.enqueue` and `markPublished` returned to a caller that then
   closed the database under the row it was about to mark. `OutboxRelay.stop(): Promise<void>` — a
   caller that does not await gets what it always got (the chain carries its own `catch`), so the
-  join is only as good as the `await`: **`packages/cli/src/dev-roles.ts` still calls it without
-  one**, in both teardown paths.
+  join is only as good as the `await`. Both of `packages/cli/src/dev-roles.ts`'s paths take it
+  today (`:306` returns `() => relay.stop()` so the rollback awaits it, `:356` is
+  `await relay?.stop()`); this file claimed the opposite until 2026-08.
+- **The outbox claim is a LEASE, and one statement is what makes it one** (`As of 2026-08`).
+  `SQL_OUTBOX_CLAIM` was a bare `select ... for update skip locked` run on the POOLED executor, and
+  those row locks last only for their own statement — under autocommit they are gone before
+  `claim()` resolves, and `x_outbox` had no claimed column, so nothing fenced the batch at all. Two
+  relays 200ms apart read the identical rows and both published them. **The idempotency key does
+  not collapse that in general**: `SQL_ENQUEUE`'s conflict target is the PARTIAL index over
+  `('ready','delayed','running','suspended')`, so a repeat landing after the first job reached a
+  terminal state inserts a second row and the handler runs twice. The mechanism is Postgres
+  semantics; the second-publish-after-terminal ordering was argued, never reproduced — say it that
+  way, in a comment or a changelog. The claim now stamps `claimed_at`/`claimed_by` in the same
+  statement that locks (the CTE shape `SQL_CLAIM` already used), and the outer
+  `select ... order by staged_at, id` is load-bearing: `update ... returning` has no defined row
+  order and the relay publishes in the order it is handed rows. `claimed_at` is a LEASE and not a
+  flag — without a reclaim window a relay that died mid-batch strands its rows forever — and
+  `OutboxStore.release` (optional, so a store written before this still compiles) hands back the
+  batch a failed publish stopped, or one pool blip would park committed work for a whole lease
+  window instead of one poll interval. `createMemoryOutboxStore` answers the SAME question, on an
+  injected clock, and `outbox-claim.test.ts` pins the two side by side.
+- **The lease is fenced on EVERY outbox mutation, and the sort key is TOTAL** (`As of 2026-08`).
+  Two holes the first version of the lease left open, both reachable without a second relay
+  process. `SQL_OUTBOX_RELEASE` and `SQL_OUTBOX_MARK_PUBLISHED` matched on `id` alone, so a relay
+  that stalled past its own lease still spoke for rows another relay had reclaimed: its late
+  `release` unclaimed a batch mid-publish (a third relay claims it, publishes it again — the
+  duplicate the lease exists to prevent, reached the long way round) and its late `markPublished`
+  retired a row nobody had published, losing the job with nothing to notice. Both now carry
+  `and claimed_by = $n`, `claim()` hands the token back as `OutboxRecord.claimedBy`, and the relay
+  passes it to both calls. `markPublished` also gained `published_at is null`, so the stamp is
+  first-writer-wins rather than a rewrite of an audit timestamp. The memory store fences the same
+  way — per CLAIM there rather than per relay, because two relays there are two `claim()` calls on
+  ONE store, and a per-store id could not tell them apart. `undefined` is unfenced in both, for the
+  reason `release` is optional: a caller holding no token is one written before the fence.
+  **`order by staged_at` was not a total order**: every row staged in one transaction shares a
+  `staged_at`, so the tie was the planner's to break — which rows the `limit` takes, and in which
+  order they publish, differed between two relays and between two runs of one. `, id` fixes it in
+  the CTE and in the projection, and needed NO DDL: `id` is a UUIDv7 minted by `uuid()`, monotonic
+  and already the primary key, so the tiebreak IS stage order. `byClaimOrder` in `outbox.ts` is the
+  memory store's copy of that key.
+- **`claimLeaseMs` is normalised in ONE place — `outbox-lease.ts`** (`As of 2026-08`). Both stores
+  call `resolveClaimLeaseMs`, which owns `DEFAULT_OUTBOX_CLAIM_LEASE_MS` and refuses anything that
+  is not a positive whole number of ms with `X_INVARIANT` (the generic, no new code: same borrow
+  `@ultimat3/db` makes). A memory default and a pg default that could drift are two answers to
+  "how long is a claim mine for", and the shorter one publishes a row twice. `0` expires before
+  `claim()` resolves and `Infinity` never expires, so both are refused at CONSTRUCTION, not at the
+  first tick where the only trace is a log line.
+- **A renewal is decided against `stopped()`, not only against the interval** (`As of 2026-08`).
+  `renewal-timer.ts` is the one shape, read by `heartbeat.ts` and `worker-fleet-slots.ts`, and it
+  exists because both files reported a LOSS for a job that had finished cleanly: `stop()` cleared
+  the interval, which does nothing to the request already on the wire, so the fenced statement came
+  back `false` — the row left `running` when `executeJob` acked it — and that answer took the loss
+  branch. `jobs.lease.lost` at error plus `recordLeaseLost(queue)` is the one signal meaning the
+  queue re-delivered a job this process was still running, so a false one is a page for a
+  non-event, and the window widens exactly when the pool is slow. Re-read the flag AFTER every
+  await and inside the reporter, the way `settleWithin`'s `decided` does in core.
+- **`stepTimeout` and `eventPoll` are DECLARED on the job, and `execute.ts` is the only place they
+  are forwarded** (`As of 2026-08`). `StepRunnerOptions` carried both, `withStepTimeout`
+  implemented the ceiling and `steps.test.ts` exercised it by building a runner BY HAND — while the
+  only production construction passed neither and `JobDefinition` had no field that could. Same
+  verdict as `job.concurrency`: a documented guarantee that silently does nothing is the worst of
+  the three options, so it is threaded rather than deleted. Both are refused at declaration when
+  non-positive, because `withStepTimeout` reads `<= 0` as "no ceiling at all".
+- **`registeredJobs()`/`registeredTasks()` sort by CODE UNITS, never `localeCompare`.** The list is
+  projected by `describeJobs()` into `x.manifest.json`, which both tracked apps commit and
+  `x verify`'s `drift` step diffs byte for byte; `localeCompare` with no locale argument answers
+  from the runtime's ICU default and collation version. Same rule as `@ultimat3/http`'s
+  `describeRoutes`, restated locally rather than imported — `http` is not below this package.
 - **A driver's semantics are pinned in ONE test with the pg statement beside them.**
   `driver-parity.test.ts` asserts the memory driver's behaviour and the SQL that has to mean the
   same thing in a single test, so neither side can move alone. `introspect.list` answered
@@ -519,6 +609,7 @@ picture from the other side.
 | `steps.ts` | `StepStore`, `StepApi`, memoized-replay executor, `StepSuspension` |
 | `outbox.ts` | staging in a `Tx`, the relay, the ambient `JobsFacade` slot |
 | `outbox-pg.ts` | `createPgOutboxStore` — `stage()` on the caller's OWN connection, claim on the pool |
+| `outbox-lease.ts` | the claim lease's one definition and its one normalisation, for both stores |
 | `leases.ts` | `LeaseStore` — fleet-wide slots, the memory one, `jobLeaseKey` |
 | `metrics.ts` | `queue_oldest_ready_seconds` and `queue_dead_jobs`, the two alertable gauges |
 | `scheduler-pg.ts` | `pgSchedulerState` (the durable watermark) + `createPgLeaseLeader` |
@@ -530,8 +621,10 @@ picture from the other side.
 | `driver-memory.ts` | `x dev` / tests |
 | `driver-redis.ts`, `driver-nats.ts` | honest `X_NOT_IMPLEMENTED` stubs |
 | `retry.ts` | backoff arithmetic, dead-letter decision |
+| `retry-classification.ts` | the OTHER half of that decision: what the thrown error says, and the stop reason the row and the log carry |
 | `execute.ts` | `executeJob` — one claimed job run and settled, and the run's deadline/cancel |
 | `heartbeat.ts` | one claimed job's lease: the renewal interval and the loss it reports |
+| `renewal-timer.ts` | the interval a renewal runs on, and the `stopped()` latch every branch after an await re-reads |
 | `worker.ts` | `worker` role, claim loop, drain |
 | `worker-run.ts` | one claimed job, wired: its heartbeat, its slot renewal, its run signal and its span, started together and handed back in one `finally` |
 | `run-signal.ts` | the signal ONE run is cancelled by — composition that can be handed back, and that the worker can abort itself |

package/README.md

@@ -315,6 +315,20 @@ Nothing can kill a body that ignores the signal, so the durable state is fenced:
 cancel every step write is refused with `X_ABORTED`, and a run that finishes anyway is logged
 as `jobs.timeout.abandoned` — the one way to find a handler that never reads `ctx.signal`.
 
+Three ceilings, declared on the job and nowhere else (`As of 2026-08` — `stepTimeout` and
+`eventPoll` had been implemented in the step runner since 1.0 with no declaration able to reach
+them, so no `job()` could ask for either):
+
+| Field | Bounds | Absent |
+|---|---|---|
+| `timeout` | the whole attempt — aborts `ctx.signal`, then fails it | no attempt deadline |
+| `stepTimeout` | ONE `step.run` — aborts that step's signal, then fails the step | no per-step ceiling |
+| `eventPoll` | how long a `step.waitForEvent` parks between polls | 30s |
+
+A zero or negative `stepTimeout` / `eventPoll` is refused at declaration, the way `concurrency: 0`
+is: `withStepTimeout` reads `<= 0` as "no ceiling at all", which is the opposite of what the author
+wrote.
+
 ## The transactional outbox
 
 ```ts
@@ -334,10 +348,15 @@ it after commit. The bug class this removes:
 
 Both are load-dependent, both pass every test you would write, and both produce "the email
 went out but the order isn't in the database". Joining the transaction closes the window.
-The relay publishes *then* marks published, so a crash re-publishes — collapsed by the
-idempotency key. A publish that FAILS stops the batch rather than letting later rows overtake it:
-`claim()` returns rows in `staged_at` order, so an app that stages `createInvoice` then
-`chargeCard` in one transaction must never have the charge run first. Set `mode: 'required'` to
+The relay publishes *then* marks published, so a crash re-publishes — and **that repeat is
+collapsed only while the first job is still live** (`As of 2026-08`): `SQL_ENQUEUE`'s conflict
+target is a partial index over `ready`/`delayed`/`running`/`suspended`, so a re-publish landing
+after the first job reached a terminal state inserts a second row and the handler runs again.
+**Handlers are at-least-once. Write them idempotent** — that is the standing contract, not a
+caveat on this paragraph. A publish that FAILS stops the batch rather than letting later rows
+overtake it: `claim()` returns rows in `staged_at, id` order — total, so two relays compose the
+same batch in the same order — and an app that stages `createInvoice` then `chargeCard` in one
+transaction must never have the charge run first. Set `mode: 'required'` to
 make an enqueue outside a transaction an `X_OUTBOX_NO_TX` error instead of a direct publish.
 
 **It is not on by default, and it is not on until you install it** (`As of 2026-08`). Three
@@ -355,6 +374,34 @@ transactional: `stage()` runs on the CALLER'S connection, never the pool. With n
 publishes straight to the driver — deliberate, so a script and a test enqueue with no wiring, but
 it is a fallback and not the guarantee.
 
+`claim()` is a **claim, not a read** (`As of 2026-08`). `for update skip locked` in a bare select
+holds its row locks only until that statement ends — under autocommit, before `claim()` even
+resolves — so two relays polling 200ms apart read the same unpublished rows and both publish them.
+`SQL_ENQUEUE` collapses that repeat only while the first job is still LIVE, because its conflict
+target is a partial index over the live states: a second publish landing after that job finished
+inserts a second row and **the handler runs twice**. So the claim stamps `claimed_at` in the same
+statement that locks the row, and that stamp is a lease — `claimLeaseMs` (a positive whole number
+of ms, default 30s; anything else is `X_INVARIANT` at construction) is how long the rows of a relay
+that DIED mid-batch wait before any relay may take them again. A batch a failed publish stopped is
+handed back at once through `release`, so a pool blip still costs one poll interval and not a lease
+window.
+
+**Every outbox mutation is fenced on the claimant, not just the claim** (`As of 2026-08`).
+`release` and `markPublished` both match on `claimed_by`, and `claim()` hands the token back on
+each record as `claimedBy`. A relay that stalls past its lease wakes up owning nothing: its late
+`release` would otherwise unclaim rows the relay that reclaimed them is mid-publish on (a third
+relay claims and republishes them), and its late `markPublished` would retire a row nobody has
+published yet — losing the job outright. Both are no-ops now, in the pg store and in the memory
+store alike.
+
+What the lease buys, precisely:
+
+| It stops | It does not stop |
+|---|---|
+| two relays holding one batch — a committed row cannot be claimed twice inside its lease | the handler running twice |
+| a lapsed claimant releasing or retiring a newer claimant's rows | a crash between publish and `markPublished` re-publishing after the first job is terminal |
+| a relay that died mid-batch stranding its rows forever | anything a **non-idempotent** handler does on its second run |
+
 The memory store (`createMemoryOutboxStore`, `x dev` and tests) **drops** a published row —
 `retained()` is the relay's backlog, not a running total; the pg store keeps `published_at` as
 the audit trail this map is not. A relay pass that throws is logged as `jobs.outbox.tick-failed`
@@ -447,6 +494,28 @@ retrySchedule({ attempts: 5, backoff: 'exponential', delay: 1000 })
 // => [1000, 2000, 4000, 8000]
 ```
 
+**The error decides too, not only the attempt count** (`As of 2026-08`). `executeJob` reads the
+thrown error's retry classification — `@ultimat3/core`'s `registerErrorRetry`, the same table
+`--json` and an HTTP client read — and a code nobody classified keeps the attempt-count path
+exactly as it had before.
+
+| Thrown | What the queue does |
+|---|---|
+| a `terminal` code (`X_SCRAPE_AUTH_FAILED`, a validation fault, a permission denial) | dead-lettered on the attempt it happened, `attempt` recorded, remaining attempts unspent — a rotated password retried five times is five more wrong passwords at a site that locks the account after three |
+| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay` — instead of the backoff. Still an attempt, still under the ceiling |
+| a `retryable` code (`X_TIMEOUT`, `X_DRAINING`) | the backoff schedule above, unchanged |
+| an **unclassified** code, or anything that is not an `UltimateError` | the backoff schedule above, unchanged. Most codes are unclassified and `retryFor` answers `terminal` for all of them, so reading that would have stopped every transient retry in every app |
+
+Classify your app's codes beside the module that declares them — that import IS the registration:
+
+```
+registerErrorRetry({ X_INVOICE_REJECTED: 'terminal', X_GATEWAY_BUSY: 'retry-after' });
+```
+
+Why a job stopped is on the row and in the log, never inferred: `jobs.attempt.failed` carries
+`stop: 'terminal' | 'attempts-exhausted'`, `JobExecution.stopReason` carries the same, and a
+terminal dead letter appends its verdict to `lastError` so `x jobs show` explains an attempt 1 of 5.
+
 ## Limits
 
 Two layers, and the difference matters:
@@ -486,6 +555,13 @@ still running it — at-least-once turning into twice. Alert on any non-zero rat
 measured from the last renewal that **landed**, on this process's clock, so a driver whose
 heartbeat hangs is caught the same as one that rejects.
 
+Neither fires for a job that finished (`As of 2026-08`). `stop()` is terminal for the renewal
+already **on the wire**, not only for the next one: a clean completion acks the row out of
+`running`, so the fenced UPDATE already in flight comes back `false` — and reported, that was
+`jobs.lease.lost` at error plus the counter, a page for a non-event, on every completed job whose
+pool was slow enough. The fleet slot's `jobs.worker.slot-lost` had the same shape and the same
+fix (`renewal-timer.ts`).
+
 ## Introspection
 
 `inspectQueues`, `inspectJob` (per-step trace), `inspectDeadLetters`, `retryFromStep`,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "2.0.0",
+  "version": "3.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "license": "MIT",
   "type": "module",
@@ -32,9 +32,9 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "2.0.0",
-    "@ultimat3/entity": "2.0.0",
-    "@ultimat3/schema": "2.0.0",
-    "@ultimat3/time": "2.0.0"
+    "@ultimat3/core": "3.0.0",
+    "@ultimat3/entity": "3.0.0",
+    "@ultimat3/schema": "3.0.0",
+    "@ultimat3/time": "3.0.0"
   }
 }

package/src/driver-pg-ddl.ts

@@ -109,6 +109,15 @@ create table if not exists x_outbox (
 create index if not exists x_outbox_unpublished_idx
   on x_outbox (staged_at) where published_at is null;
 
+-- The relays claim lease. A claim stamped in the same statement that locks the row is what stops
+-- two relays publishing one row twice: for update skip locked holds its locks only until that
+-- statement ends, which under autocommit is before claim returns. claimed_at is also what gives
+-- back the rows of a relay that died mid-batch, since a claim nothing can expire strands them.
+-- Added by alter because x_outbox shipped without them.
+alter table x_outbox add column if not exists claimed_at timestamptz;
+
+alter table x_outbox add column if not exists claimed_by text;
+
 -- The scheduler watermark. Without a durable one a redeployed scheduler has no idea what the
 -- pod it replaced already fired, so runRound takes the arming branch and every occurrence
 -- between the two processes is dropped with nothing logged.
@@ -177,4 +186,6 @@ create table if not exists x_outbox (
 );
 create index if not exists x_outbox_unpublished_idx
   on x_outbox (staged_at) where published_at is null;
+alter table x_outbox add column if not exists claimed_at timestamptz;
+alter table x_outbox add column if not exists claimed_by text;
 `.trim();

package/src/driver-pg-sql.ts

@@ -271,20 +271,81 @@ values ($1, $2, $3, $4::jsonb, $5, $6, to_timestamp($7 / 1000.0), to_timestamp($
         $9, $10, $11)
 `.trim();
 
+/**
+ * The claim, and it has to be ONE statement. `for update skip locked` in a bare select holds its
+ * row locks only until that statement ends — under autocommit, before `claim()` even resolves — so
+ * two relays polling 200ms apart read the same unpublished rows and both hand them to `enqueue`.
+ * `SQL_ENQUEUE` collapses the repeat only while the first job is still LIVE: its conflict target
+ * is a partial index over the live states, so a second publish landing after that job reached a
+ * terminal state inserts a second row and the handler runs again. (The mechanism is Postgres
+ * semantics; how often the two orderings line up in a deployment was never measured.)
+ *
+ * So the lock and the claim commit together, the CTE shape `SQL_CLAIM` already uses, and
+ * `claimed_at` is a LEASE: `$2` is the window after which a row a dead relay was holding is
+ * claimable again, because a claim nothing can expire strands its rows forever.
+ *
+ * The outer `select ... order by staged_at, id` is not cosmetic. `update ... returning` has no
+ * defined row order and the relay publishes in the order it is handed rows, so an app staging
+ * `createInvoice` then `chargeCard` in one transaction depends on this line.
+ *
+ * `, id` is what makes that key TOTAL, and the CTE needs it as much as the projection does: every
+ * row staged in one transaction shares a `staged_at`, so `staged_at` alone leaves the tie to the
+ * planner — which rows a `limit` takes, and in which order they publish, then differ between two
+ * relays and between two runs of one relay. No column was added for it: `id` is a UUIDv7 minted by
+ * `uuid()`, monotonic and already the primary key, so the tiebreak IS stage order.
+ */
 export const SQL_OUTBOX_CLAIM = `
+with claimable as (
+  select id
+    from x_outbox
+   where published_at is null
+     and (claimed_at is null
+          or claimed_at <= now() - ($2::bigint * interval '1 millisecond'))
+   order by staged_at, id
+   limit $1
+     for update skip locked
+), claimed as (
+  update x_outbox o
+     set claimed_at = now(), claimed_by = $3
+    from claimable c
+   where o.id = c.id
+  returning o.id, o.job, o.queue, o.input, o.idempotency_key, o.max_attempts, o.tenant_id,
+            o.traceparent, o.enqueued_by, o.claimed_by, o.run_at, o.staged_at
+)
 select id, job, queue, input, idempotency_key, max_attempts, tenant_id,
-       traceparent, enqueued_by,
+       traceparent, enqueued_by, claimed_by,
        (extract(epoch from run_at) * 1000)::bigint    as run_at,
        (extract(epoch from staged_at) * 1000)::bigint as staged_at
-  from x_outbox
- where published_at is null
- order by staged_at
- limit $1
-   for update skip locked
+  from claimed
+ order by staged_at, id
 `.trim();
 
+/**
+ * Hand a claim back early. The relay stops its batch on the first publish that fails, and without
+ * this the rows behind it would wait out the whole lease before any relay could retry them — a
+ * pool blip during a failover becoming tens of seconds of unpublished, committed work.
+ *
+ * Fenced on `published_at is null` so it can never unclaim a row some other pass already
+ * published, AND on `claimed_by` so it can never unclaim one a NEWER claimant now holds: a relay
+ * whose lease lapsed while it stalled wakes into a world where its batch is another relay's, and
+ * an unfenced release frees rows that relay is mid-publish on — a third relay claims them and
+ * publishes them again, which is the duplicate the lease exists to prevent.
+ */
+export const SQL_OUTBOX_RELEASE = `
+update x_outbox
+   set claimed_at = null, claimed_by = null
+ where id = any($1::uuid[]) and published_at is null and claimed_by = $2
+`.trim();
+
+/**
+ * Same fence, and here it is the more expensive one to miss: marking a row published is LOSING it,
+ * so a lapsed claimant stamping a row the current one has not published yet drops that job with
+ * nothing to notice. `published_at is null` makes the stamp first-writer-wins rather than a
+ * rewrite of an audit timestamp.
+ */
 export const SQL_OUTBOX_MARK_PUBLISHED = `
-update x_outbox set published_at = to_timestamp($2 / 1000.0) where id = $1
+update x_outbox set published_at = to_timestamp($2 / 1000.0)
+ where id = $1 and published_at is null and claimed_by = $3
 `.trim();
 
 export const SQL_OUTBOX_PENDING_COUNT = `

package/src/errors.ts

@@ -1,6 +1,6 @@
 // The X_* codes owned by @ultimat3/jobs. Every one names the command or code change that
 // fixes it — a job failure an agent cannot act on is a job failure that gets retried forever.
-import { registerErrorCodes, UltimateError } from '@ultimat3/core';
+import { registerErrorCodes, registerErrorRetry, UltimateError } from '@ultimat3/core';
 
 /** Codes this package declares and owns. */
 export const JOB_OWNED_ERROR_CODES = [
@@ -66,6 +66,29 @@ registerErrorCodes(
   Object.fromEntries(Object.entries(JOB_ERROR_TITLES).map(([code, title]) => [code, { title }])),
 );
 
+/**
+ * The codes of this package's that can be thrown INSIDE a job body, classified — `executeJob`
+ * reads this, so a `terminal` one dead-letters on the attempt it happened instead of spending the
+ * whole policy on an answer that cannot change. Same rule every package uses: retryable means the
+ * same code, run again, has a real chance of a different answer.
+ *
+ * Two are deliberately absent. `X_JOB_LEASE_LOST` and `X_JOB_SLOT_LOST` mean the row is somebody
+ * else's now, so this attempt's verdict is not this attempt's to give: dead-lettering would settle
+ * a job another worker is running. They keep the attempt-count path, which ends in the queue
+ * re-delivering — the honest outcome for "we stopped owning it".
+ */
+registerErrorRetry({
+  X_JOB_TIMEOUT: 'retryable',
+  X_DRIVER_UNAVAILABLE: 'retryable',
+  // A second `step.run` under one name is a defect in the handler, replayed identically forever.
+  X_STEP_DUPLICATE: 'terminal',
+  // A sweep whose source ran dry while its own count still matches rows: the next attempt resumes
+  // at the cursor that just ran dry and diverges again.
+  X_BACKFILL_STALLED: 'terminal',
+  X_BACKFILL_ENVIRONMENT: 'terminal',
+  X_BACKFILL_APPLIED: 'terminal',
+});
+
 const docsFor = (code: JobErrorCode): string => `https://ultimate.dev/errors/${code}`;
 
 /** An enqueue collided with a live job holding the same idempotency key under `onConflict: 'error'`. */

package/src/execute.ts

@@ -18,7 +18,8 @@ import type { ClaimedJob, JobDriver } from './driver';
 import { JobAbortedError, JobTimeoutError } from './errors';
 import { eventBus } from './events';
 import type { AnyJobHandle } from './job';
-import { nextRetry } from './retry';
+import type { JobStopReason } from './retry-classification';
+import { nextRetryForError, recordedFailure } from './retry-classification';
 import type { EventLookup, StepRecord } from './steps';
 import { createStepRunner, isStepSuspension } from './steps';
 import { jobRunActor } from './tenant';
@@ -69,6 +70,8 @@ export interface JobExecution {
   readonly durationMs: number;
   readonly resumeAt?: number;
   readonly error?: string;
+  /** Why this attempt was the last. Absent while the job is still being retried. */
+  readonly stopReason?: JobStopReason;
   readonly steps: readonly StepRecord[];
   readonly replayed: readonly string[];
 }
@@ -105,6 +108,11 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
     jobName: handle.name,
     store: driver.steps,
     signal,
+    // The DECLARED per-step ceiling and event poll. Passed here or nowhere: this is the only
+    // production construction of a runner, so a `StepRunnerOptions` field it omits is a feature
+    // no `job()` can reach — which both of these were until 2026-08.
+    ...(handle.stepTimeoutMs === undefined ? {} : { stepTimeoutMs: handle.stepTimeoutMs }),
+    ...(handle.eventPollMs === undefined ? {} : { eventPollMs: handle.eventPollMs }),
     ...(options.clock === undefined ? {} : { clock: options.clock }),
     events: options.events ?? eventBus(),
   });
@@ -166,10 +174,16 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
     }
 
     const message = error instanceof Error ? error.message : String(error);
-    const decision = nextRetry(handle.retry, claimed.attempt);
+    // The ERROR decides too, not only the attempt count. A `terminal` code — a rotated password,
+    // a schema mismatch, a permission denial — fails identically on every remaining attempt, so
+    // spending them is a queue slot, a provider bill and, at a site that locks an account after
+    // three wrong passwords, the framework destroying what it was asked to read. A code nobody
+    // classified keeps the attempt-count path exactly as it was.
+    const decision = nextRetryForError(handle.retry, claimed.attempt, error);
+    const stop = decision.stoppedBy;
     await driver.nack(claimed.id, {
       delayMs: decision.delayMs,
-      error: message,
+      error: recordedFailure(message, decision),
       countsAsAttempt: true,
       deadLetter: !decision.retry && decision.deadLetter,
     });
@@ -178,6 +192,9 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
       jobId: claimed.id,
       attempt: claimed.attempt,
       retry: decision.retry,
+      // "stopped because terminal" and "stopped because the attempts ran out" are different
+      // incidents with the same `retry: false`, and only one of them is fixed by raising attempts.
+      ...(stop === undefined ? {} : { stop }),
       error: message,
     });
     // This package's ONE error-reporting call site, and it is here rather than in the loop because
@@ -195,6 +212,7 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
           runId: claimed.runId,
           attempt: claimed.attempt,
           retry: decision.retry,
+          ...(stop === undefined ? {} : { stop }),
         },
       },
     });
@@ -205,6 +223,7 @@ export async function executeJob(options: ExecuteJobOptions): Promise<JobExecuti
       attempt: claimed.attempt,
       durationMs: nowMs(options.clock) - startedAt,
       error: message,
+      ...(stop === undefined ? {} : { stopReason: stop }),
       steps: [],
       replayed: [],
     });

package/src/heartbeat.ts

@@ -8,6 +8,7 @@ import { logger, recordLeaseLost } from '@ultimat3/core';
 import { nowMs } from './clock';
 import type { ClaimedJob, JobDriver } from './driver';
 import { LeaseLostError } from './errors';
+import { startRenewalTimer } from './renewal-timer';
 
 export interface LeaseHeartbeatOptions {
   /** Only `heartbeat` is used — a lease renews itself and settles nothing. */
@@ -49,14 +50,8 @@ export function startLeaseHeartbeat(options: LeaseHeartbeatOptions): LeaseHeartb
   let renewedAt = now();
   let renewing = false;
   let lost = false;
-  let timer: ReturnType<typeof setInterval> | undefined;
   const gone = new AbortController();
 
-  const stop = (): void => {
-    if (timer !== undefined) clearInterval(timer);
-    timer = undefined;
-  };
-
   const lapsed = (): boolean => now() - renewedAt >= visibilityTimeoutMs;
 
   /**
@@ -65,9 +60,14 @@ export function startLeaseHeartbeat(options: LeaseHeartbeatOptions): LeaseHeartb
    * ticked once per interval would count intervals, not jobs.
    */
   const reportLost = (error?: unknown, reason?: 'expired' | 'not-ours'): void => {
-    if (lost) return;
+    // `stopped()` as well as `lost`, and it is the difference between a page and a fact: a clean
+    // completion acks the row out of `running` and `worker-run.ts` stops the heartbeat, so a
+    // renewal already in flight comes back `false` for a job that FINISHED. Reported, that is
+    // `jobs.lease.lost` at error plus `recordLeaseLost(queue)` — the one signal meaning the queue
+    // re-delivered a job this process was still running — raised for a run nobody re-delivered.
+    if (lost || timer.stopped()) return;
     lost = true;
-    stop();
+    timer.stop();
     logger.error('jobs.lease.lost', {
       workerId,
       job: claimed.name,
@@ -86,7 +86,7 @@ export function startLeaseHeartbeat(options: LeaseHeartbeatOptions): LeaseHeartb
   };
 
   const renew = async (): Promise<void> => {
-    if (lost) return;
+    if (lost || timer.stopped()) return;
     // Expiry is decided BEFORE the driver is asked, because the failure that loses a lease most
     // quietly is the one that never answers: a heartbeat hung on a dead connection neither
     // resolves nor rejects, so a check that ran only on rejection would never run at all.
@@ -100,6 +100,10 @@ export function startLeaseHeartbeat(options: LeaseHeartbeatOptions): LeaseHeartb
     renewing = true;
     try {
       const held = await options.driver.heartbeat(claimed.id, { visibilityTimeoutMs, workerId });
+      // Re-read AFTER the await, never only before it: the whole point of the flag is the answer
+      // that lands past `stop()`. Every branch below decides something about a lease this process
+      // may no longer be running under.
+      if (timer.stopped()) return;
       // The driver answered, and it said the row is not ours. That is a DIFFERENT fact from an
       // expired window and the only one an operator can cause on purpose: `x jobs cancel` writes
       // a terminal state, and the renewal that misses it is what tells this attempt to stop. It
@@ -138,9 +142,7 @@ export function startLeaseHeartbeat(options: LeaseHeartbeatOptions): LeaseHeartb
     }
   };
 
-  timer = setInterval(() => {
-    void renew();
-  }, options.intervalMs);
+  const timer = startRenewalTimer(options.intervalMs, renew);
 
-  return { renew, lost: () => lost, signal: gone.signal, stop };
+  return { renew, lost: () => lost, signal: gone.signal, stop: () => timer.stop() };
 }

package/src/index.ts

@@ -107,6 +107,7 @@ export {
   SQL_NACK,
   SQL_OUTBOX_CLAIM,
   SQL_OUTBOX_MARK_PUBLISHED,
+  SQL_OUTBOX_RELEASE,
   SQL_OUTBOX_STAGE,
   SQL_OUTBOX_TABLE,
   SQL_SCHEDULER_STATE_GET,
@@ -191,6 +192,7 @@ export {
 export type {
   EnqueueOptions,
   JobsFacade,
+  MemoryOutboxOptions,
   MemoryOutboxStore,
   OutboxDeps,
   OutboxRecord,
@@ -207,11 +209,16 @@ export {
   resetJobsFacade,
   setJobsFacade,
 } from './outbox';
+// One definition of the lease, consumed by both stores — a memory default and a pg default that
+// could drift are two answers to "how long is a claim mine for", and the shorter one duplicates.
+export { DEFAULT_OUTBOX_CLAIM_LEASE_MS } from './outbox-lease';
 export type { PgOutboxOptions } from './outbox-pg';
 export { createPgOutboxStore } from './outbox-pg';
 
 export type { BackoffStrategy, Random, RetryDecision, RetryPolicy } from './retry';
 export { backoffDelayMs, DEFAULT_RETRY, nextRetry, retrySchedule } from './retry';
+export type { JobRetryDecision, JobStopReason } from './retry-classification';
+export { classifyThrown, nextRetryForError } from './retry-classification';
 export type {
   CronResolver,
   DispatchedOccurrence,

package/src/job.ts

@@ -80,6 +80,22 @@ export interface JobDefinition<I> {
    */
   readonly concurrency?: number;
   readonly timeout?: DurationInput;
+  /**
+   * Ceiling for ONE `step.run`, where `timeout` is the ceiling for the whole attempt. Folded into
+   * the signal the step body is handed, so a body reads one signal and sees whichever deadline
+   * lands first — and it ABORTS before it rejects, because the attempt that replaces this one is
+   * claimable the moment the nack lands.
+   *
+   * Declared here and nowhere else: the runner has implemented this ceiling since 1.0 and no
+   * declaration could ask for it, which is a documented guarantee that does nothing.
+   */
+  readonly stepTimeout?: DurationInput;
+  /**
+   * How long a `step.waitForEvent` parks between polls. Default 30s. Lower it for a wait a user
+   * is watching; the step suspends for exactly this long each time, so it is also the resolution
+   * of the resume, never a busy loop.
+   */
+  readonly eventPoll?: DurationInput;
   run(args: JobRunArgs<I>): Promise<unknown>;
 }
 
@@ -114,6 +130,10 @@ export interface JobHandle<I = unknown> {
   readonly retry: RetryPolicy;
   readonly concurrency: number | undefined;
   readonly timeoutMs: number | undefined;
+  /** The declared per-step ceiling in ms; `executeJob` hands it to the step runner. */
+  readonly stepTimeoutMs: number | undefined;
+  /** The declared event-poll interval in ms; `undefined` leaves the runner's 30s default. */
+  readonly eventPollMs: number | undefined;
   readonly input: StandardSchemaV1<unknown, I>;
   parse(raw: unknown): I;
   idempotencyKeyFor(input: I): string;
@@ -183,6 +203,28 @@ export function job<I>(definition: JobDefinition<I>): JobHandle<I> {
     `set a whole concurrency of 1 or more on job("${name}"), or omit the field for no cap at all`,
   );
 
+  const stepTimeoutMs =
+    definition.stepTimeout === undefined ? undefined : toMs(definition.stepTimeout);
+  const eventPollMs = definition.eventPoll === undefined ? undefined : toMs(definition.eventPoll);
+  // `withStepTimeout` reads `<= 0` as "no ceiling at all" and a poll of zero is a suspension that
+  // resumes immediately, forever. Both are an author who asked for a limit and got the opposite,
+  // so they are refused where they are written — the same answer `concurrency: 0` gets.
+  //
+  // FINITE, not merely positive: `> 0` admits `Infinity`, which is the same defect spelled the
+  // other way. `eventPoll: Infinity` parks a waiting step and schedules the poll that would wake
+  // it for never; `stepTimeout: Infinity` is a ceiling no step can reach. `NaN` fails `> 0` on its
+  // own, and is covered here so the predicate says what it means rather than passing by accident.
+  assert(
+    stepTimeoutMs === undefined || (Number.isFinite(stepTimeoutMs) && stepTimeoutMs > 0),
+    `job "${name}" declares stepTimeout ${String(definition.stepTimeout)}, which is no ceiling at all`,
+    `set a finite positive stepTimeout on job("${name}") — "30s" or 30_000 — or omit the field for no per-step ceiling`,
+  );
+  assert(
+    eventPollMs === undefined || (Number.isFinite(eventPollMs) && eventPollMs > 0),
+    `job "${name}" declares eventPoll ${String(definition.eventPoll)}, which parks a waiting step for no time at all`,
+    `set a finite positive eventPoll on job("${name}") — "5s" or 5_000 — or omit the field for the 30s default`,
+  );
+
   const handle: JobHandle<I> = {
     kind: 'job',
     name,
@@ -190,6 +232,8 @@ export function job<I>(definition: JobDefinition<I>): JobHandle<I> {
     retry: { ...DEFAULT_RETRY, ...definition.retry },
     concurrency: definition.concurrency,
     timeoutMs: definition.timeout === undefined ? undefined : toMs(definition.timeout),
+    stepTimeoutMs,
+    eventPollMs,
     input: definition.input,
     parse(raw: unknown): I {
       return parse(definition.input, raw) as I;
@@ -311,8 +355,18 @@ export function getJob(name: string): AnyJobHandle | undefined {
   return registry.get(name);
 }
 
+/**
+ * Code-unit compare, never `localeCompare`. This list is projected into `x.manifest.json`, which
+ * both tracked apps COMMIT and `x verify`'s drift step diffs byte for byte — and `localeCompare`
+ * with no locale argument answers from the runtime's ICU default and collation version, so the
+ * same source could sort two ways on two machines. `@ultimat3/http`'s `describeRoutes` states the
+ * same rule; the comparator is restated rather than imported because `http` is not below this
+ * package on the tier table.
+ */
+const byName = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0);
+
 export function registeredJobs(): readonly AnyJobHandle[] {
-  return [...registry.values()].sort((a, b) => a.name.localeCompare(b.name));
+  return [...registry.values()].sort((a, b) => byName(a.name, b.name));
 }
 
 export function resetJobs(): void {

package/src/outbox-lease.ts

@@ -0,0 +1,29 @@
+// The claim lease's one definition and its one normalisation. Both outbox stores read it, because
+// a lease the memory store defaults and the pg store validates is two answers to "how long is a
+// claim mine for" — and the shorter of the two is a row published twice.
+
+import { assert } from '@ultimat3/core';
+
+/**
+ * How long a claimed row stays its claimant's. Long enough that no healthy pass loses a batch it
+ * is still publishing, short enough that a relay killed mid-batch does not strand one for minutes.
+ */
+export const DEFAULT_OUTBOX_CLAIM_LEASE_MS = 30_000;
+
+/**
+ * Refused where it is written, the way `concurrency: 0` and `stepTimeout: 0` are. A lease of `0`
+ * expires before `claim()` resolves, so every relay reclaims every row on every tick and the lease
+ * buys nothing; a fractional one is compared against `now()` in Postgres and against whole ms
+ * here; `Infinity` never expires, so the rows of a relay that died are stranded forever — the one
+ * failure the lease exists to bound. `X_INVARIANT` because this is a caller-argument check with no
+ * dedicated code, the generic `@ultimat3/db` already borrows for the same shape.
+ */
+export function resolveClaimLeaseMs(value: number | undefined): number {
+  if (value === undefined) return DEFAULT_OUTBOX_CLAIM_LEASE_MS;
+  assert(
+    Number.isInteger(value) && value > 0,
+    `outbox claimLeaseMs is ${String(value)}, which is not a positive whole number of milliseconds`,
+    'pass a positive whole claimLeaseMs: 30_000 — createPgOutboxStore({ executor, txExecutor, claimLeaseMs: 30_000 }) — or omit the field for the 30s default',
+  );
+  return value;
+}

package/src/outbox-pg.ts

@@ -10,6 +10,7 @@
 // the transaction's connection — so the wiring is one line there and no tier crossing here.
 
 import type { Clock } from '@ultimat3/core';
+import { uuid } from '@ultimat3/core';
 import type { Tx } from '@ultimat3/entity';
 import { nowMs } from './clock';
 import type { PgExecutor } from './driver-pg';
@@ -17,9 +18,11 @@ import {
   SQL_OUTBOX_CLAIM,
   SQL_OUTBOX_MARK_PUBLISHED,
   SQL_OUTBOX_PENDING_COUNT,
+  SQL_OUTBOX_RELEASE,
   SQL_OUTBOX_STAGE,
 } from './driver-pg-sql';
 import type { OutboxRecord, OutboxStore } from './outbox';
+import { resolveClaimLeaseMs } from './outbox-lease';
 
 interface OutboxRow {
   readonly id: string;
@@ -33,6 +36,7 @@ interface OutboxRow {
   readonly tenant_id: string | null;
   readonly traceparent: string | null;
   readonly enqueued_by: string | null;
+  readonly claimed_by?: string | null;
 }
 
 export interface PgOutboxOptions {
@@ -47,6 +51,22 @@ export interface PgOutboxOptions {
    */
   readonly txExecutor: (tx: Tx) => PgExecutor;
   readonly clock?: Clock;
+  /**
+   * How long a claimed row stays this relay's before any relay may take it again. It bounds one
+   * thing only: how long the rows of a relay that DIED mid-batch sit unpublished. A pass that is
+   * merely slow keeps its rows because it published them; a pass that failed hands them back
+   * through `release`.
+   */
+  readonly claimLeaseMs?: number;
+  /**
+   * Written to `claimed_by`, and read back as the FENCE on `release` and `markPublished` — so it
+   * must be UNIQUE PER PROCESS. Two replicas passing one literal are one claimant to Postgres, and
+   * each can then release or retire the other's live batch. Omit it: the default is
+   * `relay-<uuid>`, minted once per store, which is unique by construction. Diagnostics second —
+   * it is what an operator reads to see which relay is sitting on a batch, and a value that
+   * changed every tick would answer nobody.
+   */
+  readonly relayId?: string;
 }
 
 function toRecord(row: OutboxRow): OutboxRecord {
@@ -62,6 +82,7 @@ function toRecord(row: OutboxRow): OutboxRecord {
     ...(row.tenant_id === null ? {} : { tenantId: row.tenant_id }),
     ...(row.traceparent === null ? {} : { traceparent: row.traceparent }),
     ...(row.enqueued_by === null ? {} : { enqueuedBy: row.enqueued_by }),
+    ...(typeof row.claimed_by === 'string' ? { claimedBy: row.claimed_by } : {}),
   };
 }
 
@@ -71,6 +92,13 @@ export function createPgOutboxStore(options: PgOutboxOptions): OutboxStore {
   // is neither committed nor rolled back (a process killed mid-request) leaves nothing behind.
   const staged = new WeakMap<object, OutboxRecord[]>();
   const key = (tx: Tx): object => tx as unknown as object;
+  // One id per store, minted here rather than per claim: `claimed_by` is read by an operator
+  // asking which relay is sitting on a batch, and a value that changed every tick answers nobody.
+  // Per-store is also the granularity the fence needs — two relays are two processes, two stores.
+  const relayId = options.relayId ?? `relay-${uuid()}`;
+  // Resolved once, at construction, so a lease this store could never honour fails where it was
+  // written instead of inside a relay tick whose only trace is a log line nobody reads.
+  const claimLeaseMs = resolveClaimLeaseMs(options.claimLeaseMs);
 
   return {
     async stage(tx, record) {
@@ -112,18 +140,41 @@ export function createPgOutboxStore(options: PgOutboxOptions): OutboxStore {
     },
 
     /**
-     * `for update skip locked` is in the statement, and under autocommit its row locks last only
-     * for that statement — so two relays can hand the same row to `enqueue`. That is the
-     * at-least-once the whole design already assumes and the idempotency key already collapses;
-     * what the clause buys is that two relays running side by side do not serialise on each other.
+     * A CLAIM, not a read. `for update skip locked` in a bare select held its locks only for that
+     * statement — which under autocommit is over before this method resolves — so two relays
+     * polling 200ms apart got the identical batch and both published it. The idempotency key
+     * collapses that only while the first job is still live, so the repeat that lands after it
+     * finished runs the handler a second time. `SQL_OUTBOX_CLAIM` stamps `claimed_at` in the same
+     * statement that locks the row; `skip locked` still keeps two relays from serialising.
      */
     async claim(limit) {
-      const rows = await options.executor.query<OutboxRow>(SQL_OUTBOX_CLAIM, [limit]);
+      const rows = await options.executor.query<OutboxRow>(SQL_OUTBOX_CLAIM, [
+        limit,
+        claimLeaseMs,
+        relayId,
+      ]);
       return rows.map(toRecord);
     },
 
-    async markPublished(id, at) {
-      await options.executor.query(SQL_OUTBOX_MARK_PUBLISHED, [id, at || nowMs(options.clock)]);
+    /**
+     * Fenced on the CLAIMANT, not only on the ids. A relay that stalled past its lease wakes into
+     * a world where its batch belongs to another relay, and an unfenced release frees rows that
+     * relay is mid-publish on — a third relay claims them and publishes them again. `relayId` is
+     * the fallback because it is what this store stamped: a caller with no token is this store's
+     * own relay, and one holding somebody else's token could not have got it from here.
+     */
+    async release(ids, claimant) {
+      if (ids.length === 0) return;
+      await options.executor.query(SQL_OUTBOX_RELEASE, [ids, claimant ?? relayId]);
+    },
+
+    /** Same fence, and worse to miss: marking a row published is losing the job behind it. */
+    async markPublished(id, at, claimant) {
+      await options.executor.query(SQL_OUTBOX_MARK_PUBLISHED, [
+        id,
+        at || nowMs(options.clock),
+        claimant ?? relayId,
+      ]);
     },
 
     async pendingCount() {

package/src/outbox.ts

@@ -31,6 +31,7 @@ import type { EnqueueResult, JobDriver } from './driver';
 import { DEFAULT_QUEUE, jobDriver } from './driver';
 import { DriverUnavailableError, OutboxNoTxError } from './errors';
 import type { JobHandle } from './job';
+import { resolveClaimLeaseMs } from './outbox-lease';
 
 export interface OutboxRecord {
   readonly id: string;
@@ -46,6 +47,11 @@ export interface OutboxRecord {
   readonly traceparent?: string;
   readonly enqueuedBy?: string;
   readonly publishedAt?: number;
+  /**
+   * Stamped by `claim()`, absent on a staged row. Hand it back to `release`/`markPublished`: it is
+   * the FENCE, so a claimant whose lease lapsed cannot touch the rows a newer one is publishing.
+   */
+  readonly claimedBy?: string;
 }
 
 export interface OutboxStore {
@@ -55,12 +61,44 @@ export interface OutboxStore {
   commit(tx: Tx): Promise<readonly OutboxRecord[]>;
   /** Called by the tx runner after ROLLBACK. Staged rows vanish with the transaction. */
   rollback(tx: Tx): Promise<void>;
-  /** Unpublished, committed rows — the relay's work queue. */
+  /**
+   * CLAIM unpublished, committed rows — the relay's work queue, and a lease rather than a read.
+   * A store that hands the same rows to two relays hands the same job to two workers, and the
+   * idempotency key only collapses that while the first job is still live.
+   */
   claim(limit: number): Promise<readonly OutboxRecord[]>;
-  markPublished(id: string, at: number): Promise<void>;
+  /**
+   * Hand a claim back before its lease runs out, for the batch a failed publish stopped. OPTIONAL
+   * so a store written before the claim became a lease still compiles: without it those rows wait
+   * out the whole lease, which is slower, never wrong.
+   *
+   * `claimant` is the `claimedBy` the claim stamped. Passing it is what makes a lapsed relay's
+   * late release a no-op instead of an unclaim of somebody else's live batch.
+   */
+  release?(ids: readonly string[], claimant?: string): Promise<void>;
+  /** `claimant` fences the same way, and here it is worse to miss: this retires the row. */
+  markPublished(id: string, at: number, claimant?: string): Promise<void>;
   pendingCount(): Promise<number>;
 }
 
+/**
+ * The claim's sort key, and it is TOTAL: `id` after `stagedAt`, exactly what `SQL_OUTBOX_CLAIM`
+ * orders by. Every row staged in one transaction shares a `stagedAt`, so the key ties for the
+ * batch that most depends on order — and a tie leaves both which rows a limit takes and the order
+ * they publish in to whatever the store iterated first. Code units, never `localeCompare`, for
+ * the reason `registeredJobs()` sorts that way.
+ */
+function byClaimOrder(a: OutboxRecord, b: OutboxRecord): number {
+  if (a.stagedAt !== b.stagedAt) return a.stagedAt - b.stagedAt;
+  if (a.id === b.id) return 0;
+  return a.id < b.id ? -1 : 1;
+}
+
+export interface MemoryOutboxOptions {
+  readonly clock?: Clock;
+  readonly claimLeaseMs?: number;
+}
+
 export interface MemoryOutboxStore extends OutboxStore {
   /**
    * Committed rows this process is still holding. The relay's backlog and nothing else — a
@@ -75,11 +113,29 @@ export interface MemoryOutboxStore extends OutboxStore {
  * transaction" guarantee needs no cooperation from the DB layer and rollback is a delete.
  * The pg store swaps this for a real `x_outbox` table written by the same connection.
  */
-export function createMemoryOutboxStore(): MemoryOutboxStore {
+export function createMemoryOutboxStore(options: MemoryOutboxOptions = {}): MemoryOutboxStore {
   const staged = new WeakMap<object, OutboxRecord[]>();
   const committed = new Map<string, OutboxRecord>();
+  /** Each claimed row's lease: when it was taken and by whom. Absent is `claimed_at is null`. */
+  const claims = new Map<string, { at: number; by: string }>();
+  const leaseMs = resolveClaimLeaseMs(options.claimLeaseMs);
+  // A token per CLAIM, where the pg store stamps one per RELAY. Two relays there are two stores
+  // with two ids; here they are two `claim()` calls on one store, so the claim is the only
+  // granularity at which this store can answer "is this mutation from the current holder".
+  let claimSeq = 0;
 
   const key = (tx: Tx): object => tx as unknown as object;
+  const free = (id: string, at: number): boolean => {
+    const claim = claims.get(id);
+    return claim === undefined || at - claim.at >= leaseMs;
+  };
+  /**
+   * A mutation from a claimant that no longer holds the row is a NO-OP. `undefined` is the caller
+   * that holds no token at all — a store-level caller, or one written before the fence — and is
+   * left unfenced rather than silently dropped, the way `release` itself is optional.
+   */
+  const owns = (id: string, claimant: string | undefined): boolean =>
+    claimant === undefined || claims.get(id)?.by === claimant;
 
   return {
     stage(tx, record) {
@@ -98,19 +154,36 @@ export function createMemoryOutboxStore(): MemoryOutboxStore {
       staged.delete(key(tx));
       return Promise.resolve();
     },
+    /**
+     * The same question `SQL_OUTBOX_CLAIM` answers, and it has to stay the same one: a row this
+     * store hands back is CLAIMED for `leaseMs`, so a second relay polling the same store gets
+     * nothing, and a claim whose holder died is reclaimable once the window passes.
+     */
     claim(limit) {
+      const at = nowMs(options.clock);
+      claimSeq += 1;
+      const by = `claim-${claimSeq}`;
       const ready = [...committed.values()]
-        .filter((record) => record.publishedAt === undefined)
-        .sort((a, b) => a.stagedAt - b.stagedAt)
+        .filter((record) => record.publishedAt === undefined && free(record.id, at))
+        .sort(byClaimOrder)
         .slice(0, limit);
-      return Promise.resolve(ready);
+      for (const record of ready) claims.set(record.id, { at, by });
+      return Promise.resolve(ready.map((record) => ({ ...record, claimedBy: by })));
+    },
+    release(ids, claimant) {
+      for (const id of ids) {
+        if (owns(id, claimant)) claims.delete(id);
+      }
+      return Promise.resolve();
     },
-    markPublished(id, _at) {
+    markPublished(id, _at, claimant) {
+      if (!owns(id, claimant)) return Promise.resolve();
       // Deleted, not stamped. A published row is out of the relay's reach either way, and the
       // pg store's `published_at` column is a retained audit trail this map is not: rewriting
       // it in place held every payload ever enqueued — arbitrary job input — for the life of
       // the process, and made `claim()` and `pendingCount()` walk all of them every 200ms.
       committed.delete(id);
+      claims.delete(id);
       return Promise.resolve();
     },
     pendingCount() {
@@ -331,7 +404,10 @@ export function createOutboxRelay(options: RelayOptions): OutboxRelay {
           ...(record.traceparent === undefined ? {} : { traceparent: record.traceparent }),
           ...(record.enqueuedBy === undefined ? {} : { enqueuedBy: record.enqueuedBy }),
         });
-        await options.store.markPublished(record.id, nowMs(options.clock));
+        // The claim's own token goes back with the mark. Without it a relay whose lease lapsed
+        // mid-stall retires a row the relay that reclaimed it has not published yet — the row is
+        // gone and nothing publishes it.
+        await options.store.markPublished(record.id, nowMs(options.clock), record.claimedBy);
         published += 1;
       } catch (error) {
         // STOP the batch. `claim()` returns rows in `staged_at` order and the loop used to log
@@ -347,6 +423,13 @@ export function createOutboxRelay(options: RelayOptions): OutboxRelay {
           remaining: batch.length - published,
           error: error instanceof Error ? error.message : String(error),
         });
+        // Hand the rest of the batch back rather than sit on a claim nobody is publishing. The
+        // claim is a lease now, so without this a single pool timeout parks every committed row
+        // behind it for the whole lease window instead of for one poll interval.
+        await options.store.release?.(
+          batch.slice(published).map((row) => row.id),
+          record.claimedBy,
+        );
         break;
       }
     }

package/src/renewal-timer.ts

@@ -0,0 +1,35 @@
+// A renewal loop that is TERMINAL once stopped, and the one shape two files renew against.
+// `heartbeat.ts` renews a job's lease and `worker-fleet-slots.ts` a fleet slot, and both decided a
+// LOSS from an answer that arrived after the run had already finished cleanly: `stop()` cleared
+// the interval, which does nothing to the request already on the wire. So a flag is what every
+// branch after an `await` re-reads — the shape `settleWithin`'s `decided` uses in core.
+
+export interface RenewalTimer {
+  /**
+   * True once `stop()` has been called. Read AFTER every await in the renewal body: a clean
+   * completion settles the row this renewal is fenced on, so the driver answering "not yours"
+   * past that point is a finished job, not a lost lease — and reporting it is an error-level page
+   * for a non-event, on exactly the signals that mean the queue re-delivered live work.
+   */
+  stopped(): boolean;
+  /** Stop renewing, for the pass in flight as well as the next one. Idempotent. */
+  stop(): void;
+}
+
+export function startRenewalTimer(
+  intervalMs: number,
+  renew: () => void | Promise<void>,
+): RenewalTimer {
+  let stopped = false;
+  const timer = setInterval(() => {
+    void renew();
+  }, intervalMs);
+  return {
+    stopped: () => stopped,
+    stop(): void {
+      if (stopped) return;
+      stopped = true;
+      clearInterval(timer);
+    },
+  };
+}

package/src/retry-classification.ts

@@ -0,0 +1,112 @@
+// One retry decision from two questions the executor used to ask only half of: "are there
+// attempts left?" (./retry) and "is this error worth trying again at all?" (core's classification).
+// The backoff arithmetic stays in ./retry — nothing here recomputes a delay `nextRetry` owns.
+
+import type { ErrorRetry } from '@ultimat3/core';
+import {
+  DEFAULT_ERROR_RETRY,
+  declaredErrorRetry,
+  isErrorRetry,
+  isUltimateError,
+} from '@ultimat3/core';
+import { toMs } from './clock';
+import type { Random, RetryDecision, RetryPolicy } from './retry';
+import { DEFAULT_RETRY, nextRetry } from './retry';
+
+/** Why this attempt was the last one. Absent while the job is still being retried. */
+export type JobStopReason = 'terminal' | 'attempts-exhausted';
+
+export interface JobRetryDecision extends RetryDecision {
+  readonly stoppedBy: JobStopReason | undefined;
+  /** The classification consulted, or `undefined` when nobody classified the thrown code. */
+  readonly classification: ErrorRetry | undefined;
+}
+
+/**
+ * The classification that was DECLARED for this throw, or `undefined` when there is none.
+ *
+ * Deliberately not `error.retry` alone. That field is `init.retry ?? retryFor(code)` and
+ * `retryFor` fails closed, so every unclassified `UltimateError` already carries `terminal` —
+ * reading it would dead-letter the first attempt of every job in every app whose codes nobody has
+ * classified yet. So `terminal` counts only when it can have come from somewhere: an explicit
+ * per-instance override is indistinguishable from the default here, which is why an UNCLASSIFIED
+ * code carrying an instance `retry: 'terminal'` is read as unclassified. Register the code
+ * (`registerErrorRetry({ X_YOUR_CODE: 'terminal' })`) to have it honoured — one way, and the same
+ * way every other package declares it.
+ */
+export function classifyThrown(error: unknown): ErrorRetry | undefined {
+  if (!isUltimateError(error)) return undefined;
+  const retry: unknown = error.retry;
+  if (!isErrorRetry(retry)) return undefined;
+  // Anything other than the fail-closed default can only have come from the code table or from an
+  // explicit override, so it is somebody's answer either way.
+  if (retry !== DEFAULT_ERROR_RETRY) return retry;
+  return declaredErrorRetry(error.code) === undefined ? undefined : retry;
+}
+
+/**
+ * The delay a `retry-after` error NAMED, in ms. `retryAfterSeconds` on the error's `meta` is the
+ * framework's one spelling for it — `@ultimat3/http`'s `rateLimited` writes it and the 429's
+ * `Retry-After` header renders it — so a job and an HTTP client read the same number.
+ */
+export function statedDelayMs(error: unknown): number | undefined {
+  if (!isUltimateError(error)) return undefined;
+  const seconds: unknown = error.meta?.['retryAfterSeconds'];
+  if (typeof seconds !== 'number' || !Number.isFinite(seconds) || seconds < 0) return undefined;
+  return Math.round(seconds * 1_000);
+}
+
+/**
+ * Retry, dead-letter, and when. `terminal` stops here on the attempt that failed — the same code
+ * run again is the same answer, and the attempts left are a queue slot, a provider bill, and (the
+ * case that forced this) three more wrong passwords at a site that locks the account after three.
+ *
+ * Everything else keeps the attempt count in charge: `retry-after` only replaces the delay, never
+ * the ceiling, and an unclassified code takes exactly the path it took before this existed.
+ */
+export function nextRetryForError(
+  policy: RetryPolicy,
+  attempt: number,
+  error: unknown,
+  random?: Random,
+): JobRetryDecision {
+  const classification = classifyThrown(error);
+  if (classification === 'terminal') {
+    return {
+      retry: false,
+      delayMs: 0,
+      // The policy still decides park-or-drop: `deadLetter: false` means this app does not keep
+      // failed jobs, and that is not a preference a classification gets to overturn.
+      deadLetter: policy.deadLetter ?? true,
+      nextAttempt: attempt,
+      stoppedBy: 'terminal',
+      classification,
+    };
+  }
+
+  const decision = nextRetry(policy, attempt, random);
+  if (!decision.retry) {
+    return { ...decision, stoppedBy: 'attempts-exhausted', classification };
+  }
+  if (classification !== 'retry-after')
+    return { ...decision, stoppedBy: undefined, classification };
+
+  const stated = statedDelayMs(error);
+  if (stated === undefined) return { ...decision, stoppedBy: undefined, classification };
+  // Clamped by the policy's own ceiling, which is what `maxDelay` is for: a responder naming a
+  // day is still a responder this deployment has not agreed to wait a day for.
+  const cap = toMs(policy.maxDelay ?? DEFAULT_RETRY.maxDelay);
+  return { ...decision, delayMs: Math.min(stated, cap), stoppedBy: undefined, classification };
+}
+
+/**
+ * What the job ROW records. `lastError` is the one failure field a row carries, so a dead letter
+ * that stopped at attempt 1 of 5 has to explain itself there or `x jobs show` reads as a silent
+ * early stop. Only the terminal verdict is appended: exhaustion is already legible from
+ * `attempt === maxAttempts`.
+ */
+export function recordedFailure(message: string, decision: JobRetryDecision): string {
+  return decision.stoppedBy === 'terminal'
+    ? `${message} — not retried: this code is classified terminal, so every remaining attempt fails the same way`
+    : message;
+}

package/src/task.ts

@@ -215,8 +215,18 @@ export function nameTasks(record: Readonly<Record<string, TaskHandle>>): void {
   for (const [exportName, handle] of Object.entries(record)) registerTask(exportName, handle);
 }
 
+/**
+ * Code-unit compare, never `localeCompare`. This list is projected into `x.manifest.json`, which
+ * both tracked apps COMMIT and `x verify`'s drift step diffs byte for byte — and `localeCompare`
+ * with no locale argument answers from the runtime's ICU default and collation version, so the
+ * same source could sort two ways on two machines. `@ultimat3/http`'s `describeRoutes` states the
+ * same rule; the comparator is restated rather than imported because `http` is not below this
+ * package on the tier table.
+ */
+const byName = (a: string, b: string): number => (a < b ? -1 : a > b ? 1 : 0);
+
 export function registeredTasks(): readonly TaskHandle[] {
-  return [...registry.values()].sort((a, b) => a.name.localeCompare(b.name));
+  return [...registry.values()].sort((a, b) => byName(a.name, b.name));
 }
 
 export function getTask(name: string): TaskHandle | undefined {

package/src/worker-fleet-slots.ts

@@ -8,6 +8,7 @@ import type { ClaimedJob } from './driver';
 import { getJob } from './job';
 import type { HeldLease, LeaseStore } from './leases';
 import { jobLeaseKey } from './leases';
+import { startRenewalTimer } from './renewal-timer';
 
 /**
  * A renewal that REJECTED is not a lost slot: there is a TTL behind it and the interval gets
@@ -79,21 +80,25 @@ export function createFleetSlots(options: FleetSlotOptions): FleetSlots {
     startRenewal(jobId, onLost) {
       const slot = held.get(jobId);
       if (slot === undefined) return noop;
-      const stop = (): void => {
-        clearInterval(timer);
-      };
       // Renewed on the lease heartbeat's own interval and released in the same `finally`: one
       // clock for "this worker still owns the job" and "this worker still owns the slot" is one
-      // fewer way for them to disagree.
-      const timer = setInterval(() => {
-        void options.leases
+      // fewer way for them to disagree — and `timer.stopped()` is the same latch `heartbeat.ts`
+      // reads, for the same reason.
+      const timer = startRenewalTimer(options.renewIntervalMs, () =>
+        options.leases
           ?.renew(slot, options.ttlMs)
           .then((renewed) => {
             // `=== false`, never `!renewed`, for the reason `heartbeat.ts` reads `held` that way:
             // a store written before this return value existed resolves `undefined`, and treating
             // that as a loss would cancel every job on every renewal. Only an explicit no is one.
-            if (renewed !== false) return;
-            stop();
+            //
+            // `stopped()` re-read AFTER the await for the other half: the run settles, this timer
+            // is stopped and `worker.ts` releases the slot — so the renewal already on the wire
+            // finds the row gone and answers `false` for a job that FINISHED. Reported, that is
+            // `jobs.worker.slot-lost` at error and an abort on a controller `runSignal.dispose()`
+            // has already torn down: noise about a run nobody lost.
+            if (renewed !== false || timer.stopped()) return;
+            timer.stop();
             logger.error('jobs.worker.slot-lost', {
               workerId: options.workerId,
               jobId,
@@ -102,9 +107,9 @@ export function createFleetSlots(options: FleetSlotOptions): FleetSlots {
             });
             onLost?.(slot);
           })
-          .catch(noop);
-      }, options.renewIntervalMs);
-      return stop;
+          .catch(noop),
+      );
+      return () => timer.stop();
     },
 
     async release(jobId) {