@ultimat3/jobs 30.8.0 → 31.0.0

package/README.md

@@ -216,22 +216,31 @@ input. The tenant is a fact about the work, so the job declares it.
 | a stored event the wait's `schema` refuses | `X_JOB_EVENT_INVALID`, terminal, on a hit or a replay. The error carries the issues, never the payload |
 
 A boot's bus runs on the open transaction's connection (`currentTx() ?? client`, as the outbox):
-`publishEvent` inside a transaction commits and rolls back with it. `published_at` is the
-statement's time, so a wait that begins between that publish and its commit compares later than the
-stamp and never matches; the window is the length of the publishing transaction.
-
-`resetEventBus()` is the test seam beside `resetJobDriver()`: an event is matchable until it
-expires, so on a bus shared by one `bun test` process an answer one test published resumes the
+`publishEvent` inside a transaction commits and rolls back with it. **An event committed after a
+wait began is that wait's, whatever its stamp.** `published_at` is the INSERT's time and the row is
+visible at COMMIT, so a new wait records the snapshot it began under (`EventLookup.mark()`,
+`SQL_EVENT_MARK`, persisted on the waiting step) and also matches every event whose transaction
+(`x_job_events.xact`) that snapshot could not see. A rolled-back publish matches nothing. A
+waking bus published inside `withTransaction` wakes again once it COMMITS (`SQL_EVENT_COMMIT_WAKE`,
+run from `Tx.onCommit` on the pool): a run that parked while the transaction was open is resumed
+at the commit, not at the 15 min backstop.
+
+Events are consumed in `(published_at, id)` order, stamps truncated to the millisecond; the cursor a
+refused event leaves is that pair, compared and never looked up (`events-cursor.ts`, both buses),
+so a purged cursor still excludes what it excluded. A refusal only moves it FORWARD, under the
+horizon of the search that found the event; an event refused behind it (owed through the horizon)
+is excluded by id. The wait's own line — `startedAt` and the mark's horizon (`EventSince`) — rides
+every search and is never moved by a refusal, so no event committed before the wait is handed out.
+
+`resetEventBus()` is the test seam beside `resetJobDriver()`, both on `@ultimat3/jobs/adapter`: an
+event is matchable until it expires, so on a bus shared by one `bun test` process an answer one test published resumes the
 next test's wait — call it between tests.
 
 The event bus has ONE clock. `postgresEventBus({ executor })` stamps `publishedAt` and the expiry
 with the database's `now()` — it takes no `clock` — and `bus.now()` answers that same clock. A
 consumer that wants "published after I asked" (`@ultimat3/scraping`'s `eventPrompt`) reads its
 "asked at" from `bus.now()`, never from its own process: a skew between two pods must not decide
-which answers count. `step.waitForEvent` stamps a new wait from the same clock
-(`EventLookup.now()`), so a worker running ahead of the database still matches an event published
-after it asked.
-`step.waitForEvent` stamps a new wait from that clock (`EventLookup.now()`, required) and asks it
+which answers count. `step.waitForEvent` stamps a new wait from that clock (`EventLookup.now()`, required) and asks it
 again before declaring a timeout, so a worker running ahead of the database still matches an event
 published after it asked and does not give up early. `eventsPurgeTarget(bus)` is the bus as a
 `purge()` target: `purgeExpired()` is awaited and answers the rows it removed.
@@ -473,6 +482,63 @@ the new pods serve, puts the sweeps on the queue and exits, and a slow UPDATE ne
 open against a database still serving the previous build. `--all` isolates per name and continues
 past a failure, so one wedged cleanup cannot block every later one forever.
 
+### Rebuilding a search index: `reindex()`
+
+`reindex({ name, entity, source, tenant, index? })` is a `backfill()` in two phases, one durable
+step per page. **Rows**: each row's document goes to the `searchIndex` port — entity's one builder,
+`searchDocumentOf`, so only `.searchable()` columns leave and each document keeps its tenant.
+**Sweep**: every document the index `list`s whose row `source` no longer reads — deleted while the
+index was down, moved to another tenant — is removed, tombstoned one millisecond above the version
+listed, so an id written again meanwhile keeps its newer document. `source` is therefore the WHOLE
+table: a narrowed one empties the index of the rest. `tenant: 'none'` sweeps every tenant's
+documents; a declared tenant, its own. Run it after moving from the built-in Postgres index to
+Meilisearch or Typesense, or after an index lost data. An entity with no searchable column is
+refused at declaration. The index keeps only the NEWER version of each id, so neither a replayed
+batch nor a stale read can overwrite a newer write or bring a deleted row back.
+
+```ts
+import { database, entity, text, timestamp, uuid } from '@ultimat3/entity';
+import { reindex } from '@ultimat3/jobs';
+
+const posts = entity('posts', {
+  columns: {
+    id: uuid().primaryKey(),
+    title: text().searchable(),
+    updatedAt: timestamp().defaultNow().onUpdateNow(), // each document's version
+  },
+});
+const db = database({ posts });
+
+export const reindexPosts = reindex({
+  name: 'reindex-posts',
+  entity: posts,
+  tenant: 'none',
+  source: () => db.posts,
+});
+```
+
+## Partition maintenance is a job too
+
+`partitions({ name?, cron?, tz?, clock? })` keeps every `entity({ partitionBy })` table ready: one
+step per partitioned entity runs `@ultimat3/db`'s `partitionMaintenance(description, now)` — the
+function the migration also calls — creating `ahead` partitions past the current one and
+dropping those `retain` expired. It is a job plus a `.schedule` `task()` (`<name>.schedule`,
+default `DEFAULT_PARTITIONS_CRON`, 03:17 UTC daily), the shape `consumer()`'s `.supervisor` has.
+`now` is the job's clock, never the server's, so a test moves it; a second run at one instant
+changes nothing (`create … if not exists`). Entities are read per run, so one declared later is
+maintained from its first tick.
+
+```ts
+import { partitions } from '@ultimat3/jobs';
+
+export const maintainPartitions = partitions(); // list it in defineApi({ jobs })
+```
+
+Before a `backfill()` copies rows older than the first partition, enqueue it once with `{ from:
+'<oldest row ISO>' }`: that run creates every partition back to that row's period and drops
+nothing. Every DDL statement runs under a `lock_timeout`, failing fast with `X_DB_LOCK_TIMEOUT`
+(retryable — the job's retry absorbs it) rather than queueing behind a long transaction.
+
 ## Consumers are jobs too
 
 `consumer()` reads a queue, a change stream or a vendor's cursor: a factory over `job()` plus a
@@ -723,6 +789,29 @@ throws before a row is touched (`X_SUBJECT_TABLE_UNDECLARED`, `X_SUBJECT_TREATME
 `authSubjectTarget`, `auditSubjectTarget` and `storageSubjectTarget` satisfy it from their own
 packages. `{ target, retain: '<why>' }` exports a target and never erases it.
 
+### `personalData()` coverage
+
+`.personalData(kind)` on a column (`@ultimat3/entity`) marks it as a person's data. It adds no
+second export/erase path: it is the proof that the declared subjects cover every marked column.
+
+| Check | Refuses |
+|---|---|
+| `unerasedPersonalData(plan)` | a reached table whose `{ anonymize }` names fewer than every marked column (`X_PERSONAL_DATA_UNERASED`). `'delete'` clears the row; `{ retain: '<why>' }` is a stated reason |
+| `unreachedPersonalData(plans)` | a marked entity no subject's walk reaches (`X_PERSONAL_DATA_UNREACHED`) |
+| `personalDataProblems()` | both, over `registeredSubjects()` and every marked entity — what `x exercise` lists |
+
+Enforced twice: `plannedSubject()` throws the first `X_PERSONAL_DATA_UNERASED` when a run starts, so
+`subjectExport`/`subjectErase` refuse before a row is touched, and `x exercise` lists both codes
+as findings. **An erase is audited**: with an `auditSink` installed, `subjectErase` writes one
+`allowed` record per completed erase and a `failed` record per attempt that threw: `name` the job,
+`primitive: 'action'`, `surface: 'job'`, `input: { subject, run }` — the entity and the run id,
+never the subject's key, which an app may make an email. At least once: a sink dedupes on
+`idempotencyKey` (`<job>:<runId>`). A sink that refuses fails the run; one refusing every attempt
+can hold no record, and the dead-lettered run is the trail.
+
+`gdpr.job.test.ts` proves the property, not the shape: a covered erase leaves no marked value of the
+person in any marked table, and a declaration missing one column is named and does leak it.
+
 ## Outbound webhooks are jobs too
 
 `webhook()` is a factory over `job()` and delivers **one event to one endpoint**. Retry with
@@ -856,6 +945,53 @@ export const vendorEvents = webhookReceiver({
 | `.routes()` is set on the job handle in place, never on a copy | `isJobHandle` proves the object `job()` built, so `defineApi({ jobs })` registers it like any job |
 | the path is absolute and outside `/api` and `/_x` | `X_WEBHOOK_RECEIVER_INVALID` at declaration, beside a missing `verify`/`handle` or a bad `maxBytes` |
 
+## Double-entry postings are jobs too
+
+`ledger({ name, tenant, book })` is a factory over `job()`. Its input is a posting — `{ key,
+entries: [{ account, amount: Money }], orgId? }` — and its run refuses, before a line is written,
+anything that is not balanced Money; then it writes every line in ONE durable step through the
+app's `LedgerBook`. No provider, account plan or currency policy ships (axiom 8): the accounts
+are your strings, the table is your entity.
+
+```ts
+import { defaultDriver, entity, integer, money, text } from '@ultimat3/entity';
+import { entityLedgerBook, ledger } from '@ultimat3/jobs';
+
+// The app's lines: keyed (postingKey, line) — (orgId, postingKey, line) when tenanted, where a key
+// is one posting PER TENANT and the posting's `orgId` fills the tenant column.
+export const ledgerLines = entity('ledger_lines', {
+  columns: { postingKey: text(), line: integer(), account: text(), amount: money() },
+  primaryKey: ['postingKey', 'line'],
+});
+
+export const postPayment = ledger({
+  name: 'payments.post',
+  tenant: (posting) => posting.orgId ?? 'none',
+  book: () => entityLedgerBook({ driver: defaultDriver(), entity: ledgerLines }),
+});
+
+await postPayment.enqueue({
+  key: 'pay_123',
+  entries: [
+    { account: 'cash', amount: { minor: 1299, currency: 'EUR' } },
+    { account: 'revenue', amount: { minor: -1099, currency: 'EUR' } },
+    { account: 'vat-payable', amount: { minor: -200, currency: 'EUR' } },
+  ],
+});
+```
+
+| Rule | Why |
+|---|---|
+| every currency's entries sum to zero, each on its own (`X_LEDGER_UNBALANCED`) | an FX posting balances in EUR and in USD separately; netting across currencies is the error |
+| `minor` a safe integer, the currency ISO 4217, one `scale` per currency, a non-blank account, at least two entries (`X_LEDGER_ENTRY_INVALID`) | no float money; a sum across two scales is not a sum |
+| both refusals TERMINAL | the posting is the input: attempt five sees the same entries |
+| idempotent by key: the job's live key is `<name>:<key>`; the book writes a key once per tenant, `(tenant, postingKey, line)` | a retry, a replay or a second enqueue of `pay_123` posts it once — `entityLedgerBook` is one `insert … on conflict do nothing` |
+| `assertBalancedPosting(posting)` is exported | the action that enqueues refuses at the request, before a row is queued |
+| a key already stored with other lines — another count, another line — is `X_LEDGER_POSTING_CONFLICT` (409, terminal), nothing written | a superset under one key would store only the new lines beside the old: an unbalanced book. A correction is a new key. The Postgres book inserts and reads the key back in ONE transaction |
+
+A payment provider's webhook is a `webhookReceiver()` whose `handle` enqueues the posting, keyed
+by the provider's own id — the receiver dedupes the delivery, the ledger dedupes the posting.
+
 ## The deadline cancels
 
 A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that
@@ -1032,8 +1168,11 @@ One interface: `enqueue`, `claim` (visibility timeout), `ack(jobId, { workerId,
 `nack(jobId, { workerId, claim, … })` (backoff), `heartbeat`, `stats`, plus optional `introspect`,
 `backfills` and `leases`. Both settles are fenced on the CLAIM — `claimOf(claimed)` is the
 `{ workerId, claim }` a `ClaimedJob` carries — and answer whether they landed. Zero job-code change
-between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driver` config line**:
-`JobsConfig.driver` has no reader and boot always builds `postgresJobDriver`.
+between them — swapping is `jobs: other` in `apps/<app>/runtime.ts`, and there is **no
+`jobs.driver` config line**: `JobsConfig.driver` has no reader and boot always builds
+`postgresJobDriver`. A driver's author imports the port (`JobDriver`, `EventBus` and their types),
+the errors a driver throws and the installers from **`@ultimat3/jobs/adapter`** — never the `.`
+barrel, which carries no installer (`adapter.test.ts`).
 
 | Driver | Status | Backing | Use |
 |---|---|---|---|
@@ -1044,7 +1183,7 @@ between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driv
 **`redisJobDriver({ client?, prefix?, clock?, doneTtlMs? })`** passes `@ultimat3/testing`'s
 `jobDriverConformance`, the suite the other two pass, and `driver-redis.live.test.ts` runs one
 script of operations on it and on the memory driver and compares every answer. Installed with
-`setJobDriver(redisJobDriver())` (or `ServeOptions.runtime.jobs`), imported from its own entry
+`jobs: redisJobDriver()` in `apps/<app>/runtime.ts`, imported from its own entry
 `@ultimat3/jobs/redis` so the barrel every role boots carries none of it; the client reads `REDIS_URL`.
 What it does not carry: `introspect` (so `x jobs show`/`retry`/`cancel` and queue pauses answer
 for Postgres only), `backfills`, and `leases` — a job declaring `concurrency` refuses
@@ -1259,7 +1398,7 @@ export const wake = startQueueWake({ listener: client, executor });
 | a pause is read by the claim itself | `pauseQueue` stops claims at each worker's NEXT claim — no poll to wait for. `resumeQueue` announces the queue |
 | a paused task | checked when it is due, so the pause holds for the very next occurrence; a task found paused is looked at again after `PAUSE_RECHECK_MS` (5 s), which is the resume's latency |
 | a standby scheduler | asks for the lease every `renewEveryMs`, not every round |
-| a queue no registered job names | still polled — a per-call `enqueue({ queue })` may land there — and reported once at boot as `jobs.worker.queue-unregistered`; idle, it adds nothing to the one claim |
+| a queue no registered job names | still polled — a per-call `enqueue({ queue })` may land there — and reported once at boot as `jobs.worker.queue-unregistered`, `default` excepted (where a `job()` naming no queue lands); idle, it adds nothing to the one claim |
 | an enqueuer whose clock runs ahead of the database's | its "immediate" jobs are due a moment from now (`run_at` is the enqueuer's clock); up to 1 s ahead they are still announced, past that they start on the poll |
 
 Lower `idlePollMaxMs` to trade idle load for the no-wake bound; there is no flag for the old fixed interval.
@@ -1267,7 +1406,8 @@ Lower `idlePollMaxMs` to trade idle load for the no-wake bound; there is no flag
 ## Retries
 
 `{ attempts, backoff: 'exponential' | 'linear' | 'fixed', delay, maxDelay, jitter }`.
-Equal jitter is on by default so a burst of failures does not retry in lockstep.
+`jitter` is core's `JitterMode` (`'full' | 'equal' | 'none'`); `'equal'` is the default so a burst of
+failures does not retry in lockstep.
 Exhausted jobs are dead-lettered, never dropped: `x jobs retry <id>`. Only a finished job — dead,
 cancelled, failed or done — can be requeued: a live one is `X_JOB_NOT_REQUEUEABLE` (requeueing a
 running job used to run it twice), and one whose idempotency key a live job now holds is
@@ -1286,7 +1426,7 @@ exactly as it had before.
 | Thrown | What the queue does |
 |---|---|
 | a `terminal` code (`X_SCRAPE_AUTH_FAILED`, a validation fault, a permission denial) | dead-lettered on the attempt it happened, `attempt` recorded, remaining attempts unspent — a rotated password retried five times is five more wrong passwords at a site that locks the account after three |
-| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay`, plus core's `jitterStatedDelay` spread unless `jitter: false` — instead of the backoff. Still an attempt, still under the ceiling |
+| a `retry-after` code (`X_RATE_LIMITED`, `X_OVERLOADED`) | retried at the time the responder NAMED — `meta.retryAfterSeconds`, clamped by the policy's `maxDelay`, plus core's `jitterStatedDelay` spread unless `jitter: 'none'` — instead of the backoff. Still an attempt, still under the ceiling |
 | a `retryable` code (`X_TIMEOUT`, `X_DRAINING`) | the backoff schedule above, unchanged |
 | an **unclassified** code, or anything that is not an `UltimateError` | the backoff schedule above, unchanged. Most codes are unclassified and `retryFor` answers `terminal` for all of them, so reading that would have stopped every transient retry in every app |
 
@@ -1466,6 +1606,38 @@ Nor for a job being settled (`As of 2026-10-30`): the heartbeat stops when the w
 listing, and a renewal there matched no row — `jobs.lease.lost` for a job that had finished, with
 the hook's `ctx.signal` aborted under it.
 
+### `withLock(key, fn, { ttl })`
+
+One holder fleet-wide for `fn`, held in the job driver's `LeaseStore` as one slot under
+`lock:<key>` — the rows `job.concurrency` already counts, so no table, no Redis, no second
+mechanism. Not a primitive: a helper a job or action body calls.
+
+```ts
+import { withLock } from '@ultimat3/jobs';
+
+// The app's two halves: the slow read, and a write its store refuses under an older token.
+declare function recompute(id: string, options: { signal: AbortSignal }): Promise<number>;
+declare function saveTotal(id: string, total: number, token: number): Promise<void>;
+
+export const recount = (id: string): Promise<void> =>
+  withLock(
+    `invoice:${id}`,
+    async ({ token, signal }) => {
+      const total = await recompute(id, { signal });
+      await saveTotal(id, total, token);
+    },
+    { ttl: '30s', wait: '5s' },
+  );
+```
+
+| Option / fact | Meaning |
+|---|---|
+| `ttl` (required) | how long the key outlives a dead holder; renewed at a third of it while `fn` runs |
+| `wait` (default one `ttl`, `0` = one try) | then `X_LOCK_NOT_ACQUIRED` (409, retryable), naming the holder |
+| `token` | a NUMBER the lease store assigns per grant (`HeldLease.fence`; Postgres: `x_job_leases.fence` from one sequence), greater than every earlier holder's — never a client clock. Write it beside the data and refuse a smaller one. A store answering no fence is `X_LOCK_STORE_MISSING` |
+| a renewal refused, or none landing within `ttl` — decided before each call, one in flight, a late `true` still late | `fn`'s `signal` aborts with `X_LOCK_LOST`, and the call rejects with it even if `fn` resolved — its work was not exclusive. A renew hung on a dead socket is caught by the next tick |
+| `store` | default the installed driver's `leases`; none is `X_LOCK_STORE_MISSING` |
+
 ## The operator surface
 
 Every operator capability is a member of `JobIntrospection` (`driver.introspect`), implemented by
@@ -1585,9 +1757,16 @@ aborts the attempt — `steps.ts` then refuses every write. `ack` and `nack` are
 
 ## Trace and actor
 
-An enqueue stamps the current span's `traceparent` onto the row, and the worker opens the job's
-span as a **child** of it: a checkout trace shows the HTTP span, the action span and the charge
-that ran two seconds later as one trace.
+Every `enqueue` / `enqueueMany` opens a `job.<name>.enqueue` **producer** span (a child of the
+current span, or a new root) and stamps ITS `traceparent` onto the row; the worker opens the run's
+`job.<name>` **consumer** span as a child of that: a checkout trace shows the HTTP span, the action
+span, the enqueue and the charge that ran two seconds later as one trace. Both carry
+OpenTelemetry's messaging attributes (`messaging.system=ultimate`, `messaging.operation.type`,
+`messaging.destination.name` = the queue, `messaging.message.id`) beside `ultimate.job.*`.
+
+On core's instrumentation bus, sampled or not: `job.enqueue` (`{ job, queue, id }`) per enqueued
+job, `job.perform` (`{ job, queue, id, attempt, durationMs, error? }`) per attempt that ran — a
+failed attempt carries its error. `subscribe('job.perform', …)` from `@ultimat3/core`.
 
 `handle.as(actor, input)` also records `enqueuedBy` — the actor's id. **Attribution, never
 authority.** A job body runs with system authority; the framework does not impersonate the
@@ -1625,59 +1804,27 @@ FOR a user takes that user's id in its input and re-authorises it in the body.
 ### Error classes
 
 Every error class `src/index.ts` exports, for `instanceof` inside one process. Across a wire or
-a job boundary the class is gone and the `code` is what survives — match on that.
+a job boundary the class is gone and the `code` is what survives — match on that. Every other refusal (as of 31.0.0) has no exported class: match its `code`. (`./adapter`) marks the ones a driver author throws, exported from `@ultimat3/jobs/adapter` only.
 
 | Class | Code | Declared in |
 |---|---|---|
-| `ActionJobUnbridgedError` | `X_ACTION_JOB_UNBRIDGED` | `src/errors.ts` |
-| `BackfillAppliedError` | `X_BACKFILL_APPLIED` | `src/backfill-errors.ts` |
-| `BackfillEnvironmentError` | `X_BACKFILL_ENVIRONMENT` | `src/backfill-errors.ts` |
-| `BackfillMigrationPendingError` | `X_BACKFILL_MIGRATION_PENDING` | `src/backfill-errors.ts` |
 | `BackfillPendingError` | `X_BACKFILL_PENDING` | `src/backfill-errors.ts` |
 | `BackfillRunningError` | `X_BACKFILL_RUNNING` | `src/backfill-errors.ts` |
-| `BackfillStalledError` | `X_BACKFILL_STALLED` | `src/backfill-errors.ts` |
 | `BackfillUnknownError` | `X_BACKFILL_UNKNOWN` | `src/backfill-errors.ts` |
-| `CancelUnsupportedError` | `X_JOB_NOT_CANCELLABLE` | `src/errors.ts` |
-| `ClaimQueuesEmptyError` | `X_JOB_CLAIM_QUEUES_EMPTY` | `src/errors.ts` |
+| `ClaimQueuesEmptyError` (`./adapter`) | `X_JOB_CLAIM_QUEUES_EMPTY` | `src/errors.ts` |
 | `ConcurrencyUnenforceableError` | `X_JOB_CONCURRENCY_UNENFORCEABLE` | `src/errors-concurrency.ts` |
 | `DriverUnavailableError` | `X_DRIVER_UNAVAILABLE` | `src/errors.ts` |
-| `ExportPartTooLargeError` | `X_EXPORT_PART_TOO_LARGE` | `src/export-errors.ts` |
-| `ExportRowInvalidError` | `X_EXPORT_ROW_INVALID` | `src/export-errors.ts` |
-| `IdempotencyRequiredError` | `X_IDEMPOTENCY_REQUIRED` | `src/errors.ts` |
 | `JobAbortedError` | `X_ABORTED` | `src/errors.ts` |
-| `JobConcurrencyInvalidError` | `X_JOB_DECLARATION_INVALID` | `src/errors-concurrency.ts` |
-| `JobConcurrencyKeyInvalidError` | `X_JOB_DECLARATION_INVALID` | `src/errors-concurrency.ts` |
-| `JobDeclarationInvalidError` | `X_JOB_DECLARATION_INVALID` | `src/errors-declaration.ts` |
 | `JobDrainedError` | `X_DRAINING` | `src/errors.ts` |
 | `JobDuplicateError` | `X_JOB_DUPLICATE` | `src/errors.ts` |
 | `JobKeyBusyError` | `X_JOB_KEY_BUSY` | `src/errors-concurrency.ts` |
-| `JobMaxAttemptsError` | `X_JOB_MAX_ATTEMPTS` | `src/errors.ts` |
-| `JobNameTakenError` | `X_JOB_DUPLICATE` | `src/errors.ts` |
 | `JobNotCancellableError` | `X_JOB_NOT_CANCELLABLE` | `src/errors.ts` |
 | `JobNotFoundError` | `X_JOB_NOT_FOUND` | `src/errors-requeue.ts` |
-| `JobNotPromotableError` | `X_JOB_NOT_PROMOTABLE` | `src/errors-operator.ts` |
-| `JobNotRemovableError` | `X_JOB_NOT_REMOVABLE` | `src/errors-operator.ts` |
-| `JobNotRequeueableError` | `X_JOB_NOT_REQUEUEABLE` | `src/errors-requeue.ts` |
-| `JobOnSettledFailedError` | `X_JOB_ON_SETTLED_FAILED` | `src/errors-operator.ts` |
-| `JobPageInvalidError` | `X_JOB_PAGE_INVALID` | `src/errors-operator.ts` |
-| `JobRowStatusUnknownError` | `X_JOB_ROW_STATUS_UNKNOWN` | `src/errors.ts` |
-| `JobSlotLostError` | `X_JOB_SLOT_LOST` | `src/errors.ts` |
-| `JobTenantRequiredError` | `X_JOB_TENANT_REQUIRED` | `src/errors.ts` |
-| `JobEventInvalidError` | `X_JOB_EVENT_INVALID` | `src/errors-steps.ts` |
+| `JobNotRequeueableError` (`./adapter`) | `X_JOB_NOT_REQUEUEABLE` | `src/errors-requeue.ts` |
+| `JobRowStatusUnknownError` (`./adapter`) | `X_JOB_ROW_STATUS_UNKNOWN` | `src/errors.ts` |
 | `JobTenantMismatchError` | `X_JOB_TENANT_MISMATCH` | `src/errors-tenant.ts` |
 | `JobTimeoutError` | `X_JOB_TIMEOUT` | `src/errors.ts` |
 | `LeaseLostError` | `X_JOB_LEASE_LOST` | `src/errors.ts` |
-| `OutboxNoTxError` | `X_OUTBOX_NO_TX` | `src/errors.ts` |
-| `StepDuplicateError` | `X_STEP_DUPLICATE` | `src/errors.ts` |
-| `StepOutputUnserializableError` | `X_STEP_OUTPUT_UNSERIALIZABLE` | `src/errors-steps.ts` |
-| `WebhookDeliveryFailedError` | `X_WEBHOOK_DELIVERY_FAILED` | `src/webhook-errors.ts` |
-| `WebhookDeliveryRejectedError` | `X_WEBHOOK_DELIVERY_REJECTED` | `src/webhook-errors.ts` |
-| `WebhookDeliveryThrottledError` | `X_WEBHOOK_DELIVERY_THROTTLED` | `src/webhook-errors.ts` |
-| `WebhookEndpointDisabledError` | `X_WEBHOOK_ENDPOINT_DISABLED` | `src/webhook-errors.ts` |
-| `WebhookEndpointInvalidError` | `X_WEBHOOK_ENDPOINT_INVALID` | `src/webhook-errors.ts` |
-| `WebhookEndpointUnknownError` | `X_WEBHOOK_ENDPOINT_UNKNOWN` | `src/webhook-errors.ts` |
-| `WebhookEventInvalidError` | `X_WEBHOOK_EVENT_INVALID` | `src/webhook-errors.ts` |
-| `WebhookEventUnknownError` | `X_WEBHOOK_EVENT_UNKNOWN` | `src/webhook-errors.ts` |
 
 ## Boundary
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "30.8.0",
+  "version": "31.0.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
   "keywords": [
     "ultimate",
@@ -26,6 +26,7 @@
   },
   "exports": {
     ".": "./src/index.ts",
+    "./adapter": "./src/adapter.ts",
     "./redis": "./src/driver-redis.ts"
   },
   "files": [
@@ -44,11 +45,11 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "30.8.0",
-    "@ultimat3/db": "30.8.0",
-    "@ultimat3/entity": "30.8.0",
-    "@ultimat3/http": "30.8.0",
-    "@ultimat3/schema": "30.8.0",
-    "@ultimat3/time": "30.8.0"
+    "@ultimat3/core": "31.0.0",
+    "@ultimat3/db": "31.0.0",
+    "@ultimat3/entity": "31.0.0",
+    "@ultimat3/http": "31.0.0",
+    "@ultimat3/schema": "31.0.0",
+    "@ultimat3/time": "31.0.0"
   }
 }

package/src/adapter.ts

@@ -0,0 +1,37 @@
+// `@ultimat3/jobs/adapter` — what a queue, an event bus or a durable store is built and installed
+// from: the port interfaces (outbox, step and lease stores too), a driver's errors, the installers
+// and their resets. Never the `.` barrel: an app swaps a backend in `apps/<app>/runtime.ts`, so
+// the callers here are the cli boot, `@ultimat3/testing` and a driver's own author.
+
+export type {
+  AckOptions,
+  ClaimedJob,
+  ClaimIdentity,
+  ClaimOptions,
+  ConflictPolicy,
+  EnqueueRequest,
+  EnqueueResult,
+  HeartbeatOptions,
+  JobDriver,
+  JobRecord,
+  JobState,
+  NackOptions,
+  QueueDemand,
+  QueueStats,
+  SettleBy,
+} from './driver';
+export { installJobDriver, resetJobDriver } from './driver';
+export {
+  ClaimQueuesEmptyError,
+  DriverUnavailableError,
+  JobDuplicateError,
+  JobRowStatusUnknownError,
+  LeaseLostError,
+} from './errors';
+export { JobNotFoundError, JobNotRequeueableError } from './errors-requeue';
+export type { EventBus, JobEvent, PublishOptions } from './events';
+export { freshDefaultEventBus, installEventBus, resetEventBus } from './events';
+export type { LeaseStore } from './leases';
+export type { OutboxRecord, OutboxStore } from './outbox';
+export { fireThroughDriver } from './scheduler-state';
+export type { StepStore } from './steps-store';

package/src/backfill-ledger.ts

@@ -10,7 +10,7 @@
 // become — the checkpoints are transactional with the work, and this row is not.
 
 import type { Clock } from '@ultimat3/core';
-import { describeValue, finiteCount, systemClock, UltimateError } from '@ultimat3/core';
+import { describeValue, finiteCount, liveClock, UltimateError } from '@ultimat3/core';
 import { nowMs } from './clock';
 
 /**
@@ -153,7 +153,7 @@ export function decideBackfill(
   return { run: force, previous: completed, changed: completed.checksum !== checksum };
 }
 
-export function memoryBackfillLedger(clock: Clock = systemClock): BackfillLedger {
+export function memoryBackfillLedger(clock: Clock = liveClock): BackfillLedger {
   const runs = new Map<string, BackfillRun>();
   const patch = (runId: string, fields: Partial<BackfillRun>): void => {
     const existing = runs.get(runId);

package/src/backfill-pass.ts

@@ -248,7 +248,8 @@ export async function backfillPass<Row>(
               //
               // The signal is the run's cancellation composed with this step's ceiling, so a cancelled
               // pass unwinds out of the wait instead of sitting in a timer nobody is waiting for.
-              await pace.wait({ signal, step: stepName });
+              // The clock is the run's, so a pass under a frozen one waits on it, never in real time.
+              await pace.wait({ signal, step: stepName, clock: args.ctx.clock });
               // A backfill visits every row once, in key order: the scan is the point, and the query-plan
               // sweep is told so rather than asked to index a column no read filters on.
               const page = await expectedSeqScan(BACKFILL_SCAN, () => reader.read(cursor, signal));

package/src/backfill-rate.ts

@@ -3,7 +3,7 @@
 // serving, so a backfill that runs flat out is an outage of its own making — the pacer is what
 // makes "one pass over the table" background work rather than a load test.
 
-import type { Clock } from '@ultimat3/core';
+import type { Clock, FrozenClock } from '@ultimat3/core';
 import { assert, systemClock } from '@ultimat3/core';
 import { nowMs } from './clock';
 import { JobAbortedError } from './errors';
@@ -33,10 +33,12 @@ export interface PacerOptions {
   readonly rate: number;
   /** The name a cancellation is reported under — one pacer belongs to exactly one backfill. */
   readonly job: string;
+  /** Overrides the run's clock for every wait. Unset: each wait reads the clock it is handed. */
   readonly clock?: Clock | undefined;
   /**
    * The wait itself, injectable for the same reason the clock is: a test asserts what was ASKED
    * for instead of spending it. Must settle when `signal` aborts rather than waiting the rest out.
+   * Unset: a frozen clock is advanced by the wait, any other is waited out on a real timer.
    */
   readonly sleep?: ((ms: number, signal: AbortSignal) => Promise<void>) | undefined;
 }
@@ -48,7 +50,15 @@ export interface Pacer {
    * Hold the caller until this batch's slot comes round. Rejects with `JobAbortedError` when the
    * attempt was cancelled — before the wait or during it, never after sitting the timer out.
    */
-  wait(args: { readonly signal: AbortSignal; readonly step?: string | undefined }): Promise<void>;
+  wait(args: {
+    readonly signal: AbortSignal;
+    readonly step?: string | undefined;
+    /**
+     * The clock the RUN measures by — `ctx.clock`. The pacer is built once per declaration, so
+     * the run's clock can only reach it here; a pass under a frozen clock is paced on that clock.
+     */
+    readonly clock?: Clock | undefined;
+  }): Promise<void>;
 }
 
 /** Real timers, cleared on abort so a cancelled pass leaves nothing pending behind it. */
@@ -64,6 +74,24 @@ function timerSleep(ms: number, signal: AbortSignal): Promise<void> {
   });
 }
 
+/** Structural, as `Clock` itself is: a clock that only moves when told to has `advance`. */
+const isFrozenClock = (clock: Clock): clock is FrozenClock =>
+  typeof (clock as { readonly advance?: unknown }).advance === 'function';
+
+/**
+ * The default wait, on the clock the batches are measured by. A frozen clock never moves on its
+ * own, so a real timer under one spent wall time the pacer could not even see — every batch read
+ * as instant and paid the whole interval for real. Moving the clock IS the wait there: the sweep
+ * finishes at once and the clock shows the time the throttle cost.
+ */
+function clockSleep(clock: Clock): (ms: number, signal: AbortSignal) => Promise<void> {
+  if (!isFrozenClock(clock)) return timerSleep;
+  return (ms) => {
+    clock.advance(ms);
+    return Promise.resolve();
+  };
+}
+
 /**
  * Built once, at declaration, and shared by every attempt of that backfill in this process: the
  * rate is a property of the table being swept and the pool it is swept through, not of whichever
@@ -79,14 +107,15 @@ export function backfillPacer(options: PacerOptions): Pacer {
     `backfillPacer({ rate: ${String(options.rate)} }) for "${options.job}" — a rate is batches per second, greater than zero`,
     `pass rate: ${DEFAULT_BACKFILL_RATE} — to sweep faster raise the number, there is no unthrottled mode`,
   );
-  const clock = options.clock ?? systemClock;
-  const sleep = options.sleep ?? timerSleep;
   const intervalMs = 1000 / options.rate;
   let lastAt: number | undefined;
 
   return {
     rate: options.rate,
-    async wait({ signal, step }) {
+    async wait({ signal, step, clock: runClock }) {
+      // Real time by default: the pacer's wait is a real timer between batches, not a reading of now.
+      const clock = options.clock ?? runClock ?? systemClock;
+      const sleep = options.sleep ?? clockSleep(clock);
       const aborted = (): JobAbortedError =>
         new JobAbortedError({ job: options.job, ...(step === undefined ? {} : { step }) });
       if (signal.aborted) throw aborted();

package/src/clock.ts

@@ -2,13 +2,13 @@
 // injected Clock, so tests never sleep and a frozen clock cannot be bypassed.
 
 import type { Clock } from '@ultimat3/core';
-import { finiteOption, systemClock } from '@ultimat3/core';
+import { finiteOption, liveClock } from '@ultimat3/core';
 import { parseDuration } from '@ultimat3/time';
 
 export type DurationInput = string | number;
 
 /** A Clock may hand back a `Date` or epoch ms; scheduling needs one comparable number. */
-export function nowMs(clock: Clock = systemClock): number {
+export function nowMs(clock: Clock = liveClock): number {
   const reading: unknown = clock.now();
   if (reading instanceof Date) return reading.getTime();
   return Number(reading);

package/src/drain-wait.ts

@@ -92,6 +92,7 @@ export async function settleAllBy(
     // timer on a macrotask.
     const unwatch = budget.watch((deadlineAt) => {
       if (timer !== undefined) clearTimeout(timer);
+      // Real time, not `liveClock`: this races a real `setTimeout`, and a frozen clock under it never fires.
       timer = setTimeout(() => decide(false), Math.max(0, deadlineAt - systemClock.monotonic()));
       timer.unref?.();
     });

package/src/driver-memory.ts

@@ -3,7 +3,7 @@
 // real claim/ack/nack paths rather than a mock that always succeeds.
 
 import type { Clock } from '@ultimat3/core';
-import { finiteCount, systemClock, uuidV7 } from '@ultimat3/core';
+import { finiteCount, liveClock, uuidV7 } from '@ultimat3/core';
 import type { BackfillLedger } from './backfill-ledger';
 import { memoryBackfillLedger } from './backfill-ledger';
 import { nowMs } from './clock';
@@ -66,7 +66,7 @@ export interface MemoryJobDriverOptions {
 export type MemoryJobDriver = JobDriver & { close(): Promise<void> };
 
 export function memoryJobDriver(options: MemoryJobDriverOptions = {}): MemoryJobDriver {
-  const clock = options.clock ?? systemClock;
+  const clock = options.clock ?? liveClock;
   const stepStore = options.steps ?? memoryStepStore();
   // `SQL_STEP_PUT`'s fence: a write made under a claim lands only while that claim holds the row.
   const steps: StepStore = {

package/src/driver-pg-ddl.ts

@@ -180,6 +180,11 @@ create table if not exists x_job_leases (
 
 create index if not exists x_job_leases_expiry_idx on x_job_leases (expires_at);
 
+-- The fencing number every grant takes, from ONE sequence: increasing across holders and keys,
+-- and a count the database keeps, never a client clock. A table created before it gains the column.
+create sequence if not exists x_job_lease_fence;
+alter table x_job_leases add column if not exists fence bigint;
+
 -- Events step.waitForEvent consumes. Stored and not broadcast: the publisher is a web pod and
 -- the resumer is a worker pod, so an in-heap bus strands every waiting run in a real deployment.
 create table if not exists x_job_events (
@@ -194,6 +199,14 @@ create table if not exists x_job_events (
 create index if not exists x_job_events_lookup_idx
   on x_job_events (name, published_at);
 
+-- The transaction that published the event. A wait records what it could see when it began and
+-- also matches every event that snapshot could not, since a publish is stamped at insert and seen
+-- at commit. Null on a row from before the column, which is matched by its stamp alone. Added by
+-- alter because x_job_events shipped without it.
+alter table x_job_events add column if not exists xact xid8;
+
+create index if not exists x_job_events_xact_idx on x_job_events (name, xact);
+
 -- The steps parked on an event, which is what a waking publish reads. Partial, so it holds only
 -- the open waits: without it each publish is a scan of every step ever written.
 create index if not exists x_job_steps_waiting_idx
@@ -222,6 +235,12 @@ create index if not exists x_jobs_finished_idx
   on x_jobs (updated_at)
   where state in ('done', 'failed', 'dead', 'cancelled');
 
+-- The done half of the retention sweep, on its own. Done rows are kept a shorter window than
+-- failed, dead and cancelled ones, so a walk of the index above in updated_at order stepped over
+-- every younger dead row between the old done ones on every batch. Partial over the one state, so
+-- the sweep seeks to its cutoff and every row it reads is a row it deletes.
+create index if not exists x_jobs_done_idx on x_jobs (updated_at) where state = 'done';
+
 -- What the queue depth read (SQL_STATS) reads, every 15 seconds from every worker. Partial over
 -- the six states it counts, so the read is a range over the counted rows and never a pass over the
 -- finished ones the retention window keeps. Its predicate is the same list SQL_STATS filters on,