@ultimat3/jobs 29.4.0 → 30.1.0

package/README.md

@@ -3,6 +3,8 @@
 Durable background work. Steps that replay, a transactional outbox, and one driver interface so
 a job's code never names a backend.
 
+`bun add @ultimat3/jobs` · guide: [Jobs And Workflows](https://github.com/developerz-ai/ultimate/wiki/Jobs-And-Workflows)
+
 ```ts
 import { job, t } from '@ultimat3/jobs';
 
@@ -36,6 +38,7 @@ nightlyDigest.describe();                         // { kind: 'task', cron, tz, j
 | Member | On | Behaviour |
 |---|---|---|
 | `enqueue(input, options?)` | `JobHandle` | queues through `jobsFacade()`; joins the caller's `tx` when the outbox is installed. Answers `{ id, runId, deduped }` — the job's own ids on every path, staged or direct. `options.runId` names the run instead of having one minted |
+| `enqueueMany(inputs, options?)` | `JobHandle` | many of one job: one round trip per 1,000 inputs (a statement on Postgres, a script on Redis) on the same path `enqueue` takes, answered in input order. At most `MAX_ENQUEUE_BATCH` (10,000, else `X_JOB_ENQUEUE_BATCH_TOO_LARGE`); `EnqueueManyOptions` is `EnqueueOptions` without `runId`. A loop of `enqueue` is `X_N_PLUS_ONE_WRITE`, whose fix names this call. `JobDriver.enqueueMany` is required of every driver (it opens with `assertEnqueueMany`: `onConflict: 'error'` is refused in a batch); `OutboxStore.stageMany` / `markPublishedMany` are optional: absent, the facade and the relay loop |
 | `as(actor, input, options?)` | `JobHandle` | enqueue on behalf of an actor — fills `tenantId` from `actor.orgId` so per-tenant limits apply |
 | `describe()` | `JobHandle` | the `JobDescriptor` the manifest, `/_x` and MCP read |
 | `entries()` | `TaskHandle` | the declared `[job, input]` pairs |
@@ -175,6 +178,20 @@ author has nothing to wrap and the pass opens the scope itself. See the backfill
 Omitting the field is a type error, and
 `X_JOB_TENANT_REQUIRED` for generated code and JS callers.
 
+**The declared tenant is checked at the enqueue.** A payload whose `tenant(input)` names org A,
+enqueued with `tenantId` B — explicitly, or by `.as(actorOfB, …)` — or under an ambient actor of
+org B, is `X_JOB_TENANT_MISMATCH` before anything is staged, on `enqueue` and `enqueueMany`. A row
+already queued that way is refused at its run, terminally, before the body. An actor with no org,
+and `tenant: 'none'`, are never compared. The ambient half has one escape, `@ultimat3/entity`'s:
+inside `crossTenant('<why>', () => onboardOrg.enqueue({ orgId: newOrg }))`, by an actor holding
+`tenancy:cross`, the enqueue is admitted (read through entity's own write guard, re-proved at the
+call). An explicit `tenantId` or `.as()` that contradicts the payload is never admitted.
+
+**A row is queued for the org its payload names** when the enqueue states none: a plain
+`enqueue(input)` stamps `tenant(input)`, so two orgs' identical idempotency keys are two jobs and
+`perTenant` / `ratePerTenant` count the org. An explicit `tenantId` or `.as()` still wins — and the
+refusal above guarantees they agree.
+
 A single boot-supplied service actor would have closed the same hole with one identity for every
 job in the app — which is a cross-tenant read waiting for the first job that takes an org id in its
 input. The tenant is a fact about the work, so the job declares it.
@@ -183,10 +200,25 @@ input. The tenant is a fact about the work, so the job declares it.
 
 | Call | Behaviour |
 |---|---|
-| `step.run(name, fn)` | runs once ever; result persisted before the next step starts. `fn` receives an `AbortSignal` |
+| `step.run(name, fn)` | runs once ever; result persisted before the next step starts, and it answers what it persisted: `StepOutput<T>`, the JSON form, on every attempt. `fn` receives an `AbortSignal` |
+| `step.run(name, fn, { compensate })` | `compensate(output, { signal })` undoes it if the run fails FOR GOOD; needs `job({ compensates: true })` |
+| `step.run(name, fn, { once: true })` | at most once: `started` persisted before `fn`; a replay without `completed` is terminal `X_STEP_OUTCOME_UNKNOWN` |
 | `step.sleep(name, '3d')` | suspends the run, requeues it for the wake time |
 | `step.sleep('3d')` | same, step name derived from the duration |
-| `step.waitForEvent(name, event, { match, timeout })` | suspends until `publishEvent()` matches |
+| `step.waitForEvent(name, event, { schema, match, timeout })` | suspends until `publishEvent()` matches; answers the payload parsed by the required `schema` |
+
+| A step answer | Is |
+|---|---|
+| a `Date`, anything with a `toJSON()` | what that returns: `new Date(value)` at the read, a compile error otherwise |
+| `undefined`, a function or a symbol at the top, or in an array | `null`: compare `=== null` |
+| `undefined` or a function on a member | absent |
+| a `bigint`, a cycle | `X_STEP_OUTPUT_UNSERIALIZABLE`, terminal: `String(id)`, the ids rather than the object graph |
+| a stored event the wait's `schema` refuses | `X_JOB_EVENT_INVALID`, terminal, on a hit or a replay. The error carries the issues, never the payload |
+
+A boot's bus runs on the open transaction's connection (`currentTx() ?? client`, as the outbox):
+`publishEvent` inside a transaction commits and rolls back with it. `published_at` is the
+statement's time, so a wait that begins between that publish and its commit compares later than the
+stamp and never matches; the window is the length of the publishing transaction.
 
 `resetEventBus()` is the test seam beside `resetJobDriver()`: an event is matchable until it
 expires, so on a bus shared by one `bun test` process an answer one test published resumes the
@@ -204,10 +236,68 @@ again before declaring a timeout, so a worker running ahead of the database stil
 published after it asked and does not give up early. `eventsPurgeTarget(bus)` is the bus as a
 `purge()` target: `purgeExpired()` is awaited and answers the rows it removed.
 
+**A publish wakes its waiters** on `postgresEventBus({ executor, wakes: true })` — what boot
+installs over its own `postgresJobDriver`: `SQL_EVENT_PUBLISH_WAKE` inserts the event, makes every
+`suspended` run with a matching `waiting` step due, and notifies each of their queues, in ONE
+statement (a rollback undoes all three; `x_job_steps_waiting_idx` keeps it a seek). A waiting step
+then parks for `WAKING_EVENT_POLL_MS` (15 min) — the backstop and the timeout's clock — instead of
+`DEFAULT_EVENT_POLL_MS` (30 s), and asks `find` once more after writing `waiting`, so an event
+committed in between is not left to the backstop. A declared `eventPoll` wins.
+
+### Compensation and `once`
+
+```ts
+import { job, t } from '@ultimat3/jobs';
+
+type Order = { readonly orgId: string; readonly orderId: string };
+declare function charge(order: Order): Promise<{ id: string }>;
+declare function refund(chargeId: string): Promise<void>;
+declare function ship(order: Order): Promise<void>;
+declare const vendor: { pay(order: Order): Promise<void> };
+
+export const checkout = job({
+  input: t.object({ orgId: t.uuid, orderId: t.uuid }),
+  tenant: ({ orgId }) => orgId,
+  idempotencyKey: ({ orderId }) => `checkout:${orderId}`,
+  compensates: true,
+  retry: { attempts: 3 },
+  run: async ({ input, step }) => {
+    await step.run('charge', () => charge(input), { compensate: (c) => refund(c.id) });
+    await step.run('ship', () => ship(input));
+    await step.run('payout', () => vendor.pay(input), { once: true });
+  },
+});
+```
+
+| Rule | Behaviour |
+|---|---|
+| when | only once the run fails for good (`!decision.retry`); never a retry, cancel, drain or suspension |
+| order | last-in first-out, each as step `compensate:<name>`, so a replay skips what ran |
+| durable | `unwinding` is written first; any later attempt replays to its first incomplete step (`UnwindReached`) and unwinds — `x jobs retry` too |
+| crash on the final attempt | rows are enqueued with `UNWIND_MARGIN` (2) attempts past `retry.attempts` (`enqueuedAttempts`); the claim re-offers the lapsed row and it unwinds |
+| a compensator throws | handed back while the unwind has attempts; then dead, `X_JOB_COMPENSATION_FAILED` |
+| `once` unknown | `x jobs resolve-step <id> <step> --ran --output '<json>'` or `--not-ran` (`resolveJobStep`), then `x jobs retry <id>` |
+| reserved | `unwinding`, `compensate:*` as a body's step name: `X_JOB_DECLARATION_INVALID` |
+
 Step names are the replay key, so they must be deterministic and unique in a run — a
 duplicate is `X_STEP_DUPLICATE`, not a silent overwrite. Suspension is control flow
 (`StepSuspension`), never a failure: it does not burn a retry attempt.
 
+What a crash does to an effect left outside a step is proven, not reviewed: `x exercise` kills
+every job it reaches at every step boundary (`crashProof`, `@ultimat3/testing`), and an effect a
+killed run makes more often than a clean one is `X_JOB_CRASH_DIVERGED`, red in `x verify`. A job
+test asks the same with `crashProof(job, input)`.
+[Testing](https://github.com/developerz-ai/ultimate/wiki/Testing#the-crash-proof).
+
+## Rollups and materialized views are jobs
+
+`rollup({ name, into, table, key, lookback, bucket, source, map, tenant })` re-aggregates every
+bucket from the one holding `now - lookback` on, read from that bucket's first instant in
+`bucket.timeZone`, into the app's table with one `upsertAll` on `key` (a unique index on `into`, checked at
+declaration: `X_JOB_ROLLUP_INVALID`). `refreshView({ name, view, concurrently? })` refreshes a
+materialized view entity (`X_JOB_REFRESH_VIEW_INVALID`). Both are factories over `job()`; the
+schedule is the app's `task`. Reference: [Jobs and workflows](https://github.com/developerz-ai/ultimate/wiki/Jobs-And-Workflows#rollups-and-materialized-views).
+
 ## Backfills are jobs
 
 `backfill()` is a **factory over `job()`**, not a ninth primitive — one pass over every row a
@@ -249,6 +339,13 @@ without a statement, and the iteration reopens at the cursor they left behind.
 | `rate` throttles, and there is no way off | a sweep shares its pool with the requests the app is still serving; to go faster raise the number |
 | the pause is spent **inside** the step | a resumed attempt replays 500 checkpoints and re-pays none of their pauses |
 
+### A remote source, and one pass per scope
+
+`pages: async ({ ctx, cursor, signal, input }) => ({ rows, next })` beside `source` — exactly one
+of the two, or `X_BACKFILL_SOURCE_INVALID`. One page per step, checkpointed `{ cursor, rows: n }`;
+`next: null` ends the pass. `input` + `scope(input)` makes one pass per scope: the ledger row and
+the live idempotency key are `<name>:<scope>`. An `input` with no `scope` is refused.
+
 ### The throttle
 
 `rate` is batches per second, defaulting to `DEFAULT_BACKFILL_RATE` (5) — 5,000 rows/sec at the
@@ -376,6 +473,44 @@ the new pods serve, puts the sweeps on the queue and exits, and a slow UPDATE ne
 open against a database still serving the previous build. `--all` isolates per name and continues
 past a failure, so one wedged cleanup cannot block every later one forever.
 
+## Consumers are jobs too
+
+`consumer()` reads a queue, a change stream or a vendor's cursor: a factory over `job()` plus a
+`.supervisor` `task()`. One running body per key, `idempotencyKey: consumer:<name>:<key>`, no
+timeout, the cursor saved in the batch's own transaction, the drain honoured.
+
+```ts
+import { consumer } from '@ultimat3/jobs';
+
+interface Order { readonly id: string }
+declare const feed: { since(cursor: string | null, signal: AbortSignal): AsyncIterable<Order> };
+declare function upsertOrders(rows: readonly Order[]): Promise<void>;
+declare const cursors: {
+  of(key: string): Promise<string | null>;
+  save(key: string, next: string): Promise<void>;
+};
+
+export const syncOrders = consumer<Order, string>({
+  name: 'sync-orders',
+  tenant: 'none',
+  keys: () => ['orders'],
+  open: ({ cursor, signal }) => feed.since(cursor, signal),     // the app's AsyncIterable
+  handle: async ({ items }) => {
+    await upsertOrders(items);
+    return items[items.length - 1]?.id ?? '';
+  },
+  cursor: { load: (key) => cursors.of(key), save: (key, next) => cursors.save(key, next) },
+});
+```
+
+| Rule | Why |
+|---|---|
+| `handle` returns the next cursor; the framework saves it in the same `withTransaction` | cursor and effect commit together or not at all |
+| `.supervisor` (`<name>.supervisor`, every `rearm`, UTC) enqueues `<name>.arm`, which enqueues each of `keys()` | a live key's re-arm is deduped; a settled or dead one restarts from its saved cursor |
+| a self-`.enqueue` from inside the run is `X_CONSUMER_SELF_ENQUEUE` | it would dedupe onto the running row and arm nothing |
+| no `cursor`, a `timeout`, a bad `batch` or `rearm` is `X_CONSUMER_INVALID` | refused at declaration |
+| at least once per item | a drained or failed batch is delivered again: `handle` is idempotent per item |
+
 ## Retention sweeps are jobs too
 
 `purge()` is the **second factory over `job()`**, and it exists because three framework stores
@@ -503,6 +638,91 @@ exports/<orgId>/<exportId>/manifest.json
 this package holds no storage dependency, and taking one so a queue could name a disk would put the
 object store on tier 3's import graph.
 
+## Importing a large file is a job too
+
+`importRows()` is `exportRows()`'s twin: a factory over `job()` that reads one object from storage
+a page at a time, through the app's schema, into the app's `handle`. It exists because
+`parse(await disk.get(key))` holds the whole upload, and a retried job re-applies every row from
+zero. Which columns mean what, and who may import, stay the app's.
+
+```ts
+import { importRows, t } from '@ultimat3/jobs';
+import { disk } from '@ultimat3/storage';
+
+interface Order {
+  readonly id: string;
+  readonly orgId: string;
+  readonly total: number;
+}
+
+// The app's own table: `handle` takes whatever `db.orders.upsertAll(…)` answers.
+declare const db: {
+  readonly orders: {
+    upsertAll(rows: readonly Order[], args: { onConflict: readonly string[] }): Promise<unknown>;
+  };
+};
+
+export const importOrders = importRows({
+  name: 'orders.import',
+  input: t.object({ orgId: t.uuid, key: t.string }),
+  tenant: ({ orgId }) => orgId,
+  source: () => disk('imports'), // a thunk; a StorageDriver IS an ImportSource
+  key: ({ key }) => key,
+  format: 'csv',
+  columns: ['id', 'total'],
+  schema: t.object({ id: t.uuid, total: t.number.int() }),
+  handle: ({ rows, input }) =>
+    db.orders.upsertAll(rows.map((row) => ({ ...row, orgId: input.orgId })), {
+      onConflict: ['orgId', 'id'],
+    }),
+  errors: { sink: () => disk('imports'), prefix: ({ orgId }) => `imports/${orgId}/errors` },
+});
+```
+
+| Rule | Why |
+|---|---|
+| one step per page; its output `{ offset, line, rows, rejected, errorPart, header }` is the checkpoint | a killed attempt resumes at the byte offset. Never the rows: a step's output is retained for the whole run |
+| no carry between steps | bytes after the last complete record are re-read by the next page; a record boundary is found on BYTES (`0x0A` outside `"`), which no UTF-8 sequence contains |
+| `handle` runs at least once per page | `upsertAll` is the idempotent spelling, as for `backfill()` |
+| `pageBytes` (1 MiB) must exceed `maxRecordBytes` (64 KiB) | so every window holds a whole record; past `maxRecordBytes` is `X_IMPORT_RECORD_TOO_LARGE` |
+| a page uses its whole `pageBytes` read, handed to `handle` `batch` (500) records at a time | bytes read are O(file) and steps about file ÷ `pageBytes`, while one call is still one bounded statement. Measured: a 10 MB file of `a\n` rows is 10 reads, 10,000,004 bytes and 10 page steps |
+| csv: header = `columns`, any order; empty cell `null`, quoted `""` the empty string; cells coerced as an HTTP form is (`coerceInput`) | `X_IMPORT_HEADER_MISMATCH` names missing and undeclared |
+| ndjson: a key outside `columns` refuses the row | `record.undeclared`, naming the key |
+| a refused row is `{ line, field, code }` in `<prefix>/<runId>/errors-NNNNN.ndjson` (`importErrorKey`); the result's `errorPrefix` names it, so a re-import never mixes runs | the code is a format rule (`IMPORT_REJECTIONS`) or the schema's `rule`; **never the value** |
+| past `maxRowErrors` (1,000) the run fails `X_IMPORT_TOO_MANY_ERRORS` | the `batch` that crossed it unhandled, its report written; `0` stops at the first bad batch |
+| the report is `{ rows, rejected, parts, bytes, errorPrefix }` | bounded, for `x jobs show` |
+| a csv with no first line — empty, a BOM, blank lines — is `X_IMPORT_HEADER_MISMATCH` | zero rows from a file that never named its columns is not a success |
+| a header mismatch prints the DECLARED names whole and the file's as a count plus five escaped, cut names | a header cell is file content: raw, it forges a log line or prompts the agent reading `x jobs show` |
+| `tenant: 'none'` gets no cross-tenant escape | a scoped write in `handle` fails closed with `X_TENANCY_ACTOR_ORG_REQUIRED` |
+| no formula guard is undone | an import cannot know a leading `'` was the exporter's |
+
+Measured `As of 2026-11-03` (`pageBytes` 1 MiB, `batch` 5,000): a 200 MB csv imported with the
+heap up 42 MB, a 20 MB one with it up 24 MB — a page and the garbage behind it, not the file.
+`ImportSource` is a seam for `ExportSink`'s reason; `scripts/export-sink.test.ts` compiles
+`StorageDriver` into both. `memoryImportSource()` is the test source.
+
+## One person's data is a job too
+
+`defineSubject(entity, { driver, key, tables, targets?, columns?, batch? })` declares the person
+once; `subjectExport({ name, input, tenant, subject, sink, prefix, format })` and
+`subjectErase({ name, input, tenant, subject })` are factories over `job()` that take it. The
+reach is planned when a run starts (`planSubject`, pure), from every registered entity's foreign
+keys, continued only through tables the erase deletes; the first refusal of the declaration
+throws before a row is touched (`X_SUBJECT_TABLE_UNDECLARED`, `X_SUBJECT_TREATMENT_INVALID`,
+`X_SUBJECT_CYCLE`, all terminal).
+
+| Piece | Does |
+|---|---|
+| `subject-plan.ts` | reach, children-first order, loop-closing keys to clear first, every refusal |
+| `subject-select.ts` | a table's rows of the subject a page of parent keys at a time, through `database()` over the app's driver |
+| `subject-erase-pass.ts` | targets (fresh subject row each step), cleared keys, then each table; one step per page, each write `deleteWhere`/`updateWhere` on one key |
+| `subject-export-pass.ts` | one object per page per table and target, `index.json` last; sealed columns left out, instants as ISO UTC |
+| `subjectEraseTables(subject, key, step)` | the table passes alone, under the caller's context — what `@ultimat3/testing`'s `subjectProof` drives |
+
+`SubjectTarget` (`name`, `columns`, `exportPage`, `erase`) is structural, as `PurgeTarget` is:
+`authSubjectTarget`, `auditSubjectTarget` and `storageSubjectTarget` satisfy it from their own
+packages. `{ target, retain: '<why>' }` exports a target and never erases it.
+
 ## Outbound webhooks are jobs too
 
 `webhook()` is a factory over `job()` and delivers **one event to one endpoint**. Retry with
@@ -591,6 +811,51 @@ default: past it the worker logs `jobs.worker.drain-abandoned` and exits, and th
 only when its lease lapses, as a COUNTED attempt the receiver may already have processed.
 `timeout: '10s'` against the default budget ends every drained delivery inside it.
 
+## Inbound webhooks are jobs too
+
+`webhookReceiver()` is the twin of `webhook()`: a factory over `job()` plus the plain `Route` that
+feeds it. The route reads the exact bytes under `maxBytes`, asks your `verify`, enqueues the
+payload it answered with `outbox: false`, and only then answers `200 { ok: true, eventId,
+deduped }`. `handle` is the job's body and receives that payload, already verified. **No vendor
+verifier ships**: `verify` is a plain function, `verifyBodyHmac` from `@ultimat3/http` the one
+helper. **Declaring it is its registration**: list it in `defineApi({ jobs })`, and both boots mount
+its routes (`webhookReceiverRoutes()`), with no `runtime.ts` line.
+
+```ts
+import { verifyBodyHmac } from '@ultimat3/http';
+import { webhookReceiver } from '@ultimat3/jobs';
+
+export const vendorEvents = webhookReceiver({
+  name: 'vendor.events',
+  path: '/hooks/vendor/:account',
+  tenant: 'none',
+  verify: (inbound) => {
+    const { text } = verifyBodyHmac({
+      bytes: inbound.bytes,
+      secret: Bun.env['VENDOR_SECRET'] ?? '',
+      given: inbound.headers.get('x-vendor-signature'),
+      encoding: 'hex',
+    });
+    // The id from the SIGNED bytes: an unsigned header would replay one signature forever.
+    const id = /"id":"([^"]{1,256})"/.exec(text ?? '')?.[1] ?? '';
+    return { eventId: id, payload: { text } };
+  },
+  handle: async ({ eventId, payload }) => ({ eventId, payload }),
+});
+```
+
+| Rule | Why |
+|---|---|
+| nothing is parsed before `verify`, and no content type is required | a schema that answers first turns a vendor's shape change into a refusal, and XML, `octet-stream` or no type would never reach the check |
+| `verify` answers `{ eventId, payload }` after the mac check, and `handle` receives the payload | a retry an hour later must not re-verify against a 5-minute window; what the queue holds was verified once, at the door |
+| `eventId` comes from verified data, 1–256 printable ASCII characters | it is the dedupe key, so an unsigned one replays a captured signature under fresh ids. Anything else is `X_WEBHOOK_EVENT_ID_INVALID` (422), never echoed, never a 5xx a vendor retries forever |
+| the ack follows the durable row | `outbox: false`: the row is written, not staged behind a commit. An enqueue failure is 503 + `Retry-After` (`X_WEBHOOK_RECEIVER_UNAVAILABLE`), so the vendor redelivers |
+| dedupe is the live window | a live redelivery answers `deduped: true`. After the job settles, `handle` runs again: at least once, and the seen-set is your table — a claim before the enqueue would lose the event if the enqueue failed |
+| the payload is JSON, `.sensitive()` | `x jobs show` prints `[redacted]`. It stays until `jobs.doneRetentionMs` / `deadRetentionMs` |
+| `verify` may answer `{ answer: Response }`, and `handshake` mounts `GET` | a subscription confirm or a challenge is the vendor's protocol; nothing queues |
+| `.routes()` is set on the job handle in place, never on a copy | `isJobHandle` proves the object `job()` built, so `defineApi({ jobs })` registers it like any job |
+| the path is absolute and outside `/api` and `/_x` | `X_WEBHOOK_RECEIVER_INVALID` at declaration, beside a missing `verify`/`handle` or a bad `maxBytes` |
+
 ## The deadline cancels
 
 A job's `timeout` aborts `ctx.signal` **before** it fails the attempt, because the nack that
@@ -636,7 +901,8 @@ them, so no `job()` could ask for either):
 |---|---|---|
 | `timeout` | the whole attempt — aborts `ctx.signal`, then fails it | no attempt deadline |
 | `stepTimeout` | ONE `step.run` — aborts that step's signal, then fails the step | no per-step ceiling |
-| `eventPoll` | how long a `step.waitForEvent` parks between polls | 30s |
+| `eventPoll` | how long a `step.waitForEvent` (its `schema` required) parks between polls | 30s; 15 min on a waking bus |
+| `compensates` | `step.run(…, { compensate })` is legal; rows carry `UNWIND_MARGIN` more attempts | false |
 
 A zero or negative `stepTimeout` / `eventPoll` is refused at declaration, the way `concurrency: 0`
 is: `withStepTimeout` reads `<= 0` as "no ceiling at all", which is the opposite of what the author
@@ -772,14 +1038,14 @@ between them — swapping is `setJobDriver(other)`, and there is **no `jobs.driv
 | Driver | Status | Backing | Use |
 |---|---|---|---|
 | `pg` | **default** | `SELECT ... FOR UPDATE SKIP LOCKED`, a partial unique index on `(name, coalesce(tenant_id, ''), idempotency_key)` over live rows (`x_jobs_name_tenant_idempotency_live_idx`), lease-based leader, `x_job_leases` | zero-infra start, most apps |
-| `redis` | complete, `As of 2026-10` | `Bun.redis`; one Lua script per operation, every key in one `{hash tag}` slot | a queue off the database; no `introspect`, `backfills` or `leases` |
+| `redis` | complete, `As of 2026-10` | core's `redisClient()` (`@ultimat3/core/redis`); one Lua script per operation, every key in one `{hash tag}` slot | a queue off the database; no `introspect`, `backfills` or `leases` |
 | `memory` | complete | in-process maps | `x dev`, tests |
 
 **`redisJobDriver({ client?, prefix?, clock?, doneTtlMs? })`** passes `@ultimat3/testing`'s
 `jobDriverConformance`, the suite the other two pass, and `driver-redis.live.test.ts` runs one
 script of operations on it and on the memory driver and compares every answer. Installed with
 `setJobDriver(redisJobDriver())` (or `ServeOptions.runtime.jobs`), imported from its own entry
-`@ultimat3/jobs/redis` so the barrel every role boots carries none of it; `Bun.redis` reads `REDIS_URL`.
+`@ultimat3/jobs/redis` so the barrel every role boots carries none of it; the client reads `REDIS_URL`.
 What it does not carry: `introspect` (so `x jobs show`/`retry`/`cancel` and queue pauses answer
 for Postgres only), `backfills`, and `leases` — a job declaring `concurrency` refuses
 `jobWorker().start()` (`X_JOB_CONCURRENCY_UNENFORCEABLE`). A `done` row and its steps expire
@@ -787,6 +1053,16 @@ after `doneTtlMs` (a week; `0` keeps them); `dead` and `failed` rows are kept. T
 stages in Postgres — the relay publishes committed rows onto it with their ids. The NATS stub
 deleted in 25.0.0 stays deleted.
 
+**Redis down refuses, it never waits** `As of 2026-11`. The driver takes core's `redisClient()`,
+not `Bun.redis`: no offline queue, so a command while disconnected is refused at once, and a
+reconnect loop that never gives up, so a restarted server is back on every pod with no restart.
+`Bun.redis` blocked an `enqueue` while it retried (31 s, measured on a stopped server), then
+stayed closed after its retries ran out. A refusal for want of a connection (`ERR_REDIS_CONNECTION_CLOSED`) is
+`X_DRIVER_UNAVAILABLE` (`driver-redis-refusal.ts`), on `enqueue` and on `claim` alike; nothing was
+queued, so the same call is safe to repeat. Catch it where a queued retry was implied. A server's
+own error reply passes through unmapped. There is no per-command deadline here — a stuck command
+has the request's. A `client` handed in keeps its own behaviour and still gets the mapping.
+
 The pg SQL lives verbatim in `src/driver-pg-*sql.ts` (`SQL_CLAIM`, `SQL_ENQUEUE`, `SQL_NACK`, …)
 so an agent debugging a stuck queue can read and run the exact statement. The barrel exports
 only the statement another package runs: `SQL_JOBS_TABLE` (the install) — 25.0.0 took the rest
@@ -918,7 +1194,7 @@ export const executor: PgExecutor = dbExecutor(() => client);
 | Not this | Because |
 |---|---|
 | `{ executor: Bun.sql }` | `Bun.sql` is a tagged template with no `.query(text, values)`: the first claim is a `TypeError` |
-| a hand adapter over raw `Bun.SQL`'s `unsafe(text, values)` | observed `As of 2026-10`: a `$n::jsonb` parameter is encoded twice unless the client is built with `prepare: false`, and a JS array is refused where the statement expects `text[]` (`SQL_CLAIM`'s `queue = any($1::text[])`) unless it is written as an array literal. `@ultimat3/db`'s client handles both |
+| a hand adapter over raw `Bun.SQL`'s `unsafe(text, values)` | observed `As of 2026-10`: a `$n::jsonb` parameter is encoded twice unless the client is built with `prepare: false`, and a JS array is refused where the statement expects `text[]` (`SQL_CLAIM`'s `unnest($1::text[])`) unless it is written as an array literal. `@ultimat3/db`'s client handles both |
 
 ## Idle cost and pickup latency
 
@@ -1066,10 +1342,11 @@ queue slots is the bound, as before.
 What a scaler reads, per process: `worker.stats()` → `inFlight` and `maxInFlight` (`null` when
 uncapped); the registry row's `concurrency` is the tighter of the cap and the slot sum; and two
 unlabelled gauges, `job_worker_in_flight` and `job_worker_capacity`, each the SUM over the live workers in the process (a stopped worker leaves both). Pods needed is
-`ceil((queue_depth + Σ job_worker_in_flight) / job_worker_capacity)`. A fleet-wide ceiling is a
+`ceil(queue_demand / job_worker_capacity)`, with the demand deduplicated: `max by (queue) (queue_demand)`. A fleet-wide ceiling is a
 different number: `maxInFlight` × replicas, or `job.concurrency` for one job.
 
-**A scaler reading Postgres** (KEDA's `postgresql` trigger) runs the statement below — the
+**A scaler reading Postgres** (KEDA's `postgresql` trigger) runs the statement below — the rows
+`queue_demand` counts per queue (`driver-pg-demand-sql.ts`), in the
 package's `scaler-depth-fixture.ts`, proved against a real Postgres (`scaler-depth.job.test.ts`)
 and pinned to this block (`scaler-depth-readme.test.ts`); KEDA binds
 nothing, so the queues are inlined as `'{a,b}'::text[]`: running jobs plus due,
@@ -1206,7 +1483,7 @@ the pg and the memory driver alike. `x jobs`, `/_x` and a dashboard read nothing
 | `promote(id)` | a job waiting on its `runAt` — delayed, or backing off — is due now | — |
 | `pauseQueue(q)` / `resumeQueue(q)` / `pausedQueues()` | a paused queue is claimed by no worker; enqueues still land | effective at every worker's next claim |
 | `pauseTask(t)` / `resumeTask(t)` / `pausedTasks()` | a paused task is not dispatched; on resume its own `catchUp` decides what it missed | — |
-| `workers()` | every live worker: id, host, started, queues, slots, in-flight job ids, last heartbeat, `drainingAt` once it drains | rows expire one visibility timeout after the last heartbeat; `MAX_WORKER_IN_FLIGHT` = 100 ids |
+| `workers()` | every live worker: id, host, `build` when `jobWorker({ build })` named one, started, queues, slots, in-flight job ids, last heartbeat, `drainingAt` once it drains | rows expire one visibility timeout after the last heartbeat; `MAX_WORKER_IN_FLIGHT` = 100 ids |
 | `counters({ job, sinceMs })` / `counterTotals(sinceMs)` | per-job history: `done`, `retried`, `failed`, `dead`, `durationMs` per bucket | `COUNTER_TIERS`: 1-minute buckets for 24 h, 5-minute for 7 d, 1-hour for 30 d — 4,176 rows per job name at most |
 | `rollupCounters()` | fold buckets past their tier into the next; the scheduler leader calls it once a minute | idempotent |
 | `recordProgress(id, { workerId, claim }, progress)` | what `run({ progress })` writes | at most one write per `PROGRESS_INTERVAL_MS` (1 s) per run, plus the last before the settle |
@@ -1287,6 +1564,10 @@ export const syncAccountWatched = job({
 last `progress`), `inspectDeadLetters`, `retryFromStep`, `cancelJob`, `removeJob`, `promoteJob`,
 `pauseQueue`, `resumeQueue`, `inspectManifest` — all `--json`-shaped, shared by `/_x`, the CLI and the MCP tools.
 
+`inspectJob`'s `input` masks every `.sensitive()` field of the job's own `input:` schema, then every
+`redactKeys` key. A job this process has not registered — its declaration deleted or renamed —
+answers `input: '[redacted]'` whole: with no schema nothing can say which keys were sensitive.
+
 `cancelJob(driver, id, reason?)` is the answer to a runaway pass. A queued row becomes `cancelled`
 immediately; a RUNNING one stops at its next heartbeat, which no longer matches its own row and
 aborts the attempt — `steps.ts` then refuses every write. `ack` and `nack` are fenced on
@@ -1296,7 +1577,8 @@ aborts the attempt — `steps.ts` then refuses every write. `ack` and `nack` are
 
 | Series | Kind | Answers |
 |---|---|---|
-| `queue_depth{queue}` | gauge | how much work is waiting — the HPA's signal |
+| `queue_depth{queue}` | gauge | how much work is waiting: the backlog alert's series. It reads 0 while a fleet holds every job it has, so nothing scales on it |
+| `queue_demand{queue}` | gauge | what serving workers must hold: running on one, or due and claimable. The chart's worker HPA reads it. `JobDriver.demand()`, asked only when `stats()` shows work, required of every driver; Redis has no worker registry, so it counts every running row. A draining worker publishes 0 |
 | `jobs_total{queue,outcome}` | counter | is any of it succeeding |
 | `queue_oldest_ready_seconds{queue}` | gauge | *"page if the oldest job in `payments` is older than 5 minutes"* |
 | `queue_dead_jobs{queue}` | gauge | a dead-letter queue that filled overnight and stopped growing — a counter's rate is flat there |
@@ -1319,6 +1601,8 @@ FOR a user takes that user's id in its input and re-authorises it in the body.
 |---|---|
 | `X_IDEMPOTENCY_REQUIRED` | job defined without an idempotency key (JS callers) |
 | `X_STEP_DUPLICATE` | two steps share a name in one run |
+| `X_STEP_OUTPUT_UNSERIALIZABLE` | a step returned a value JSON cannot hold |
+| `X_JOB_EVENT_INVALID` | a waited event failed its schema |
 | `X_JOB_DUPLICATE` | enqueue collided with a live key under `onConflict: 'error'`; or two different job/task handles registered under one name |
 | `X_JOB_TIMEOUT` | job or required `waitForEvent` exceeded its timeout |
 | `X_JOB_MAX_ATTEMPTS` | retries exhausted, job dead-lettered |
@@ -1379,11 +1663,13 @@ a job boundary the class is gone and the `code` is what survives — match on th
 | `JobRowStatusUnknownError` | `X_JOB_ROW_STATUS_UNKNOWN` | `src/errors.ts` |
 | `JobSlotLostError` | `X_JOB_SLOT_LOST` | `src/errors.ts` |
 | `JobTenantRequiredError` | `X_JOB_TENANT_REQUIRED` | `src/errors.ts` |
+| `JobEventInvalidError` | `X_JOB_EVENT_INVALID` | `src/errors-steps.ts` |
 | `JobTenantMismatchError` | `X_JOB_TENANT_MISMATCH` | `src/errors-tenant.ts` |
 | `JobTimeoutError` | `X_JOB_TIMEOUT` | `src/errors.ts` |
 | `LeaseLostError` | `X_JOB_LEASE_LOST` | `src/errors.ts` |
 | `OutboxNoTxError` | `X_OUTBOX_NO_TX` | `src/errors.ts` |
 | `StepDuplicateError` | `X_STEP_DUPLICATE` | `src/errors.ts` |
+| `StepOutputUnserializableError` | `X_STEP_OUTPUT_UNSERIALIZABLE` | `src/errors-steps.ts` |
 | `WebhookDeliveryFailedError` | `X_WEBHOOK_DELIVERY_FAILED` | `src/webhook-errors.ts` |
 | `WebhookDeliveryRejectedError` | `X_WEBHOOK_DELIVERY_REJECTED` | `src/webhook-errors.ts` |
 | `WebhookDeliveryThrottledError` | `X_WEBHOOK_DELIVERY_THROTTLED` | `src/webhook-errors.ts` |
@@ -1397,3 +1683,11 @@ a job boundary the class is gone and the `code` is what survives — match on th
 
 Tier 3. Imports `@ultimat3/core`, `@ultimat3/schema`, `@ultimat3/entity`,
 `@ultimat3/policy`, `@ultimat3/cache`, `@ultimat3/time`. Never HTTP, render or UI.
+
+## Supporting exports
+
+`As of 2026-11-02`.
+
+| Name | What |
+|---|---|
+| `ON_SETTLED_TIMEOUT_MS` | 30,000: the `onSettled` ceiling across every try unless the job declares `onSettledTimeout` |

package/package.json

@@ -1,7 +1,18 @@
 {
   "name": "@ultimat3/jobs",
-  "version": "29.4.0",
+  "version": "30.1.0",
   "description": "Durable background work: steps, transactional outbox, cron tasks, one driver interface",
+  "keywords": [
+    "ultimate",
+    "bun",
+    "framework",
+    "jobs",
+    "queue",
+    "workflows",
+    "cron",
+    "outbox"
+  ],
+  "homepage": "https://github.com/developerz-ai/ultimate/wiki/Jobs-And-Workflows",
   "license": "MIT",
   "type": "module",
   "repository": {
@@ -33,10 +44,11 @@
     "test": "bun test"
   },
   "dependencies": {
-    "@ultimat3/core": "29.4.0",
-    "@ultimat3/db": "29.4.0",
-    "@ultimat3/entity": "29.4.0",
-    "@ultimat3/schema": "29.4.0",
-    "@ultimat3/time": "29.4.0"
+    "@ultimat3/core": "30.1.0",
+    "@ultimat3/db": "30.1.0",
+    "@ultimat3/entity": "30.1.0",
+    "@ultimat3/http": "30.1.0",
+    "@ultimat3/schema": "30.1.0",
+    "@ultimat3/time": "30.1.0"
   }
 }