@fullstackhouse/open-mercato-durable-work 0.1.0

package/README.md

@@ -0,0 +1,136 @@
+# @fullstackhouse/open-mercato-durable-work
+
+Durable, at-least-once background work for [Open Mercato](https://github.com/open-mercato/open-mercato).
+
+A job is a row in your database with a lease on it. A worker claims the lease, does a slice of
+the work, and hands the rest back; if that worker dies, the lease expires and a reconciler
+repairs the job. Nothing depends on a process remembering anything, which is what makes a
+deploy, a crash and a network partition the same event.
+
+## What it is for
+
+Work that is too long to redo:
+
+- a multi-day data import that a deploy would otherwise kill
+- a job that stays `running` forever because the worker that held it is gone
+- a transient error at hour nine throwing away the first eight
+- two workers driving one job over a single cursor
+
+## Install
+
+The packages are not on npm yet, so install them from this repository. Both build on pack, so a
+git dependency delivers a built package, and yarn pins the resolved commit in your lockfile.
+
+```yaml
+# .yarnrc.yml
+approvedGitRepositories:
+  - "https://github.com/fullstackhouse/*"
+```
+
+```bash
+REPO="git+https://github.com/fullstackhouse/open-mercato-durable.git"
+yarn add \
+  "@fullstackhouse/open-mercato-durable-work@$REPO#workspace=@fullstackhouse/open-mercato-durable-work" \
+  "@fullstackhouse/open-mercato-data-sync-durable@$REPO#workspace=@fullstackhouse/open-mercato-data-sync-durable"
+```
+
+HTTPS needs no credentials, which is what makes this work in CI without a deploy key. Add
+`&commit=<sha>` to pin explicitly; otherwise yarn records the resolved commit in the lockfile
+and re-resolves only when you ask it to.
+
+**Add both, always.** `durable-work` is a *peer* of the adopter, and installing exactly one copy
+is not tidiness: it exports a process-wide registry, and a second copy would mean job kinds
+register into one while the worker reads the other — silently.
+
+Once the packages are published this becomes `yarn mercato module add @fullstackhouse/…`.
+
+`src/modules.ts`:
+
+```ts
+{ id: 'durable_work', from: '@fullstackhouse/open-mercato-durable-work' },
+```
+
+Then `yarn generate && yarn db:migrate`, and run the worker as its own process:
+
+```bash
+yarn mercato durable_work worker
+```
+
+Its own process on purpose: a slice can run for minutes, and hosting that inside the web
+process means a deploy either kills work mid-batch or waits out a slice.
+
+## Configuration
+
+| Variable | Default | What it does |
+|---|---|---|
+| `DURABLE_WORK_TRANSPORT` | `pgboss` | `pgboss`, `bullmq` or `memory` |
+| `DURABLE_WORK_REDIS_URL` | `QUEUE_REDIS_URL` | BullMQ only |
+| `DURABLE_WORK_PGBOSS_SCHEMA` | `durable_work_boss` | keeps pg-boss's tables out of `public` |
+| `DURABLE_WORK_TICK_MS` | `15000` | how often the reconciler runs |
+| `DURABLE_WORK_DRAIN_TIMEOUT_MS` | `30000` | how long a SIGTERM waits for slices to hand back |
+
+`pgboss` is the default because it needs nothing but the database you already have, and it is
+the only transport that can enqueue a delivery inside your own transaction. Use `bullmq` if the
+app already runs Redis. `memory` is for development and is refused in production.
+
+## Declaring work
+
+```ts
+import { registry } from '@fullstackhouse/open-mercato-durable-work'
+
+registry.register({
+  kind: 'catalog.reindex',
+  queue: 'durable-work.catalog',
+  // A job nobody declares idempotent is parked for a human rather than re-run automatically.
+  orphanPolicy: 'redrive',
+
+  async step(ctx) {
+    let done = ctx.checkpoint?.done ?? 0
+    while (done < total) {
+      // Stop at a boundary when the budget is spent or the process is shutting down.
+      if (ctx.shouldYield()) return 'budget'
+
+      await ctx.fencedWrite(async (tx) => {
+        // Rolls back if the lease was lost, so this cannot outlive the right to write it.
+      })
+
+      done += 1
+      await ctx.checkpoint_({ done }) // resume point, and a committed unit of work
+    }
+    return 'drained'
+  },
+
+  // Mirrors the terminal state onto your own row, in the same transaction.
+  async onTransition(job, scope, tx) { /* … */ return { matched: 1 } },
+  async onRedrive(job, scope, tx) { /* … */ return { matched: 1 } },
+})
+```
+
+Throw `TransientError` (or anything unrecognised) to retry, `TerminalError` to stop, and
+`UnrecoverableError` to stop and require `{ force: true }` before it can run again.
+
+## Operating
+
+```
+GET    /api/durable_work/jobs           durable_work.view
+GET    /api/durable_work/jobs/[id]      durable_work.view
+POST   /api/durable_work/jobs/[id]/redrive   durable_work.operate
+DELETE /api/durable_work/jobs/[id]      durable_work.operate
+```
+
+A re-drive refuses with a code rather than a generic failure, because the three refusals have
+different answers: `lock_key_held` (wait for or cancel the job holding the key),
+`not_redrivable` (completed and cancelled jobs are done), `unrecoverable_requires_force`.
+
+`mercato durable_work reconcile` runs one repair pass and prints what it repaired.
+
+## What it promises, and what it does not
+
+**Promises.** No job stays `running` forever. A worker that lost its lease cannot write. Work
+resumes from the last committed checkpoint. One live job per lock key per tenant.
+
+**Does not.** Exactly-once execution: delivery is at-least-once, so a process killed between a
+side effect and its checkpoint will redo that side effect. Make them idempotent, or put them
+behind `onTransition`, which runs in the terminal transaction.
+
+MIT. Part of [open-mercato-durable](https://github.com/fullstackhouse/open-mercato-durable).

package/dist/core/errors.js

@@ -0,0 +1,81 @@
+class TransientError extends Error {
+  constructor(message, options) {
+    super(message, options?.cause !== void 0 ? { cause: options.cause } : void 0);
+    this.durableErrorClass = "transient";
+    this.name = "TransientError";
+    this.code = options?.code;
+  }
+}
+class TerminalError extends Error {
+  constructor(message, options) {
+    super(message, options?.cause !== void 0 ? { cause: options.cause } : void 0);
+    this.durableErrorClass = "terminal";
+    this.name = "TerminalError";
+    this.code = options?.code;
+  }
+}
+class UnrecoverableError extends Error {
+  constructor(message, options) {
+    super(message, options?.cause !== void 0 ? { cause: options.cause } : void 0);
+    this.durableErrorClass = "unrecoverable";
+    this.name = "UnrecoverableError";
+    this.code = options?.code;
+  }
+}
+class LeaseLostError extends Error {
+  constructor(lease) {
+    super(`Lease lost for job ${lease.jobId} (owner ${lease.owner}, epoch ${lease.epoch})`);
+    this.lease = lease;
+    this.durableErrorClass = "transient";
+    this.name = "LeaseLostError";
+  }
+}
+class LockKeyHeldError extends Error {
+  constructor(lockKey, heldBy) {
+    super(`Lock key ${JSON.stringify(lockKey)} is held by a live job${heldBy ? ` (${heldBy})` : ""}`);
+    this.lockKey = lockKey;
+    this.heldBy = heldBy;
+    this.durableErrorClass = "terminal";
+    this.name = "LockKeyHeldError";
+  }
+}
+class NoFurtherAttempts extends Error {
+  constructor(reason) {
+    super(`No further transport attempts: ${reason}`);
+    this.reason = reason;
+    this.name = "NoFurtherAttempts";
+  }
+}
+class UnknownKindError extends Error {
+  constructor(kind) {
+    super(`No handler registered for durable job kind ${JSON.stringify(kind)}`);
+    this.kind = kind;
+    this.name = "UnknownKindError";
+  }
+}
+const CLASSES = /* @__PURE__ */ new Set(["transient", "terminal", "unrecoverable"]);
+function classifyError(error) {
+  const candidate = error?.durableErrorClass;
+  return typeof candidate === "string" && CLASSES.has(candidate) ? candidate : "transient";
+}
+function errorCodeOf(error) {
+  const code = error?.code;
+  return typeof code === "string" && code.length > 0 ? code : null;
+}
+function errorMessageOf(error) {
+  if (error instanceof Error) return error.message;
+  return typeof error === "string" ? error : "Unknown error";
+}
+export {
+  LeaseLostError,
+  LockKeyHeldError,
+  NoFurtherAttempts,
+  TerminalError,
+  TransientError,
+  UnknownKindError,
+  UnrecoverableError,
+  classifyError,
+  errorCodeOf,
+  errorMessageOf
+};
+//# sourceMappingURL=errors.js.map

package/dist/core/errors.js.map

@@ -0,0 +1,7 @@
+{
+  "version": 3,
+  "sources": ["../../src/core/errors.ts"],
+  "sourcesContent": ["// The error taxonomy. Anything a step throws that is not one of these is `transient`: an\n// unrecognised failure is far more often a blip than a dead end, and the cost of guessing\n// wrong is one retry rather than a multi-day run thrown away.\n\nimport type { ErrorClass } from './types'\n\n/** Retry this delivery. The default class for anything unrecognised. */\nexport class TransientError extends Error {\n  readonly durableErrorClass: ErrorClass = 'transient'\n  constructor(message: string, options?: { cause?: unknown; code?: string }) {\n    super(message, options?.cause !== undefined ? { cause: options.cause } : undefined)\n    this.name = 'TransientError'\n    this.code = options?.code\n  }\n  readonly code?: string\n}\n\n/** Fail the job now, without further transport attempts. Still re-drivable by an operator. */\nexport class TerminalError extends Error {\n  readonly durableErrorClass: ErrorClass = 'terminal'\n  constructor(message: string, options?: { cause?: unknown; code?: string }) {\n    super(message, options?.cause !== undefined ? { cause: options.cause } : undefined)\n    this.name = 'TerminalError'\n    this.code = options?.code\n  }\n  readonly code?: string\n}\n\n/** Fail the job now and refuse a plain re-drive: an operator must pass `{ force: true }`.\n *  For failures where running the work again is known to be wrong, not merely useless. */\nexport class UnrecoverableError extends Error {\n  readonly durableErrorClass: ErrorClass = 'unrecoverable'\n  constructor(message: string, options?: { cause?: unknown; code?: string }) {\n    super(message, options?.cause !== undefined ? { cause: options.cause } : undefined)\n    this.name = 'UnrecoverableError'\n    this.code = options?.code\n  }\n  readonly code?: string\n}\n\n/** Thrown by `fencedWrite` when the lease was lost or taken while the slice was running. The\n *  transaction it guards has already rolled back, so nothing the slice believed it wrote is\n *  in the database. Never retried inside the slice: the job now belongs to someone else. */\nexport class LeaseLostError extends Error {\n  readonly durableErrorClass: ErrorClass = 'transient'\n  constructor(readonly lease: { jobId: string; owner: string; epoch: number }) {\n    super(`Lease lost for job ${lease.jobId} (owner ${lease.owner}, epoch ${lease.epoch})`)\n    this.name = 'LeaseLostError'\n  }\n}\n\n/** Another live job already holds this `lockKey` in this scope. The single-runner guarantee,\n *  surfaced as a 409 rather than a duplicate run. */\nexport class LockKeyHeldError extends Error {\n  readonly durableErrorClass: ErrorClass = 'terminal'\n  constructor(\n    readonly lockKey: string,\n    readonly heldBy?: string,\n  ) {\n    super(`Lock key ${JSON.stringify(lockKey)} is held by a live job${heldBy ? ` (${heldBy})` : ''}`)\n    this.name = 'LockKeyHeldError'\n  }\n}\n\n/** Signals the transport to end this delivery without scheduling another attempt. The job's\n *  own state already says what happened; a further attempt would claim a row that refuses it. */\nexport class NoFurtherAttempts extends Error {\n  constructor(readonly reason: string) {\n    super(`No further transport attempts: ${reason}`)\n    this.name = 'NoFurtherAttempts'\n  }\n}\n\n/** A registry lookup for a kind no process registered. */\nexport class UnknownKindError extends Error {\n  constructor(readonly kind: string) {\n    super(`No handler registered for durable job kind ${JSON.stringify(kind)}`)\n    this.name = 'UnknownKindError'\n  }\n}\n\nconst CLASSES: ReadonlySet<string> = new Set<ErrorClass>(['transient', 'terminal', 'unrecoverable'])\n\n/** Reads the class off an error, defaulting to `transient`.\n *\n *  Matches on a `durableErrorClass` property rather than `instanceof`, so an error that\n *  crossed a package boundary \u2014 two copies of this package in one install, a re-thrown cause,\n *  a structured-clone \u2014 is still classified correctly. Getting this wrong the other way would\n *  silently downgrade an `unrecoverable` to a retry loop. */\nexport function classifyError(error: unknown): ErrorClass {\n  const candidate = (error as { durableErrorClass?: unknown } | null | undefined)?.durableErrorClass\n  return typeof candidate === 'string' && CLASSES.has(candidate) ? (candidate as ErrorClass) : 'transient'\n}\n\n/** The `error_code` to persist, if the error carries one. */\nexport function errorCodeOf(error: unknown): string | null {\n  const code = (error as { code?: unknown } | null | undefined)?.code\n  return typeof code === 'string' && code.length > 0 ? code : null\n}\n\nexport function errorMessageOf(error: unknown): string {\n  if (error instanceof Error) return error.message\n  return typeof error === 'string' ? error : 'Unknown error'\n}\n"],
+  "mappings": "AAOO,MAAM,uBAAuB,MAAM;AAAA,EAExC,YAAY,SAAiB,SAA8C;AACzE,UAAM,SAAS,SAAS,UAAU,SAAY,EAAE,OAAO,QAAQ,MAAM,IAAI,MAAS;AAFpF,SAAS,oBAAgC;AAGvC,SAAK,OAAO;AACZ,SAAK,OAAO,SAAS;AAAA,EACvB;AAEF;AAGO,MAAM,sBAAsB,MAAM;AAAA,EAEvC,YAAY,SAAiB,SAA8C;AACzE,UAAM,SAAS,SAAS,UAAU,SAAY,EAAE,OAAO,QAAQ,MAAM,IAAI,MAAS;AAFpF,SAAS,oBAAgC;AAGvC,SAAK,OAAO;AACZ,SAAK,OAAO,SAAS;AAAA,EACvB;AAEF;AAIO,MAAM,2BAA2B,MAAM;AAAA,EAE5C,YAAY,SAAiB,SAA8C;AACzE,UAAM,SAAS,SAAS,UAAU,SAAY,EAAE,OAAO,QAAQ,MAAM,IAAI,MAAS;AAFpF,SAAS,oBAAgC;AAGvC,SAAK,OAAO;AACZ,SAAK,OAAO,SAAS;AAAA,EACvB;AAEF;AAKO,MAAM,uBAAuB,MAAM;AAAA,EAExC,YAAqB,OAAwD;AAC3E,UAAM,sBAAsB,MAAM,KAAK,WAAW,MAAM,KAAK,WAAW,MAAM,KAAK,GAAG;AADnE;AADrB,SAAS,oBAAgC;AAGvC,SAAK,OAAO;AAAA,EACd;AACF;AAIO,MAAM,yBAAyB,MAAM;AAAA,EAE1C,YACW,SACA,QACT;AACA,UAAM,YAAY,KAAK,UAAU,OAAO,CAAC,yBAAyB,SAAS,KAAK,MAAM,MAAM,EAAE,EAAE;AAHvF;AACA;AAHX,SAAS,oBAAgC;AAMvC,SAAK,OAAO;AAAA,EACd;AACF;AAIO,MAAM,0BAA0B,MAAM;AAAA,EAC3C,YAAqB,QAAgB;AACnC,UAAM,kCAAkC,MAAM,EAAE;AAD7B;AAEnB,SAAK,OAAO;AAAA,EACd;AACF;AAGO,MAAM,yBAAyB,MAAM;AAAA,EAC1C,YAAqB,MAAc;AACjC,UAAM,8CAA8C,KAAK,UAAU,IAAI,CAAC,EAAE;AADvD;AAEnB,SAAK,OAAO;AAAA,EACd;AACF;AAEA,MAAM,UAA+B,oBAAI,IAAgB,CAAC,aAAa,YAAY,eAAe,CAAC;AAQ5F,SAAS,cAAc,OAA4B;AACxD,QAAM,YAAa,OAA8D;AACjF,SAAO,OAAO,cAAc,YAAY,QAAQ,IAAI,SAAS,IAAK,YAA2B;AAC/F;AAGO,SAAS,YAAY,OAA+B;AACzD,QAAM,OAAQ,OAAiD;AAC/D,SAAO,OAAO,SAAS,YAAY,KAAK,SAAS,IAAI,OAAO;AAC9D;AAEO,SAAS,eAAe,OAAwB;AACrD,MAAI,iBAAiB,MAAO,QAAO,MAAM;AACzC,SAAO,OAAO,UAAU,WAAW,QAAQ;AAC7C;",
+  "names": []
+}

package/dist/core/ids.js

@@ -0,0 +1,30 @@
+function deliveryId(delivery) {
+  return `dw-${delivery.jobId}-${delivery.seq}-${delivery.redrives}`;
+}
+const DELIVERY_ID = /^dw-(.+)-(\d+)-(\d+)$/;
+function parseDeliveryId(id) {
+  const match = DELIVERY_ID.exec(id);
+  if (!match) return null;
+  return { jobId: match[1], seq: Number(match[2]), redrives: Number(match[3]) };
+}
+function queueNameFor(group) {
+  return `durable-work.${group}`;
+}
+const PORTABLE_QUEUE_NAME = /^[A-Za-z0-9_\-./]+$/;
+function sliceIdempotencyKey(jobId, seq) {
+  return `${jobId}:${seq}`;
+}
+function makeOwnerId(prefix = "dw") {
+  const random = Math.random().toString(36).slice(2, 10);
+  const pid = typeof process !== "undefined" && process.pid ? process.pid : 0;
+  return `${prefix}-${pid}-${Date.now().toString(36)}-${random}`;
+}
+export {
+  PORTABLE_QUEUE_NAME,
+  deliveryId,
+  makeOwnerId,
+  parseDeliveryId,
+  queueNameFor,
+  sliceIdempotencyKey
+};
+//# sourceMappingURL=ids.js.map

package/dist/core/ids.js.map

@@ -0,0 +1,7 @@
+{
+  "version": 3,
+  "sources": ["../../src/core/ids.ts"],
+  "sourcesContent": ["// Delivery identity and queue naming.\n\nimport type { Delivery } from './types'\n\n/** The id a transport carries for one delivery of one job.\n *\n *  Colon-free on purpose: BullMQ 6 rejects `:` in a job id (it is the separator in its own\n *  Redis keys), and the id has to be usable unchanged as a pg-boss `singletonKey` too.\n *\n *  It encodes `(jobId, seq, redrives)` because that triple IS the fence: a delivery whose seq\n *  or redrives no longer match the row is a straggler from a previous slice or re-drive, and\n *  `claim` refuses it. Making the id carry the triple means a duplicate delivery is refused by\n *  the database rather than deduplicated by the broker, which is the only version of that\n *  guarantee that survives a broker restart. */\nexport function deliveryId(delivery: Delivery): string {\n  return `dw-${delivery.jobId}-${delivery.seq}-${delivery.redrives}`\n}\n\nconst DELIVERY_ID = /^dw-(.+)-(\\d+)-(\\d+)$/\n\nexport function parseDeliveryId(id: string): Delivery | null {\n  const match = DELIVERY_ID.exec(id)\n  if (!match) return null\n  return { jobId: match[1]!, seq: Number(match[2]), redrives: Number(match[3]) }\n}\n\n/**\n * Queues are named per kind group so one worker process can bind a subset of kinds.\n *\n * A period, not a colon. pg-boss validates queue names against\n * `[alphanumeric, _, -, ., /]` and rejects a colon outright, and BullMQ gives `:` special\n * meaning inside its own Redis keys. One name has to work on every adapter \u2014 a per-adapter\n * rewrite would mean the queue an operator sees in the code is not the queue they can find\n * in the broker.\n */\nexport function queueNameFor(group: string): string {\n  return `durable-work.${group}`\n}\n\n/** The characters every supported broker accepts in a queue name. */\nexport const PORTABLE_QUEUE_NAME = /^[A-Za-z0-9_\\-./]+$/\n\n/** The idempotency key handed to a slice, and the one it should forward to any external\n *  side effect. Stable across retries of the same slice, different for the next slice. */\nexport function sliceIdempotencyKey(jobId: string, seq: number): string {\n  return `${jobId}:${seq}`\n}\n\n/** Identifies one worker process for the lifetime of that process.\n *\n *  The random suffix is what makes \"is this lease mine?\" answerable after a crash: a restarted\n *  process on the same host must not match the lease its predecessor held, or a stalled\n *  redelivery to the new process would be accepted while the old row still looks alive. */\nexport function makeOwnerId(prefix = 'dw'): string {\n  const random = Math.random().toString(36).slice(2, 10)\n  const pid = typeof process !== 'undefined' && process.pid ? process.pid : 0\n  return `${prefix}-${pid}-${Date.now().toString(36)}-${random}`\n}\n"],
+  "mappings": "AAcO,SAAS,WAAW,UAA4B;AACrD,SAAO,MAAM,SAAS,KAAK,IAAI,SAAS,GAAG,IAAI,SAAS,QAAQ;AAClE;AAEA,MAAM,cAAc;AAEb,SAAS,gBAAgB,IAA6B;AAC3D,QAAM,QAAQ,YAAY,KAAK,EAAE;AACjC,MAAI,CAAC,MAAO,QAAO;AACnB,SAAO,EAAE,OAAO,MAAM,CAAC,GAAI,KAAK,OAAO,MAAM,CAAC,CAAC,GAAG,UAAU,OAAO,MAAM,CAAC,CAAC,EAAE;AAC/E;AAWO,SAAS,aAAa,OAAuB;AAClD,SAAO,gBAAgB,KAAK;AAC9B;AAGO,MAAM,sBAAsB;AAI5B,SAAS,oBAAoB,OAAe,KAAqB;AACtE,SAAO,GAAG,KAAK,IAAI,GAAG;AACxB;AAOO,SAAS,YAAY,SAAS,MAAc;AACjD,QAAM,SAAS,KAAK,OAAO,EAAE,SAAS,EAAE,EAAE,MAAM,GAAG,EAAE;AACrD,QAAM,MAAM,OAAO,YAAY,eAAe,QAAQ,MAAM,QAAQ,MAAM;AAC1E,SAAO,GAAG,MAAM,IAAI,GAAG,IAAI,KAAK,IAAI,EAAE,SAAS,EAAE,CAAC,IAAI,MAAM;AAC9D;",
+  "names": []
+}

package/dist/core/reconciler.js

@@ -0,0 +1,149 @@
+import { errorMessageOf } from "./errors.js";
+import { runAfterTransition, runTerminalTransition } from "./terminal.js";
+import {
+  cancelPending,
+  markMirrored,
+  park,
+  redrivePending,
+  selectCancelling,
+  selectOrphans,
+  selectStalePending,
+  takeOrphan
+} from "./store.js";
+const DEFAULT_GRACE_MS = 2e4;
+const DEFAULT_BATCH = 100;
+const REDRIVE_BASE_MS = 15e3;
+const REDRIVE_CAP_MS = 6e5;
+const scopeOf = (job) => ({ tenantId: job.tenantId, organizationId: job.organizationId });
+const leaseOf = (job) => ({ jobId: job.id, owner: job.leaseOwner ?? "reconciler", epoch: job.leaseEpoch, ttlMs: 0 });
+async function reconcileOnce(deps) {
+  const report = { scanned: 0, cancelled: 0, redriven: 0, parked: 0, errors: 0 };
+  const limit = deps.batchSize ?? DEFAULT_BATCH;
+  const graceMs = deps.graceMs ?? DEFAULT_GRACE_MS;
+  for (const job of await select(deps, (tx) => selectCancelling(tx, limit, deps.tenantId))) {
+    report.scanned += 1;
+    try {
+      if (await endCancelled(deps, job)) report.cancelled += 1;
+    } catch (error) {
+      report.errors += 1;
+      deps.log?.("durable_work.reconcile_cancel_failed", { jobId: job.id, error: errorMessageOf(error) });
+    }
+  }
+  for (const job of await select(deps, (tx) => selectOrphans(tx, { graceMs, pendingTtlMs: widestPendingTtl(deps), limit, tenantId: deps.tenantId }))) {
+    report.scanned += 1;
+    try {
+      const outcome = await repairOrphan(deps, job, graceMs);
+      if (outcome === "redriven") report.redriven += 1;
+      if (outcome === "parked") report.parked += 1;
+    } catch (error) {
+      report.errors += 1;
+      deps.log?.("durable_work.reconcile_orphan_failed", { jobId: job.id, error: errorMessageOf(error) });
+    }
+  }
+  for (const job of await select(deps, (tx) => selectStalePending(tx, { pendingTtlMs: widestPendingTtl(deps), limit, tenantId: deps.tenantId }))) {
+    report.scanned += 1;
+    try {
+      const outcome = await repairPending(deps, job);
+      if (outcome === "redriven") report.redriven += 1;
+      if (outcome === "parked") report.parked += 1;
+    } catch (error) {
+      report.errors += 1;
+      deps.log?.("durable_work.reconcile_pending_failed", { jobId: job.id, error: errorMessageOf(error) });
+    }
+  }
+  return report;
+}
+async function select(deps, query) {
+  return deps.sql.transaction(query);
+}
+function widestPendingTtl(deps) {
+  const kinds = deps.registry.list();
+  return kinds.length ? Math.max(...kinds.map((k) => k.lease.pendingTtlMs)) : 9e5;
+}
+async function endCancelled(deps, job) {
+  const kind = deps.registry.get(job.kind);
+  if (job.status === "pending") {
+    const ended = await deps.sql.transaction(async (tx) => {
+      const row = await cancelPending(tx, job.id);
+      if (!row) return null;
+      if (kind?.onCancel) await kind.onCancel(row, scopeOf(row), tx);
+      if (kind?.onTransition) {
+        const { matched } = await kind.onTransition(row, scopeOf(row), tx);
+        if (matched < 1) throw new Error(`Domain mirror matched no rows for job ${row.id}`);
+      }
+      await markMirrored(tx, row.id);
+      return row;
+    });
+    if (!ended) return false;
+    if (kind) await runAfterTransition(kind, ended, scopeOf(ended));
+    deps.log?.("durable_work.job_cancelled", { jobId: job.id, by: "reconciler" });
+    return true;
+  }
+  if (job.leaseExpiresAt && job.leaseExpiresAt.getTime() > Date.now()) return false;
+  if (!kind) {
+    const parked = await park(deps.sql, job.id, "no_handler", "No handler registered for this kind");
+    return parked != null;
+  }
+  const result = await runTerminalTransition(deps.sql, kind, leaseOf(job), scopeOf(job), { type: "cancel" });
+  if (!result) return false;
+  await runAfterTransition(kind, result.job, scopeOf(result.job));
+  deps.log?.("durable_work.job_cancelled", { jobId: job.id, by: "reconciler" });
+  return true;
+}
+async function repairOrphan(deps, job, graceMs) {
+  const kind = deps.registry.get(job.kind);
+  if (!kind) return await parkJob(deps, job, void 0, "no_handler", "No handler registered for this kind") ? "parked" : "skipped";
+  if (job.errorCode === "unrecoverable" || job.errorCode === "retry_exhausted") {
+    return await parkJob(deps, job, kind, job.errorCode, job.errorMessage) ? "parked" : "skipped";
+  }
+  if (kind.orphanPolicy !== "redrive") {
+    return await parkJob(deps, job, kind, "orphaned", job.errorMessage ?? "Worker stopped without releasing the lease") ? "parked" : "skipped";
+  }
+  if (job.redrivesSinceCommit >= kind.budget.poisonRedrivesWithoutCommit) {
+    return await parkJob(deps, job, kind, "poison", "Re-driven repeatedly without committing progress") ? "parked" : "skipped";
+  }
+  const backoffMs = Math.min(REDRIVE_BASE_MS * 2 ** job.redrivesSinceCommit, REDRIVE_CAP_MS);
+  const taken = await takeOrphan(deps.sql, job.id, { graceMs, pendingTtlMs: kind.lease.pendingTtlMs, backoffMs });
+  if (!taken) return "skipped";
+  await deps.enqueue(taken);
+  deps.log?.("durable_work.job_orphaned", { jobId: job.id, redrives: taken.redrives, backoffMs });
+  return "redriven";
+}
+async function repairPending(deps, job) {
+  const kind = deps.registry.get(job.kind);
+  if (!kind) return await parkJob(deps, job, void 0, "no_handler", "No handler registered for this kind") ? "parked" : "skipped";
+  if (job.redrivesSinceCommit >= kind.budget.maxRedrives) {
+    return await parkJob(deps, job, kind, "never_started", "Delivery never arrived after repeated re-drives") ? "parked" : "skipped";
+  }
+  const redriven = await redrivePending(deps.sql, job.id, { pendingTtlMs: kind.lease.pendingTtlMs });
+  if (!redriven) return "skipped";
+  await deps.enqueue(redriven);
+  deps.log?.("durable_work.job_redriven", { jobId: job.id, redrives: redriven.redrives, reason: "never_started" });
+  return "redriven";
+}
+async function parkJob(deps, job, kind, reason, message) {
+  if (!kind?.onTransition) {
+    const parked2 = await park(deps.sql, job.id, reason, message);
+    if (parked2) {
+      await deps.sql.query(`update durable_work_jobs set domain_mirrored_at = now() where id = $1`, [job.id]);
+      deps.log?.("durable_work.job_parked", { jobId: job.id, reason });
+    }
+    return parked2 != null;
+  }
+  const parked = await deps.sql.transaction(async (tx) => {
+    const row = await park(tx, job.id, reason, message);
+    if (!row) return null;
+    const { matched } = await kind.onTransition(row, scopeOf(row), tx);
+    if (matched < 1) throw new Error(`Domain mirror matched no rows for job ${row.id}`);
+    await markMirrored(tx, row.id);
+    return row;
+  });
+  if (!parked) return false;
+  await runAfterTransition(kind, parked, scopeOf(parked));
+  deps.log?.("durable_work.job_parked", { jobId: job.id, reason });
+  return true;
+}
+export {
+  reconcileOnce
+};
+//# sourceMappingURL=reconciler.js.map

package/dist/core/reconciler.js.map

@@ -0,0 +1,7 @@
+{
+  "version": 3,
+  "sources": ["../../src/core/reconciler.ts"],
+  "sourcesContent": ["// The server-side repair loop: what makes \"no job stays running forever\" true.\n//\n// Three queries, in a fixed order, each selecting a bounded batch with `for update skip\n// locked` in its own short transaction. The order is not cosmetic \u2014 see the note on Q3.\n//\n// Everything here is decided from the row, never from anything a process remembers. That is\n// what lets any process run the reconciler, lets two run at once, and lets the whole fleet\n// restart without losing track of a single job.\n\nimport { errorMessageOf } from './errors'\nimport type { KindRegistry, ResolvedKind } from './registry'\nimport { runAfterTransition, runTerminalTransition } from './terminal'\nimport {\n  cancelPending,\n  markMirrored,\n  park,\n  redrivePending,\n  selectCancelling,\n  selectOrphans,\n  selectStalePending,\n  takeOrphan,\n} from './store'\nimport type { DurableJob, Lease, Scope, SqlTransactor } from './types'\n\nexport type ReconcileReport = {\n  scanned: number\n  cancelled: number\n  redriven: number\n  parked: number\n  errors: number\n}\n\nexport type ReconcilerDeps = {\n  sql: SqlTransactor\n  registry: KindRegistry\n  /** Enqueues a delivery for a job that has just been re-driven. */\n  enqueue: (job: DurableJob) => Promise<void>\n  /** How long past its expiry a lease is tolerated before the job counts as orphaned. Short,\n   *  because lease expiry is a database-clock fact about a driver. */\n  graceMs?: number\n  batchSize?: number\n  /** Repair only this tenant's jobs. Unset means every tenant, which is what a single worker\n   *  should do; set it to shard the loop across a fleet. */\n  tenantId?: string\n  log?: (event: string, fields: Record<string, unknown>) => void\n}\n\nconst DEFAULT_GRACE_MS = 20_000\nconst DEFAULT_BATCH = 100\n/** Backoff between successive re-drives of the same job, so a job that keeps orphaning does\n *  not spin. Doubles per re-drive since the last committed unit, capped. */\nconst REDRIVE_BASE_MS = 15_000\nconst REDRIVE_CAP_MS = 600_000\n\nconst scopeOf = (job: DurableJob): Scope => ({ tenantId: job.tenantId, organizationId: job.organizationId })\nconst leaseOf = (job: DurableJob): Lease => ({ jobId: job.id, owner: job.leaseOwner ?? 'reconciler', epoch: job.leaseEpoch, ttlMs: 0 })\n\n/**\n * One pass. Safe to run concurrently with itself: every query takes its rows with\n * `skip locked`, so two reconcilers partition the work rather than fighting over it.\n */\nexport async function reconcileOnce(deps: ReconcilerDeps): Promise<ReconcileReport> {\n  const report: ReconcileReport = { scanned: 0, cancelled: 0, redriven: 0, parked: 0, errors: 0 }\n  const limit = deps.batchSize ?? DEFAULT_BATCH\n  const graceMs = deps.graceMs ?? DEFAULT_GRACE_MS\n\n  // Q3 \u2014 cancellations first.\n  //\n  // A running job whose driver died after an operator asked to cancel it matches both \"dead\n  // cancel\" and \"orphan\". Without a precedence, the orphan query would park it as orphaned\n  // with the cancellation never honoured, or re-drive it \u2014 restarting work somebody\n  // explicitly asked to stop. So cancellations are settled before anything else looks.\n  for (const job of await select(deps, (tx) => selectCancelling(tx, limit, deps.tenantId))) {\n    report.scanned += 1\n    try {\n      if (await endCancelled(deps, job)) report.cancelled += 1\n    } catch (error) {\n      report.errors += 1\n      deps.log?.('durable_work.reconcile_cancel_failed', { jobId: job.id, error: errorMessageOf(error) })\n    }\n  }\n\n  // Q1 \u2014 orphans: a job whose driver stopped heartbeating.\n  for (const job of await select(deps, (tx) => selectOrphans(tx, { graceMs, pendingTtlMs: widestPendingTtl(deps), limit, tenantId: deps.tenantId }))) {\n    report.scanned += 1\n    try {\n      const outcome = await repairOrphan(deps, job, graceMs)\n      if (outcome === 'redriven') report.redriven += 1\n      if (outcome === 'parked') report.parked += 1\n    } catch (error) {\n      report.errors += 1\n      deps.log?.('durable_work.reconcile_orphan_failed', { jobId: job.id, error: errorMessageOf(error) })\n    }\n  }\n\n  // Q2 \u2014 pending jobs whose delivery never arrived: a lost hand-back, or an enqueue that\n  // never reached the broker because the process died between commit and enqueue.\n  for (const job of await select(deps, (tx) => selectStalePending(tx, { pendingTtlMs: widestPendingTtl(deps), limit, tenantId: deps.tenantId }))) {\n    report.scanned += 1\n    try {\n      const outcome = await repairPending(deps, job)\n      if (outcome === 'redriven') report.redriven += 1\n      if (outcome === 'parked') report.parked += 1\n    } catch (error) {\n      report.errors += 1\n      deps.log?.('durable_work.reconcile_pending_failed', { jobId: job.id, error: errorMessageOf(error) })\n    }\n  }\n\n  return report\n}\n\n/** Selection commits \u2014 and so releases its locks \u2014 before any per-row work runs. Holding a\n *  row lock across a domain mirror would block a second reconciler for the length of that\n *  mirror, and turn a slow domain into a stalled repair loop. */\nasync function select(deps: ReconcilerDeps, query: (tx: Parameters<Parameters<SqlTransactor['transaction']>[0]>[0]) => Promise<DurableJob[]>): Promise<DurableJob[]> {\n  return deps.sql.transaction(query)\n}\n\n/** Selection is a coarse filter, so it uses the widest tolerance any registered kind declares\n *  and lets the per-row statements re-check with that row's own kind. Selecting on the\n *  narrowest instead would silently exclude jobs of a more tolerant kind from being repaired\n *  at all; over-selecting only costs a re-check. */\nfunction widestPendingTtl(deps: ReconcilerDeps): number {\n  const kinds = deps.registry.list()\n  return kinds.length ? Math.max(...kinds.map((k) => k.lease.pendingTtlMs)) : 900_000\n}\n\nasync function endCancelled(deps: ReconcilerDeps, job: DurableJob): Promise<boolean> {\n  const kind = deps.registry.get(job.kind)\n\n  if (job.status === 'pending') {\n    // No lease to fence on; `pending` is the fence.\n    const ended = await deps.sql.transaction(async (tx) => {\n      const row = await cancelPending(tx, job.id)\n      if (!row) return null\n      if (kind?.onCancel) await kind.onCancel(row, scopeOf(row), tx)\n      if (kind?.onTransition) {\n        const { matched } = await kind.onTransition(row, scopeOf(row), tx)\n        if (matched < 1) throw new Error(`Domain mirror matched no rows for job ${row.id}`)\n      }\n      await markMirrored(tx, row.id)\n      return row\n    })\n    if (!ended) return false\n    if (kind) await runAfterTransition(kind, ended, scopeOf(ended))\n    deps.log?.('durable_work.job_cancelled', { jobId: job.id, by: 'reconciler' })\n    return true\n  }\n\n  // A `running` job whose lease has expired: nobody is driving it, so the reconciler settles\n  // the cancellation on its behalf. A live lease is left alone \u2014 its own slice will observe\n  // the request at the next heartbeat, which is both faster and safer.\n  if (job.leaseExpiresAt && job.leaseExpiresAt.getTime() > Date.now()) return false\n\n  if (!kind) {\n    const parked = await park(deps.sql, job.id, 'no_handler', 'No handler registered for this kind')\n    return parked != null\n  }\n  const result = await runTerminalTransition(deps.sql, kind, leaseOf(job), scopeOf(job), { type: 'cancel' })\n  if (!result) return false\n  await runAfterTransition(kind, result.job, scopeOf(result.job))\n  deps.log?.('durable_work.job_cancelled', { jobId: job.id, by: 'reconciler' })\n  return true\n}\n\nasync function repairOrphan(deps: ReconcilerDeps, job: DurableJob, graceMs: number): Promise<'redriven' | 'parked' | 'skipped'> {\n  const kind = deps.registry.get(job.kind)\n\n  // In order; the first match wins.\n  if (!kind) return (await parkJob(deps, job, undefined, 'no_handler', 'No handler registered for this kind')) ? 'parked' : 'skipped'\n  if (job.errorCode === 'unrecoverable' || job.errorCode === 'retry_exhausted') {\n    // The slice already reached a conclusion but could not commit it. Park with that verdict\n    // preserved \u2014 the orphan policy is never consulted for a job that has already decided.\n    return (await parkJob(deps, job, kind, job.errorCode, job.errorMessage)) ? 'parked' : 'skipped'\n  }\n  if (kind.orphanPolicy !== 'redrive') {\n    return (await parkJob(deps, job, kind, 'orphaned', job.errorMessage ?? 'Worker stopped without releasing the lease')) ? 'parked' : 'skipped'\n  }\n  if (job.redrivesSinceCommit >= kind.budget.poisonRedrivesWithoutCommit) {\n    // Re-driven this many times without committing anything: the job is not making progress\n    // and re-running it again is guessing. A human decides from here.\n    return (await parkJob(deps, job, kind, 'poison', 'Re-driven repeatedly without committing progress')) ? 'parked' : 'skipped'\n  }\n\n  const backoffMs = Math.min(REDRIVE_BASE_MS * 2 ** job.redrivesSinceCommit, REDRIVE_CAP_MS)\n  const taken = await takeOrphan(deps.sql, job.id, { graceMs, pendingTtlMs: kind.lease.pendingTtlMs, backoffMs })\n  if (!taken) return 'skipped' // the row moved under us; another pass will see it\n  await deps.enqueue(taken)\n  deps.log?.('durable_work.job_orphaned', { jobId: job.id, redrives: taken.redrives, backoffMs })\n  return 'redriven'\n}\n\nasync function repairPending(deps: ReconcilerDeps, job: DurableJob): Promise<'redriven' | 'parked' | 'skipped'> {\n  const kind = deps.registry.get(job.kind)\n  if (!kind) return (await parkJob(deps, job, undefined, 'no_handler', 'No handler registered for this kind')) ? 'parked' : 'skipped'\n\n  // A lost hand-back is cheap and is not evidence that the work is bad, so this budget is\n  // deliberately wider than the poison budget the orphan path uses.\n  if (job.redrivesSinceCommit >= kind.budget.maxRedrives) {\n    return (await parkJob(deps, job, kind, 'never_started', 'Delivery never arrived after repeated re-drives')) ? 'parked' : 'skipped'\n  }\n\n  const redriven = await redrivePending(deps.sql, job.id, { pendingTtlMs: kind.lease.pendingTtlMs })\n  if (!redriven) return 'skipped'\n  await deps.enqueue(redriven)\n  deps.log?.('durable_work.job_redriven', { jobId: job.id, redrives: redriven.redrives, reason: 'never_started' })\n  return 'redriven'\n}\n\n/** Parks through the terminal protocol when the kind has a mirror to run, and through the\n *  plain statement when it does not \u2014 an unregistered kind has no domain row to agree with,\n *  so \"no mirror\" is a satisfied mirror rather than a pending one. */\nasync function parkJob(\n  deps: ReconcilerDeps,\n  job: DurableJob,\n  kind: ResolvedKind | undefined,\n  reason: string,\n  message: string | null,\n): Promise<boolean> {\n  if (!kind?.onTransition) {\n    const parked = await park(deps.sql, job.id, reason as never, message)\n    if (parked) {\n      await deps.sql.query(`update durable_work_jobs set domain_mirrored_at = now() where id = $1`, [job.id])\n      deps.log?.('durable_work.job_parked', { jobId: job.id, reason })\n    }\n    return parked != null\n  }\n\n  const parked = await deps.sql.transaction(async (tx) => {\n    const row = await park(tx, job.id, reason as never, message)\n    if (!row) return null\n    const { matched } = await kind.onTransition!(row, scopeOf(row), tx)\n    if (matched < 1) throw new Error(`Domain mirror matched no rows for job ${row.id}`)\n    await markMirrored(tx, row.id)\n    return row\n  })\n  if (!parked) return false\n  await runAfterTransition(kind, parked, scopeOf(parked))\n  deps.log?.('durable_work.job_parked', { jobId: job.id, reason })\n  return true\n}\n"],
+  "mappings": "AASA,SAAS,sBAAsB;AAE/B,SAAS,oBAAoB,6BAA6B;AAC1D;AAAA,EACE;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,OACK;AA0BP,MAAM,mBAAmB;AACzB,MAAM,gBAAgB;AAGtB,MAAM,kBAAkB;AACxB,MAAM,iBAAiB;AAEvB,MAAM,UAAU,CAAC,SAA4B,EAAE,UAAU,IAAI,UAAU,gBAAgB,IAAI,eAAe;AAC1G,MAAM,UAAU,CAAC,SAA4B,EAAE,OAAO,IAAI,IAAI,OAAO,IAAI,cAAc,cAAc,OAAO,IAAI,YAAY,OAAO,EAAE;AAMrI,eAAsB,cAAc,MAAgD;AAClF,QAAM,SAA0B,EAAE,SAAS,GAAG,WAAW,GAAG,UAAU,GAAG,QAAQ,GAAG,QAAQ,EAAE;AAC9F,QAAM,QAAQ,KAAK,aAAa;AAChC,QAAM,UAAU,KAAK,WAAW;AAQhC,aAAW,OAAO,MAAM,OAAO,MAAM,CAAC,OAAO,iBAAiB,IAAI,OAAO,KAAK,QAAQ,CAAC,GAAG;AACxF,WAAO,WAAW;AAClB,QAAI;AACF,UAAI,MAAM,aAAa,MAAM,GAAG,EAAG,QAAO,aAAa;AAAA,IACzD,SAAS,OAAO;AACd,aAAO,UAAU;AACjB,WAAK,MAAM,wCAAwC,EAAE,OAAO,IAAI,IAAI,OAAO,eAAe,KAAK,EAAE,CAAC;AAAA,IACpG;AAAA,EACF;AAGA,aAAW,OAAO,MAAM,OAAO,MAAM,CAAC,OAAO,cAAc,IAAI,EAAE,SAAS,cAAc,iBAAiB,IAAI,GAAG,OAAO,UAAU,KAAK,SAAS,CAAC,CAAC,GAAG;AAClJ,WAAO,WAAW;AAClB,QAAI;AACF,YAAM,UAAU,MAAM,aAAa,MAAM,KAAK,OAAO;AACrD,UAAI,YAAY,WAAY,QAAO,YAAY;AAC/C,UAAI,YAAY,SAAU,QAAO,UAAU;AAAA,IAC7C,SAAS,OAAO;AACd,aAAO,UAAU;AACjB,WAAK,MAAM,wCAAwC,EAAE,OAAO,IAAI,IAAI,OAAO,eAAe,KAAK,EAAE,CAAC;AAAA,IACpG;AAAA,EACF;AAIA,aAAW,OAAO,MAAM,OAAO,MAAM,CAAC,OAAO,mBAAmB,IAAI,EAAE,cAAc,iBAAiB,IAAI,GAAG,OAAO,UAAU,KAAK,SAAS,CAAC,CAAC,GAAG;AAC9I,WAAO,WAAW;AAClB,QAAI;AACF,YAAM,UAAU,MAAM,cAAc,MAAM,GAAG;AAC7C,UAAI,YAAY,WAAY,QAAO,YAAY;AAC/C,UAAI,YAAY,SAAU,QAAO,UAAU;AAAA,IAC7C,SAAS,OAAO;AACd,aAAO,UAAU;AACjB,WAAK,MAAM,yCAAyC,EAAE,OAAO,IAAI,IAAI,OAAO,eAAe,KAAK,EAAE,CAAC;AAAA,IACrG;AAAA,EACF;AAEA,SAAO;AACT;AAKA,eAAe,OAAO,MAAsB,OAAyH;AACnK,SAAO,KAAK,IAAI,YAAY,KAAK;AACnC;AAMA,SAAS,iBAAiB,MAA8B;AACtD,QAAM,QAAQ,KAAK,SAAS,KAAK;AACjC,SAAO,MAAM,SAAS,KAAK,IAAI,GAAG,MAAM,IAAI,CAAC,MAAM,EAAE,MAAM,YAAY,CAAC,IAAI;AAC9E;AAEA,eAAe,aAAa,MAAsB,KAAmC;AACnF,QAAM,OAAO,KAAK,SAAS,IAAI,IAAI,IAAI;AAEvC,MAAI,IAAI,WAAW,WAAW;AAE5B,UAAM,QAAQ,MAAM,KAAK,IAAI,YAAY,OAAO,OAAO;AACrD,YAAM,MAAM,MAAM,cAAc,IAAI,IAAI,EAAE;AAC1C,UAAI,CAAC,IAAK,QAAO;AACjB,UAAI,MAAM,SAAU,OAAM,KAAK,SAAS,KAAK,QAAQ,GAAG,GAAG,EAAE;AAC7D,UAAI,MAAM,cAAc;AACtB,cAAM,EAAE,QAAQ,IAAI,MAAM,KAAK,aAAa,KAAK,QAAQ,GAAG,GAAG,EAAE;AACjE,YAAI,UAAU,EAAG,OAAM,IAAI,MAAM,yCAAyC,IAAI,EAAE,EAAE;AAAA,MACpF;AACA,YAAM,aAAa,IAAI,IAAI,EAAE;AAC7B,aAAO;AAAA,IACT,CAAC;AACD,QAAI,CAAC,MAAO,QAAO;AACnB,QAAI,KAAM,OAAM,mBAAmB,MAAM,OAAO,QAAQ,KAAK,CAAC;AAC9D,SAAK,MAAM,8BAA8B,EAAE,OAAO,IAAI,IAAI,IAAI,aAAa,CAAC;AAC5E,WAAO;AAAA,EACT;AAKA,MAAI,IAAI,kBAAkB,IAAI,eAAe,QAAQ,IAAI,KAAK,IAAI,EAAG,QAAO;AAE5E,MAAI,CAAC,MAAM;AACT,UAAM,SAAS,MAAM,KAAK,KAAK,KAAK,IAAI,IAAI,cAAc,qCAAqC;AAC/F,WAAO,UAAU;AAAA,EACnB;AACA,QAAM,SAAS,MAAM,sBAAsB,KAAK,KAAK,MAAM,QAAQ,GAAG,GAAG,QAAQ,GAAG,GAAG,EAAE,MAAM,SAAS,CAAC;AACzG,MAAI,CAAC,OAAQ,QAAO;AACpB,QAAM,mBAAmB,MAAM,OAAO,KAAK,QAAQ,OAAO,GAAG,CAAC;AAC9D,OAAK,MAAM,8BAA8B,EAAE,OAAO,IAAI,IAAI,IAAI,aAAa,CAAC;AAC5E,SAAO;AACT;AAEA,eAAe,aAAa,MAAsB,KAAiB,SAA6D;AAC9H,QAAM,OAAO,KAAK,SAAS,IAAI,IAAI,IAAI;AAGvC,MAAI,CAAC,KAAM,QAAQ,MAAM,QAAQ,MAAM,KAAK,QAAW,cAAc,qCAAqC,IAAK,WAAW;AAC1H,MAAI,IAAI,cAAc,mBAAmB,IAAI,cAAc,mBAAmB;AAG5E,WAAQ,MAAM,QAAQ,MAAM,KAAK,MAAM,IAAI,WAAW,IAAI,YAAY,IAAK,WAAW;AAAA,EACxF;AACA,MAAI,KAAK,iBAAiB,WAAW;AACnC,WAAQ,MAAM,QAAQ,MAAM,KAAK,MAAM,YAAY,IAAI,gBAAgB,4CAA4C,IAAK,WAAW;AAAA,EACrI;AACA,MAAI,IAAI,uBAAuB,KAAK,OAAO,6BAA6B;AAGtE,WAAQ,MAAM,QAAQ,MAAM,KAAK,MAAM,UAAU,kDAAkD,IAAK,WAAW;AAAA,EACrH;AAEA,QAAM,YAAY,KAAK,IAAI,kBAAkB,KAAK,IAAI,qBAAqB,cAAc;AACzF,QAAM,QAAQ,MAAM,WAAW,KAAK,KAAK,IAAI,IAAI,EAAE,SAAS,cAAc,KAAK,MAAM,cAAc,UAAU,CAAC;AAC9G,MAAI,CAAC,MAAO,QAAO;AACnB,QAAM,KAAK,QAAQ,KAAK;AACxB,OAAK,MAAM,6BAA6B,EAAE,OAAO,IAAI,IAAI,UAAU,MAAM,UAAU,UAAU,CAAC;AAC9F,SAAO;AACT;AAEA,eAAe,cAAc,MAAsB,KAA6D;AAC9G,QAAM,OAAO,KAAK,SAAS,IAAI,IAAI,IAAI;AACvC,MAAI,CAAC,KAAM,QAAQ,MAAM,QAAQ,MAAM,KAAK,QAAW,cAAc,qCAAqC,IAAK,WAAW;AAI1H,MAAI,IAAI,uBAAuB,KAAK,OAAO,aAAa;AACtD,WAAQ,MAAM,QAAQ,MAAM,KAAK,MAAM,iBAAiB,iDAAiD,IAAK,WAAW;AAAA,EAC3H;AAEA,QAAM,WAAW,MAAM,eAAe,KAAK,KAAK,IAAI,IAAI,EAAE,cAAc,KAAK,MAAM,aAAa,CAAC;AACjG,MAAI,CAAC,SAAU,QAAO;AACtB,QAAM,KAAK,QAAQ,QAAQ;AAC3B,OAAK,MAAM,6BAA6B,EAAE,OAAO,IAAI,IAAI,UAAU,SAAS,UAAU,QAAQ,gBAAgB,CAAC;AAC/G,SAAO;AACT;AAKA,eAAe,QACb,MACA,KACA,MACA,QACA,SACkB;AAClB,MAAI,CAAC,MAAM,cAAc;AACvB,UAAMA,UAAS,MAAM,KAAK,KAAK,KAAK,IAAI,IAAI,QAAiB,OAAO;AACpE,QAAIA,SAAQ;AACV,YAAM,KAAK,IAAI,MAAM,yEAAyE,CAAC,IAAI,EAAE,CAAC;AACtG,WAAK,MAAM,2BAA2B,EAAE,OAAO,IAAI,IAAI,OAAO,CAAC;AAAA,IACjE;AACA,WAAOA,WAAU;AAAA,EACnB;AAEA,QAAM,SAAS,MAAM,KAAK,IAAI,YAAY,OAAO,OAAO;AACtD,UAAM,MAAM,MAAM,KAAK,IAAI,IAAI,IAAI,QAAiB,OAAO;AAC3D,QAAI,CAAC,IAAK,QAAO;AACjB,UAAM,EAAE,QAAQ,IAAI,MAAM,KAAK,aAAc,KAAK,QAAQ,GAAG,GAAG,EAAE;AAClE,QAAI,UAAU,EAAG,OAAM,IAAI,MAAM,yCAAyC,IAAI,EAAE,EAAE;AAClF,UAAM,aAAa,IAAI,IAAI,EAAE;AAC7B,WAAO;AAAA,EACT,CAAC;AACD,MAAI,CAAC,OAAQ,QAAO;AACpB,QAAM,mBAAmB,MAAM,QAAQ,QAAQ,MAAM,CAAC;AACtD,OAAK,MAAM,2BAA2B,EAAE,OAAO,IAAI,IAAI,OAAO,CAAC;AAC/D,SAAO;AACT;",
+  "names": ["parked"]
+}

package/dist/core/registry.js

@@ -0,0 +1,72 @@
+import { UnknownKindError } from "./errors.js";
+const DEFAULT_LEASE = { ttlMs: 6e4, sliceBudgetMs: 3e5, pendingTtlMs: 9e5 };
+const DEFAULT_BUDGET = { maxRedrives: 10, maxConsecutiveFailures: 5, poisonRedrivesWithoutCommit: 3 };
+const DEFAULT_RETRY = { attempts: 5, backoff: { type: "exponential", delayMs: 5e3, maxDelayMs: 3e5 } };
+function resolveKind(definition) {
+  if (definition.onTransition && !definition.onRedrive) {
+    throw new Error(
+      `Kind ${JSON.stringify(definition.kind)} declares onTransition without onRedrive: a domain mirror with no way back would leave an operator able to re-drive the job while the domain row stays terminal.`
+    );
+  }
+  return {
+    ...definition,
+    lease: { ...DEFAULT_LEASE, ...definition.lease },
+    budget: { ...DEFAULT_BUDGET, ...definition.budget },
+    retry: {
+      attempts: definition.retry?.attempts ?? DEFAULT_RETRY.attempts,
+      backoff: { ...DEFAULT_RETRY.backoff, ...definition.retry?.backoff }
+    },
+    orphanPolicy: definition.orphanPolicy ?? "park",
+    concurrency: definition.concurrency ?? 1
+  };
+}
+function nextAttemptDelayMs(kind, attempt) {
+  if (attempt >= kind.retry.attempts) return null;
+  const { type, delayMs, maxDelayMs } = kind.retry.backoff;
+  const raw = type === "fixed" ? delayMs : delayMs * 2 ** Math.max(0, attempt - 1);
+  return Math.min(raw, maxDelayMs);
+}
+class KindRegistry {
+  constructor() {
+    this.kinds = /* @__PURE__ */ new Map();
+  }
+  register(definition) {
+    const resolved = resolveKind(definition);
+    const existing = this.kinds.get(definition.kind);
+    if (existing && existing.step !== definition.step) {
+      throw new Error(`Duplicate durable job kind ${JSON.stringify(definition.kind)}: two different handlers registered under one id.`);
+    }
+    this.kinds.set(definition.kind, resolved);
+  }
+  get(kind) {
+    return this.kinds.get(kind);
+  }
+  require(kind) {
+    const found = this.get(kind);
+    if (!found) throw new UnknownKindError(kind);
+    return found;
+  }
+  has(kind) {
+    return this.kinds.has(kind);
+  }
+  list() {
+    return [...this.kinds.values()];
+  }
+  queues() {
+    return [...new Set(this.list().map((k) => k.queue))];
+  }
+  clear() {
+    this.kinds.clear();
+  }
+}
+const registry = new KindRegistry();
+export {
+  DEFAULT_BUDGET,
+  DEFAULT_LEASE,
+  DEFAULT_RETRY,
+  KindRegistry,
+  nextAttemptDelayMs,
+  registry,
+  resolveKind
+};
+//# sourceMappingURL=registry.js.map

package/dist/core/registry.js.map

@@ -0,0 +1,7 @@
+{
+  "version": 3,
+  "sources": ["../../src/core/registry.ts"],
+  "sourcesContent": ["// What a caller declares when they hand work to the mechanism, and the process-wide map of\n// those declarations.\n//\n// The registry is process-wide rather than per-container because a job's handler has to be\n// resolvable in every process that can act on the job: the worker runs its slices, the web\n// process re-drives and cancels it, the reconciler parks it. A registration attached to a\n// request-scoped container would be invisible to the next request.\n\nimport { UnknownKindError } from './errors'\nimport type {\n  BudgetSettings,\n  DurableJob,\n  ErrorClass,\n  Lease,\n  LeaseSettings,\n  RetrySettings,\n  Scope,\n  SliceOutcome,\n  SqlExecutor,\n} from './types'\n\n/** What a slice is given. Everything it needs to make progress and to stop safely. */\nexport interface SliceContext<TInput = unknown, TCheckpoint = unknown> {\n  job: DurableJob\n  scope: Scope\n  lease: Lease\n  input: TInput\n  checkpoint: TCheckpoint | null\n\n  /** Aborts on shutdown, on cancellation, and when the lease is lost. A slice that checks it\n   *  at batch boundaries is the difference between a clean stop and a killed process. */\n  signal: AbortSignal\n\n  /** Milliseconds this slice may run before it should hand back. */\n  budgetMs: number\n\n  /** Stable per (job, slice). Forward it to any external side effect so a redelivered slice\n   *  is recognised as the same request rather than a second one. */\n  idempotencyKey: string\n\n  /** Extends the lease and reports progress. `committed: true` records that a unit of work\n   *  is durably written, which resets the failure and orphan budgets. Throws `LeaseLostError`\n   *  when the lease is gone. */\n  heartbeat(patch?: { processedCount?: number; totalCount?: number | null; committed?: boolean }): Promise<void>\n\n  /** Records resume state and counts as a committed unit, under the fence. */\n  checkpoint_(state: TCheckpoint, patch?: { processedCount?: number; totalCount?: number | null }): Promise<void>\n\n  /** Runs `fn` in a transaction that also re-asserts this lease. If the lease is gone the\n   *  transaction rolls back and `LeaseLostError` is thrown \u2014 so a worker that lost its lease\n   *  mid-write cannot land a write that outlives its right to make one. */\n  fencedWrite<T>(fn: (tx: SqlExecutor) => Promise<T>): Promise<T>\n\n  /** True once the slice budget is spent or a stop was requested. Check it at boundaries. */\n  shouldYield(): boolean\n}\n\nexport interface KindDefinition<TInput = unknown, TCheckpoint = unknown> {\n  kind: string\n  /** Which queue carries this kind. Kinds sharing a queue share a worker's concurrency. */\n  queue: string\n  /** ACL features an operator needs to re-drive this kind, beyond `durable_work.operate`. */\n  requiredFeatures?: string[]\n  concurrency?: number\n\n  lease?: Partial<LeaseSettings>\n  budget?: Partial<BudgetSettings>\n  retry?: { attempts?: number; backoff?: Partial<RetrySettings['backoff']> }\n\n  /** What the reconciler does with an orphan. Defaults to `park`: a job nobody declared\n   *  idempotent is not re-run automatically just because its worker died. */\n  orphanPolicy?: 'redrive' | 'park'\n\n  /** One slice under a held lease. Return when the budget is spent or the signal aborts. */\n  step(ctx: SliceContext<TInput, TCheckpoint>): Promise<SliceOutcome>\n\n  /**\n   * Mirrors a terminal transition onto the domain row, inside the terminal transaction.\n   *\n   * Must be idempotent and must return how many domain rows its update matched: `matched: 0`\n   * is treated exactly like a throw, because \"mirrored\" means \"the domain row agrees\", not\n   * \"the callback ran\". No events, no enqueues \u2014 those belong in `onAfterTransition`, which\n   * runs after the commit.\n   */\n  onTransition?(job: DurableJob, scope: Scope, tx: SqlExecutor): Promise<{ matched: number }>\n\n  /**\n   * Re-opens the domain row when an operator re-drives. The mirror image of `onTransition`,\n   * in the same transaction, with the same contract.\n   *\n   * Required whenever `onTransition` is declared \u2014 a mirror with no way back would leave an\n   * operator able to re-drive the job while the domain row stays terminal.\n   */\n  onRedrive?(job: DurableJob, scope: Scope, tx: SqlExecutor): Promise<{ matched: number }>\n\n  /** Release external resources before a cancellation commits. Same transaction, same\n   *  idempotency rule as `onTransition`. */\n  onCancel?(job: DurableJob, scope: Scope, tx: SqlExecutor): Promise<void>\n\n  /** After-commit hooks. Best-effort and at-most-once: a throw is logged, never retried, and\n   *  never affects the committed row. Domain events and log writes belong here. */\n  onAfterTransition?(job: DurableJob, scope: Scope): Promise<void>\n  onAfterRedrive?(job: DurableJob, scope: Scope): Promise<void>\n\n  /** Override the default classification for errors this kind understands. */\n  classify?(error: unknown): ErrorClass | null\n}\n\nexport const DEFAULT_LEASE: LeaseSettings = { ttlMs: 60_000, sliceBudgetMs: 300_000, pendingTtlMs: 900_000 }\nexport const DEFAULT_BUDGET: BudgetSettings = { maxRedrives: 10, maxConsecutiveFailures: 5, poisonRedrivesWithoutCommit: 3 }\nexport const DEFAULT_RETRY: RetrySettings = { attempts: 5, backoff: { type: 'exponential', delayMs: 5_000, maxDelayMs: 300_000 } }\n\nexport type ResolvedKind<TInput = unknown, TCheckpoint = unknown> = KindDefinition<TInput, TCheckpoint> & {\n  lease: LeaseSettings\n  budget: BudgetSettings\n  retry: RetrySettings\n  orphanPolicy: 'redrive' | 'park'\n  concurrency: number\n}\n\nexport function resolveKind<TInput, TCheckpoint>(definition: KindDefinition<TInput, TCheckpoint>): ResolvedKind<TInput, TCheckpoint> {\n  if (definition.onTransition && !definition.onRedrive) {\n    throw new Error(\n      `Kind ${JSON.stringify(definition.kind)} declares onTransition without onRedrive: a domain mirror with no way back would leave an operator able to re-drive the job while the domain row stays terminal.`,\n    )\n  }\n  return {\n    ...definition,\n    lease: { ...DEFAULT_LEASE, ...definition.lease },\n    budget: { ...DEFAULT_BUDGET, ...definition.budget },\n    retry: {\n      attempts: definition.retry?.attempts ?? DEFAULT_RETRY.attempts,\n      backoff: { ...DEFAULT_RETRY.backoff, ...definition.retry?.backoff },\n    },\n    orphanPolicy: definition.orphanPolicy ?? 'park',\n    concurrency: definition.concurrency ?? 1,\n  }\n}\n\n/** The delay before the transport's next attempt, or null when none is coming. */\nexport function nextAttemptDelayMs(kind: ResolvedKind, attempt: number): number | null {\n  if (attempt >= kind.retry.attempts) return null\n  const { type, delayMs, maxDelayMs } = kind.retry.backoff\n  const raw = type === 'fixed' ? delayMs : delayMs * 2 ** Math.max(0, attempt - 1)\n  return Math.min(raw, maxDelayMs)\n}\n\nexport class KindRegistry {\n  private readonly kinds = new Map<string, ResolvedKind<never, never>>()\n\n  register<TInput, TCheckpoint>(definition: KindDefinition<TInput, TCheckpoint>): void {\n    const resolved = resolveKind(definition)\n    const existing = this.kinds.get(definition.kind)\n    // Re-registering the identical definition is a no-op so a module loaded twice (two entry\n    // points, a test re-import) is not a crash; a *different* definition under the same id is\n    // a genuine conflict and must not be resolved silently by last-write-wins.\n    if (existing && existing.step !== definition.step) {\n      throw new Error(`Duplicate durable job kind ${JSON.stringify(definition.kind)}: two different handlers registered under one id.`)\n    }\n    this.kinds.set(definition.kind, resolved as unknown as ResolvedKind<never, never>)\n  }\n\n  get(kind: string): ResolvedKind | undefined {\n    return this.kinds.get(kind) as ResolvedKind | undefined\n  }\n\n  require(kind: string): ResolvedKind {\n    const found = this.get(kind)\n    if (!found) throw new UnknownKindError(kind)\n    return found\n  }\n\n  has(kind: string): boolean {\n    return this.kinds.has(kind)\n  }\n\n  list(): ResolvedKind[] {\n    return [...this.kinds.values()] as ResolvedKind[]\n  }\n\n  queues(): string[] {\n    return [...new Set(this.list().map((k) => k.queue))]\n  }\n\n  clear(): void {\n    this.kinds.clear()\n  }\n}\n\n/**\n * The process-wide registry. Modules register into this at import time.\n *\n * Module-scoped, so there is exactly one per copy of this package in the process \u2014 which is why\n * anything registering kinds must depend on this package as a PEER, never as a dependency. A\n * nested second copy would give the adopter its own registry: kinds would register into one,\n * the worker would read the other, and nothing would run. No error, no warning, just jobs that\n * sit pending forever while the reconciler eventually parks them `no_handler`.\n */\nexport const registry = new KindRegistry()\n"],
+  "mappings": "AAQA,SAAS,wBAAwB;AAoG1B,MAAM,gBAA+B,EAAE,OAAO,KAAQ,eAAe,KAAS,cAAc,IAAQ;AACpG,MAAM,iBAAiC,EAAE,aAAa,IAAI,wBAAwB,GAAG,6BAA6B,EAAE;AACpH,MAAM,gBAA+B,EAAE,UAAU,GAAG,SAAS,EAAE,MAAM,eAAe,SAAS,KAAO,YAAY,IAAQ,EAAE;AAU1H,SAAS,YAAiC,YAAoF;AACnI,MAAI,WAAW,gBAAgB,CAAC,WAAW,WAAW;AACpD,UAAM,IAAI;AAAA,MACR,QAAQ,KAAK,UAAU,WAAW,IAAI,CAAC;AAAA,IACzC;AAAA,EACF;AACA,SAAO;AAAA,IACL,GAAG;AAAA,IACH,OAAO,EAAE,GAAG,eAAe,GAAG,WAAW,MAAM;AAAA,IAC/C,QAAQ,EAAE,GAAG,gBAAgB,GAAG,WAAW,OAAO;AAAA,IAClD,OAAO;AAAA,MACL,UAAU,WAAW,OAAO,YAAY,cAAc;AAAA,MACtD,SAAS,EAAE,GAAG,cAAc,SAAS,GAAG,WAAW,OAAO,QAAQ;AAAA,IACpE;AAAA,IACA,cAAc,WAAW,gBAAgB;AAAA,IACzC,aAAa,WAAW,eAAe;AAAA,EACzC;AACF;AAGO,SAAS,mBAAmB,MAAoB,SAAgC;AACrF,MAAI,WAAW,KAAK,MAAM,SAAU,QAAO;AAC3C,QAAM,EAAE,MAAM,SAAS,WAAW,IAAI,KAAK,MAAM;AACjD,QAAM,MAAM,SAAS,UAAU,UAAU,UAAU,KAAK,KAAK,IAAI,GAAG,UAAU,CAAC;AAC/E,SAAO,KAAK,IAAI,KAAK,UAAU;AACjC;AAEO,MAAM,aAAa;AAAA,EAAnB;AACL,SAAiB,QAAQ,oBAAI,IAAwC;AAAA;AAAA,EAErE,SAA8B,YAAuD;AACnF,UAAM,WAAW,YAAY,UAAU;AACvC,UAAM,WAAW,KAAK,MAAM,IAAI,WAAW,IAAI;AAI/C,QAAI,YAAY,SAAS,SAAS,WAAW,MAAM;AACjD,YAAM,IAAI,MAAM,8BAA8B,KAAK,UAAU,WAAW,IAAI,CAAC,mDAAmD;AAAA,IAClI;AACA,SAAK,MAAM,IAAI,WAAW,MAAM,QAAiD;AAAA,EACnF;AAAA,EAEA,IAAI,MAAwC;AAC1C,WAAO,KAAK,MAAM,IAAI,IAAI;AAAA,EAC5B;AAAA,EAEA,QAAQ,MAA4B;AAClC,UAAM,QAAQ,KAAK,IAAI,IAAI;AAC3B,QAAI,CAAC,MAAO,OAAM,IAAI,iBAAiB,IAAI;AAC3C,WAAO;AAAA,EACT;AAAA,EAEA,IAAI,MAAuB;AACzB,WAAO,KAAK,MAAM,IAAI,IAAI;AAAA,EAC5B;AAAA,EAEA,OAAuB;AACrB,WAAO,CAAC,GAAG,KAAK,MAAM,OAAO,CAAC;AAAA,EAChC;AAAA,EAEA,SAAmB;AACjB,WAAO,CAAC,GAAG,IAAI,IAAI,KAAK,KAAK,EAAE,IAAI,CAAC,MAAM,EAAE,KAAK,CAAC,CAAC;AAAA,EACrD;AAAA,EAEA,QAAc;AACZ,SAAK,MAAM,MAAM;AAAA,EACnB;AACF;AAWO,MAAM,WAAW,IAAI,aAAa;",
+  "names": []
+}