@checkstack/backend-api 0.18.0 → 0.20.0

package/CHANGELOG.md

@@ -1,5 +1,256 @@
 # @checkstack/backend-api
 
+## 0.20.0
+
+### Minor Changes
+
+- a57f7db: fix(backend): give advisory locks a dedicated connection pool to prevent pool-starvation deadlock
+
+  Both the session-lock service and `withXactLock` HOLD a Postgres connection for
+  the lock's whole lifetime while the gated work runs on a _different_ connection.
+  Both lock and work were drawing from the single shared `adminPool` (which, with
+  no explicit config, defaulted to `max: 10` and `connectionTimeoutMillis: 0` -
+  wait forever). Under concurrency >= pool size, every slot became a lock-holding
+  connection waiting for a work connection that could never free up: a permanent
+  deadlock. It surfaced as all connections stuck `idle in transaction` on
+  `pg_advisory_xact_lock` and every API request hanging into an upstream 502,
+  only after the server had been running long enough to hit that concurrency
+  (e.g. a burst of health-check evaluations or incident dedups).
+
+  Advisory locks now run on a dedicated `lockPool`, separate from `adminPool`, so
+  the acquire graph is acyclic (`lockPool -> adminPool`, never back) and the
+  deadlock class is impossible. `AdvisoryLockService` gains a pooled
+  `withXactLock({ key, fn })` method (lock on the lock pool, work on the admin
+  pool); healthcheck's per-system serializer, incident's dedup-create, and the
+  automation single-mode concurrency lock now use it. The deadlock-prone
+  standalone `withXactLock({ db, ... })` helper is REMOVED.
+
+  Both pools are explicitly configured with `connectionTimeoutMillis` so any
+  future exhaustion fails fast and self-heals instead of hanging, and both get a
+  pool-level `error` handler (an idle pooled client whose backend dies otherwise
+  crashes the pod). The lock pool additionally sets
+  `idle_in_transaction_session_timeout` and `lock_timeout` so a stalled critical
+  section is reaped server-side (auto-releasing the lock) rather than stranding a
+  key forever. The advisory-lock service also now removes its per-client error
+  listener on release (it previously leaked one listener per acquisition on each
+  reused pooled connection - an unbounded `MaxListenersExceeded` leak).
+
+  New env vars (all optional): `DATABASE_POOL_MAX` (default 20),
+  `DATABASE_LOCK_POOL_MAX` (default 10), `DATABASE_POOL_CONNECTION_TIMEOUT_MS`
+  (default 10000), `DATABASE_POOL_IDLE_TIMEOUT_MS` (default 30000),
+  `DATABASE_LOCK_IDLE_TX_TIMEOUT_MS` (default 30000), `DATABASE_LOCK_TIMEOUT_MS`
+  (default 30000). Size pools off
+  `N_pods * (DATABASE_POOL_MAX + DATABASE_LOCK_POOL_MAX) <= max_connections`.
+
+  BREAKING CHANGE: the standalone `withXactLock({ db, key, fn })` export is
+  removed - use `coreServices.advisoryLock.withXactLock({ key, fn })` instead.
+  `IncidentService`'s constructor now requires an `AdvisoryLockService` as its
+  second argument, and the healthcheck `createHealthEntitySerializer` /
+  `executeHealthCheckJob` / `setupHealthCheckWorker` helpers take `advisoryLock`
+  instead of `db` for the serializer.
+
+### Patch Changes
+
+- @checkstack/cache-api@0.3.8
+- @checkstack/queue-api@0.3.8
+
+## 0.19.0
+
+### Minor Changes
+
+- 270ef29: Fix automation provider actions and `secretEnv` script actions throwing in production.
+
+  The automation dispatch engine resolved provider-action dependencies (the integration connection store, the secret resolver) through a `getService` that was a throwing stub, so Jira / Teams / Webex actions and `secretEnv` script actions threw at execute time in production. The whole dispatch test suite stubbed `getService`, so the break was invisible.
+
+  Root cause: the plugin `env` exposed `registerService` but no resolver, so the dispatch path (the only context that resolves arbitrary cross-plugin refs outside an RPC handler) had nothing real to call.
+
+  Changes:
+
+  - `@checkstack/backend-api`: add `getService<S>(ref: ServiceRef<S>): Promise<S>` to the plugin `env` (`BackendPluginRegistry`). It resolves a service registered by any plugin through the real `ServiceRegistry` using the calling plugin's identity, and throws a clear error if the ref is not registered (never silently `undefined`). **NEW PLUGIN-AUTHOR CONTRACT**: `env.getService` is now available to resolve arbitrary cross-plugin service refs at init / afterPluginsReady time.
+  - `@checkstack/backend`: implement `env.getService` in both the plugin loader and the runtime single-plugin registration path, backed by `ServiceRegistry.get(ref, { pluginId })`.
+  - `@checkstack/automation-backend`: wire the dispatch `getService` to `env.getService` (was a throwing stub). This also activates run-wide provider-credential masking, because resolving the connection store / secret resolver now flows through the run's masking interceptor.
+
+  Also fixes a test-only seam where the `core/backend` test preload registered a no-op `registerRouter`, silently disabling oRPC router registration across the suite.
+
+- 270ef29: Fix suspend/resume durability + complete the run-wide secret-masking guarantee.
+
+  A panel review confirmed several defects in the automation dispatch engine's suspend/resume durability and in the run-wide masking choke point. These survived because the unit suite stubbed the seam under test; the fixes ship with tests that exercise the real suspend / sweep / resume paths.
+
+  Suspend/resume durability:
+
+  - **Stalled sweeper no longer re-runs intentional waits.** `findStalledRunIds` now joins `automation_runs` and returns only `status = 'running'` runs, and suspend-finalisation no longer clobbers the run's `lastActionPath` checkpoint to `null`. Previously any wait longer than the stale window (>60s) was re-walked from the top every sweep cycle, re-firing pre-wait side effects and leaking wait locks. The wait-aware sweeps now also run before the stalled-run sweep.
+  - **Stalled recovery refuses a run holding a live wait lock.** `recoverStalledRun` now only recovers a genuinely-`running` run with no wait lock; a crash-mid-wait recovery is left to the wait/resume paths instead of re-walking from the top and creating a duplicate lock + duplicate delay job.
+  - **Cancelled runs can no longer resurrect.** `resumeRun` guards on `status === 'waiting'` (mirroring `checkWaitUntil`) and drops any stale lock for a non-waiting run, so `wakeWaitingRuns` / delay-expiry / a racing queue job can't wake a cancelled or terminal run. `cancelActiveRuns` (restart mode) now deletes the cancelled runs' wait locks + run-state in the same operation.
+  - **Concurrency check-then-create is serialized.** The `mode` check + `createRun` now run under a transaction-scoped advisory lock keyed on `(automationId, scope)`, so two concurrent fires can't both pass a `single`-mode "no active run" check and double-run.
+
+  Masking guarantee (now genuinely covers scope + artifacts):
+
+  - **The run-wide masking choke point now also masks the durable scope snapshot and produced artifacts.** The `RunSecretRegistry` is threaded into `RunStateStore.upsert` (masks `scopeSnapshot`) and `ArtifactStore.record` (masks `data`) so a resolved connection credential threaded into `scope.variables` or surfaced into an artifact is redacted before persist - and therefore cannot reach a read-only user via `getRunScopeForReplay`. **GUARANTEE CHANGE**: run-wide masking now covers step output, run error, scope snapshot, and artifact data for every action.
+  - **`testConnection` / `testProviderConnection` mask provider errors.** These RPCs run outside a dispatch run, so they build a per-call mask set from the resolved/submitted connection config and run any provider error through it before returning, so a provider error echoing a token can't cross back to the browser.
+  - **Short secrets surface a warning.** `setSecret` now warns when a value is shorter than `MIN_MASKABLE_LENGTH` (4) that it cannot be auto-redacted (the threshold is intentionally not lowered).
+
+  Internal:
+
+  - `@checkstack/backend-api`: `withXactLock`'s `fn` now receives the transaction handle `tx` so a critical section can run on the locked connection; the doc clarifies why running on the pool inside the lock window is still safe. The incident dedup caller's comment is corrected accordingly. `RunStore` gains `findWaitLocksByRun`.
+
+- 270ef29: Fix several correctness defects around distributed coordination and stored-data handling.
+
+  - Dwell `for:` timers now fire via an atomic `DELETE ... RETURNING` claim, so two pods (or the stalled sweeper vs the queue consumer) can no longer both fire the same dwell.
+  - Postgres session-level advisory locks now keep connection affinity. A shared `AdvisoryLockService` (backed by a dedicated pooled client) replaces the previous acquire/release-on-different-connection pattern that leaked locks. Used by the script-packages installer election, the automation run resume + stalled sweeper, and (via a new transaction-scoped `withXactLock`) incident dedup.
+  - A storage migration that crashed mid-flight is now resumed on startup under the installer-election lock, instead of permanently wedging installs.
+  - Distributed script-package blobs carry a `blobSha256` and are verified before extraction (the SRI `integrity` hashes the npm tarball, not the transported archive). Backward-safe: entries without the field skip verification until a re-install regenerates the manifest.
+  - Archive extraction rejects zip-slip paths (absolute or `..` entries) before writing anything.
+  - `incident.create` with `dedupe_open_for_system` serializes its check-then-create per system, so concurrent triggers for the same system can't both open a duplicate incident.
+  - Seeded auto-incident filter expressions JSON-encode interpolated ids so a quote/backslash can't corrupt the expression.
+  - Stored jsonb snapshots (dwell `actorSnapshot`, wait-lock `waitConfig`) are validated with zod on load and degrade safely instead of flowing through as the wrong type.
+
+- b995afb: Harden the advisory-lock service against holder-connection termination.
+
+  A session-level advisory lock is held on a dedicated checked-out pool client.
+  If that backend is terminated (admin kill, failover, network drop) while the
+  lock is held, `pg` emits an `'error'` on the client; with no listener attached
+  that error is re-thrown by the EventEmitter and crashes the pod. The service
+  now attaches an error listener to the held client so the loss degrades
+  gracefully - the session lock is auto-released server-side when the backend
+  dies, and the key simply becomes acquirable again.
+
+  Also de-flaked the advisory-lock integration test: it now terminates only the
+  lock-holding backend (found via `pg_locks`) instead of every backend in the
+  database - the old blanket kill also tore down the pool's idle connections,
+  whose async errors flaked the run and left the pool unusable.
+
+- 270ef29: Add in-UI script testing for automation `run_script` / `run_shell` actions.
+
+  A new `testScript` RPC runs a TypeScript or shell script against an
+  editable, auto-seeded sample context using the same sandboxed runner the
+  real action uses, so operators can test scripts directly in the editor
+  without dispatching a whole automation. Surfaces beneath any script field
+  flagged `x-script-testable` via the new `ScriptTestPanel` /
+  `ContextSampleEditor` components in `@checkstack/ui` and the
+  `scriptTestRenderer` prop threaded through `DynamicForm`.
+
+  - `@checkstack/automation-common`: adds the `testScript` contract +
+    `ScriptTest*` schemas (gated by `automation.manage`).
+  - `@checkstack/automation-backend`: implements `testScript` reusing the
+    shared ESM / shell runners; central-only, time-bounded.
+  - `@checkstack/backend-api`: new `x-script-testable` config-schema
+    metadata propagated to the frontend JSON Schema.
+  - `@checkstack/ui`: new `ScriptTestPanel` + `ContextSampleEditor`
+    components and a `scriptTestRenderer` prop on `DynamicForm`.
+  - `@checkstack/automation-frontend`: wires the test panel into the action
+    editor.
+  - `@checkstack/integration-script-backend`: marks the `run_script` /
+    `run_shell` script fields as testable.
+
+- 270ef29: Activate npm packages in script execution: thread the managed
+  `resolutionRoot` into every user-script call site so an allowlisted package
+  can actually be `import`ed.
+
+  - `@checkstack/backend-api`: the ESM runner now always writes a per-run
+    `bunfig.toml` with `[install] auto = "disable"` and runs with that dir as
+    CWD. Without this Bun silently auto-installs any imported package from the
+    registry (verified), defeating the allowlist; with it, imports resolve
+    only against the reconciled `current/node_modules` (when a `resolutionRoot`
+    is set) and otherwise fail fast.
+  - `@checkstack/script-packages-backend`: `resolveResolutionRoot` /
+    `resolveResolutionRootFromStore` / `resolveResolutionRootForHost` decide a
+    host's resolution-root status (`none` / `ready` / `notReady`) from the
+    local `<store>/current`.
+  - `run_script` (integration-script-backend), the inline-script collector
+    (healthcheck-script-backend, core + satellite), and the in-UI `testScript`
+    / `testCollectorScript` endpoints all resolve the root per run and pass it
+    to the runner; `run_script` surfaces a clear "npm packages not ready"
+    error when configured-but-unsynced. Shell paths are unaffected (no module
+    resolution).
+
+  An opt-in end-to-end test (`CHECKSTACK_E2E_NETWORK=1`) proves an allowlisted
+  package imports successfully through the real `run_script` action execute
+  path, with non-network degradation tests running always.
+
+  BREAKING CHANGES: `@checkstack/backend-api`'s `defaultEsmScriptRunner` now
+  always disables Bun auto-install for the user subprocess. A script that
+  previously relied on Bun silently fetching an un-vendored package from the
+  registry at import time will now fail to resolve it. This is intentional -
+  package availability is governed by the admin allowlist - but any caller
+  depending on the old implicit auto-install behavior must add the package to
+  the allowlist instead. The new `EsmScriptRunOptions.resolutionRoot` field is
+  optional and additive (defaults to today's `os.tmpdir()` behavior when
+  unset), so the runner API itself is source-compatible.
+
+- 270ef29: Add the per-host script-package reconciler and the runner resolution root.
+
+  - `@checkstack/backend-api`: `EsmScriptRunOptions.resolutionRoot` - when
+    set, the per-run temp dir is created inside it so module resolution walks
+    up to `<resolutionRoot>/node_modules` and user scripts can `import`
+    managed npm packages. Defaults to today's `os.tmpdir()` behavior when
+    unset (backward-compatible; isolation unchanged - the subprocess still
+    only sees `SAFE_ENV_VARS`).
+  - `@checkstack/script-packages-backend`: content-addressed cache archive
+    (tar+gzip per package), pure delta diff (`computeMissingBlobs`), atomic
+    `current` symlink swap, the host reconciler (`reconcileToHash` -
+    idempotent: pull only missing blobs, materialize a versioned tree via
+    `bun install --offline`, atomically flip `current`), the concrete fs/Bun
+    adapter, the central install resolver, and the `script-packages.changed`
+    broadcast hook. An opt-in end-to-end test
+    (`CHECKSTACK_E2E_NETWORK=1`) proves resolve -> publish -> cold reconcile
+    (no registry) -> offline materialize -> import.
+
+- 270ef29: Secrets platform Phase 2: secret -> env-var mapping with central resolve, inject, and mask.
+
+  - Script consumers declare a least-privilege `secretEnv` allowlist
+    (`{ ENV_NAME: "${{ secrets.NAME }}" }`). The automation `run_script` /
+    `run_shell` actions resolve ONLY the declared secrets via
+    `secretResolverRef.resolveForRun`, inject them into the runner env for
+    that run (memory-only; the ESM runner gained a per-run `env` option), and
+    mask their values out of stdout/stderr/result/error via the run-scoped
+    masking context. A missing required secret fails the run clearly. No
+    ambient secret access.
+  - Test panel: `testScript` / `testCollectorScript` inject named
+    `__SECRET_<NAME>__` placeholders by default, or user-supplied per-secret
+    overrides; real production values are never resolved in the test path,
+    and overrides are masked out of the result.
+  - Healthcheck collectors carry the `secretEnv` field for authoring +
+    the test panel; runtime injection on satellites lands in Phase 3.
+  - Editor UX: a new `@checkstack/ui` `SecretEnvEditor` renders `x-secret-env`
+    record fields with `${{ secrets.* }}` name autocomplete (from
+    `listSecretNames`), wired into the automation action editor and the
+    healthcheck collector editor. New `withConfigMeta` helper +
+    `x-secret-env` config-meta key in `@checkstack/backend-api`.
+
+- 270ef29: Secrets platform Phase 3: just-in-time secret delivery to satellites + source-side masking, and central-execution injection for healthcheck collectors.
+
+  - New satellite WS messages `request_run_secrets` / `run_secrets`: just
+    before a satellite runs a collector that declares a `secretEnv`, it asks
+    core for that collector's resolved env; core resolves ONLY the secrets the
+    collector's OWN persisted assignment declares (least-privilege — the
+    satellite cannot choose) and replies with the env map (or a clear error).
+    The satellite injects it memory-only for the run and drops it on
+    completion. Secrets never ride the persisted assignment and never touch
+    disk.
+  - Source-side masking: the satellite runs `maskSecrets` over the collector's
+    stdout/stderr/result/error using the run's delivered values BEFORE the
+    result leaves the satellite (defense in depth).
+  - `CollectorStrategy.execute` gains an optional `secretEnv`. The
+    inline-script and shell collectors inject it into the runner
+    (`process.env` / `$VAR`) and mask the values out of their output.
+  - Healthcheck collectors running centrally (the queue executor) also resolve
+    - inject `secretEnv` via `secretResolverRef`, closing the gap where a
+      centrally-run secretEnv collector got no secrets. A missing required
+      secret fails the run clearly in all paths.
+
+### Patch Changes
+
+- Updated dependencies [270ef29]
+- Updated dependencies [270ef29]
+- Updated dependencies [270ef29]
+- Updated dependencies [b995afb]
+- Updated dependencies [b995afb]
+- Updated dependencies [270ef29]
+- Updated dependencies [270ef29]
+  - @checkstack/healthcheck-common@1.4.0
+  - @checkstack/cache-api@0.3.7
+  - @checkstack/queue-api@0.3.7
+
 ## 0.18.0
 
 ### Minor Changes

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@checkstack/backend-api",
-  "version": "0.18.0",
+  "version": "0.20.0",
   "license": "Elastic-2.0",
   "type": "module",
   "main": "./src/index.ts",
@@ -10,11 +10,11 @@
     "lint:code": "eslint . --max-warnings 0"
   },
   "dependencies": {
-    "@checkstack/common": "0.11.0",
-    "@checkstack/healthcheck-common": "1.2.0",
-    "@checkstack/cache-api": "0.3.5",
-    "@checkstack/queue-api": "0.3.5",
-    "@checkstack/signal-common": "0.2.4",
+    "@checkstack/common": "0.12.0",
+    "@checkstack/healthcheck-common": "1.3.0",
+    "@checkstack/cache-api": "0.3.6",
+    "@checkstack/queue-api": "0.3.6",
+    "@checkstack/signal-common": "0.2.5",
     "@orpc/client": "^1.13.14",
     "@orpc/contract": "^1.13.14",
     "@orpc/openapi": "^1.13.2",
@@ -26,9 +26,11 @@
     "zod": "^4.2.1"
   },
   "devDependencies": {
-    "@types/bun": "latest",
+    "@checkstack/scripts": "0.3.4",
     "@checkstack/tsconfig": "0.0.7",
-    "@checkstack/scripts": "0.3.3"
+    "@types/bun": "latest",
+    "@types/pg": "^8.20.0",
+    "pg": "^8.21.0"
   },
   "peerDependencies": {
     "hono": "^4.12.14",

package/src/advisory-lock-pool.it.test.ts

@@ -0,0 +1,282 @@
+/**
+ * Integration test (real Postgres) for the advisory-lock CONNECTION-POOL
+ * contract — the behaviour that silently wedged production and that fakes
+ * cannot model: a held advisory lock keeps its connection checked out while the
+ * gated work runs on a *different* connection, so lock-pool / work-pool sizing
+ * decides whether the system makes progress or deadlocks.
+ *
+ * It pins three things against a live server:
+ *
+ *   1. REPRODUCE THE BUG: when the lock and its work share ONE pool, concurrency
+ *      at the pool size deadlocks (every slot is a lock-holder waiting for a
+ *      work connection that can never free up). This is a guard — if a refactor
+ *      makes this stop deadlocking, the throughput test below is no longer
+ *      proving anything.
+ *   2. THE FIX: with the lock on a DEDICATED pool, the same (and much higher)
+ *      concurrency completes with zero failures.
+ *   3. CORRECTNESS ACROSS INSTANCES: independent service instances with their
+ *      OWN pools (simulating N pods on one database) serialize a find-then-
+ *      create on a shared key down to exactly ONE row — with a no-lock control
+ *      proving the lock is what enforces it.
+ *
+ * Gated behind `CHECKSTACK_IT=1`; the integration CI job provides the Postgres
+ * service container. Connection from `CHECKSTACK_IT_PG_URL`.
+ */
+import { afterAll, beforeAll, describe, expect, it } from "bun:test";
+import { Pool } from "pg";
+import { createAdvisoryLockService } from "./advisory-lock";
+
+const PG_URL =
+  process.env.CHECKSTACK_IT_PG_URL ??
+  "postgres://postgres:postgres@localhost:5432/postgres";
+
+const DEDUP_TABLE = "it_advisory_dedup";
+
+describe.skipIf(!process.env.CHECKSTACK_IT)(
+  "advisory-lock pool contract (real Postgres)",
+  () => {
+    /** Pools created during a test; ended in afterEach-style cleanup helpers. */
+    const tracked: Pool[] = [];
+    function makePool(max: number, connectionTimeoutMillis = 5000): Pool {
+      const pool = new Pool({
+        connectionString: PG_URL,
+        max,
+        connectionTimeoutMillis,
+        idleTimeoutMillis: 1000,
+      });
+      // A held-lock client can error asynchronously (timeout / termination);
+      // swallow so it never surfaces as an unhandled error and fails the file.
+      pool.on("error", () => {});
+      tracked.push(pool);
+      return pool;
+    }
+    async function endTrackedPools(): Promise<void> {
+      await Promise.all(tracked.splice(0).map((p) => p.end().catch(() => {})));
+    }
+
+    let setupPool: Pool;
+    beforeAll(async () => {
+      setupPool = new Pool({ connectionString: PG_URL });
+      await setupPool.query(
+        `CREATE TABLE IF NOT EXISTS ${DEDUP_TABLE} (lock_key text NOT NULL, id text NOT NULL)`,
+      );
+    });
+    afterAll(async () => {
+      await setupPool.query(`DROP TABLE IF EXISTS ${DEDUP_TABLE}`);
+      await setupPool.end();
+      await endTrackedPools();
+    });
+
+    /**
+     * Find-then-create on `workPool`: insert exactly once per key. The 15ms gap
+     * between the read and the write widens the race window so an UNSERIALIZED
+     * run reliably double-inserts — making the lock's effect observable.
+     */
+    async function dedupCreate(workPool: Pool, key: string): Promise<boolean> {
+      const client = await workPool.connect();
+      try {
+        const { rows } = await client.query(
+          `SELECT id FROM ${DEDUP_TABLE} WHERE lock_key = $1 LIMIT 1`,
+          [key],
+        );
+        if (rows.length > 0) return false;
+        await new Promise((r) => setTimeout(r, 15));
+        await client.query(
+          `INSERT INTO ${DEDUP_TABLE} (lock_key, id) VALUES ($1, $2)`,
+          [key, crypto.randomUUID()],
+        );
+        return true;
+      } finally {
+        client.release();
+      }
+    }
+
+    async function countFor(key: string): Promise<number> {
+      const { rows } = await setupPool.query<{ n: string }>(
+        `SELECT count(*)::text AS n FROM ${DEDUP_TABLE} WHERE lock_key = $1`,
+        [key],
+      );
+      return Number(rows[0]?.n ?? "0");
+    }
+
+    it(
+      "REPRODUCES the deadlock when lock + work share one pool (the bug)",
+      async () => {
+        const POOL_MAX = 4;
+        // Single shared pool — the pre-fix wiring. The lock client AND the work
+        // client both come from here. Short connect timeout so the deadlock
+        // surfaces as a fast rejection rather than a long hang.
+        const pool = makePool(POOL_MAX, 1500);
+        const svc = createAdvisoryLockService(pool);
+        const runId = crypto.randomUUID();
+
+        // Exactly POOL_MAX concurrent ops, each on a DISTINCT key (so there is
+        // NO lock contention — the only thing that can stall is connection
+        // accounting). Each holds a lock client, then asks the same pool for a
+        // work client that will never come.
+        const results = await Promise.allSettled(
+          Array.from({ length: POOL_MAX }, (_, i) =>
+            svc.withXactLock({
+              key: `deadlock:${runId}:${i}`,
+              fn: async () => {
+                const c = await pool.connect();
+                try {
+                  await c.query("SELECT 1");
+                } finally {
+                  c.release();
+                }
+              },
+            }),
+          ),
+        );
+
+        const rejected = results.filter((r) => r.status === "rejected").length;
+        // The deadlock manifests as connection-acquire timeouts on the work
+        // checkout. If this ever becomes 0, the single-pool design no longer
+        // deadlocks and the throughput proof below must be re-examined.
+        expect(rejected).toBeGreaterThan(0);
+
+        await endTrackedPools();
+      },
+      30_000,
+    );
+
+    it(
+      "does NOT deadlock under high throughput with a dedicated lock pool (the fix)",
+      async () => {
+        // Deliberately TINY pools so any deadlock would be hit immediately; the
+        // fix is that lock and work draw from different pools.
+        const lockPool = makePool(4);
+        const workPool = makePool(4);
+        const svc = createAdvisoryLockService(lockPool);
+        const runId = crypto.randomUUID();
+
+        const CONCURRENCY = 200;
+        const results = await Promise.allSettled(
+          Array.from({ length: CONCURRENCY }, (_, i) =>
+            svc.withXactLock({
+              key: `throughput:${runId}:${i}`,
+              fn: async () => {
+                const c = await workPool.connect();
+                try {
+                  await c.query("SELECT 1");
+                } finally {
+                  c.release();
+                }
+              },
+            }),
+          ),
+        );
+
+        const rejected = results.filter((r) => r.status === "rejected");
+        // Every single operation must complete: no deadlock, no timeout.
+        expect(rejected).toHaveLength(0);
+
+        await endTrackedPools();
+      },
+      30_000,
+    );
+
+    it(
+      "serializes find-then-create across INSTANCES to exactly one row",
+      async () => {
+        // Each "pod" is an independent service instance with its OWN pools, all
+        // pointing at the same database — the real multi-instance topology. The
+        // advisory lock space is global to the server, so they must serialize.
+        const PODS = 6;
+        const ATTEMPTS_PER_POD = 5;
+        const key = `dedupe:${crypto.randomUUID()}`;
+
+        const pods = Array.from({ length: PODS }, () => {
+          const workPool = makePool(2);
+          const svc = createAdvisoryLockService(makePool(2));
+          return { workPool, svc };
+        });
+
+        const attempts = pods.flatMap((pod) =>
+          Array.from({ length: ATTEMPTS_PER_POD }, () =>
+            pod.svc.withXactLock({
+              key,
+              fn: () => dedupCreate(pod.workPool, key),
+            }),
+          ),
+        );
+
+        const settled = await Promise.allSettled(attempts);
+        const created = settled.filter(
+          (r) => r.status === "fulfilled" && r.value === true,
+        ).length;
+
+        // Exactly one attempt created the row; the rest observed it and no-oped.
+        expect(await countFor(key)).toBe(1);
+        expect(created).toBe(1);
+
+        await endTrackedPools();
+      },
+      30_000,
+    );
+
+    it(
+      "a STALLED critical section is reaped by idle_in_transaction_session_timeout, freeing the key",
+      async () => {
+        // The lock pool sets a short idle-in-transaction timeout. A held lock
+        // sits "idle in transaction" for the whole time `fn` runs, so a hung
+        // `fn` trips it: Postgres aborts the session, auto-releasing the lock -
+        // proving a stall self-heals instead of stranding the key forever.
+        const lockPool = new Pool({
+          connectionString: PG_URL,
+          max: 4,
+          connectionTimeoutMillis: 5000,
+          idle_in_transaction_session_timeout: 1000,
+        });
+        lockPool.on("error", () => {});
+        tracked.push(lockPool);
+        const svc = createAdvisoryLockService(lockPool);
+        const key = `stall:${crypto.randomUUID()}`;
+
+        let releaseHang!: () => void;
+        const hang = new Promise<void>((r) => (releaseHang = r));
+
+        // Holder whose critical section hangs (never issues another query).
+        const stalled = svc
+          .withXactLock({ key, fn: () => hang })
+          .catch(() => "rejected-as-expected");
+
+        // Wait past the 1s idle timeout so the server reaps the stalled holder.
+        await new Promise((r) => setTimeout(r, 1800));
+
+        // The key must be acquirable again now that the stalled session was
+        // aborted server-side.
+        const t0 = Date.now();
+        const got = await svc.withXactLock({ key, fn: async () => "ok" });
+        expect(got).toBe("ok");
+        expect(Date.now() - t0).toBeLessThan(3000);
+
+        releaseHang();
+        await stalled; // let the stalled call unwind (COMMIT fails on dead conn)
+        await endTrackedPools();
+      },
+      30_000,
+    );
+
+    it(
+      "CONTROL: the same workload WITHOUT the lock races into duplicates",
+      async () => {
+        // Proves the lock — not some incidental ordering — is what enforces
+        // single-creation above. Same widened-window find-then-create, run
+        // concurrently with NO advisory lock, must double-insert.
+        const workPool = makePool(8);
+        const key = `dedupe-nolock:${crypto.randomUUID()}`;
+
+        await Promise.all(
+          Array.from({ length: 8 }, () => dedupCreate(workPool, key)),
+        );
+
+        expect(await countFor(key)).toBeGreaterThan(1);
+
+        await endTrackedPools();
+      },
+      30_000,
+    );
+  },
+);