velocious 1.0.611 → 1.0.613

package/README.md

@@ -2396,7 +2396,7 @@ VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS=5400000
 
 `maxConcurrentInlineJobs` (default: `4`) caps how many `executionMode: "inline"` jobs a single `background-jobs-worker` process runs in parallel. Concurrency is at the JS event-loop level: every job in flight shares the worker's process and DB connection pool, so the cap should fit the pool, not the CPU count. Forking remains the right tool when you need memory isolation across long-running jobs or want to use more cores; select it with `executionMode: "forked"`.
 
-New jobs default to `executionMode: "pooled"`: a worker runs them in warm, reusable Node child runners. `pooledRunnerCount` (default: `4`) bounds this independent per-worker pool, and `pooledRunnerConcurrency` (default: `1`) sets how many jobs each child runs at once on its own event loop, so total pooled capacity is `pooledRunnerCount × pooledRunnerConcurrency` — raise concurrency for I/O-bound jobs to get high throughput from a bounded, isolated set of processes. Workers advertise that exact free-slot count and the main consumes one slot per durable handoff, allowing one readiness notification to fill the pool. `pooledRunnerCount`, `pooledRunnerConcurrency`, and `pooledRunnerMaxJobs` must be finite positive integers; the RSS and lifetime limits must be finite positive numbers. A child is recycled after an acknowledged job when it reaches `pooledRunnerMaxJobs` (default: `100`), `pooledRunnerMaxRssBytes` (default: `536870912`, or 512 MiB), or `pooledRunnerMaxLifetimeMs` (default: `3600000`, or one hour). `execution_mode` is the single source of truth for a job's runtime — pooled rows persist as `execution_mode = "pooled"` directly. See [execution modes and pooled runners](docs/background-jobs.md#execution-modes-and-pooled-runners).
+New jobs default to `executionMode: "pooled"`: a worker runs them in warm, reusable Node child runners. `pooledRunnerCount` (default: `4`) bounds this independent per-worker pool, and `pooledRunnerConcurrency` (default: `1`) sets how many jobs each child runs at once on its own event loop, so total pooled capacity is `pooledRunnerCount × pooledRunnerConcurrency` — raise concurrency for I/O-bound jobs to get high throughput from a bounded, isolated set of processes. Workers advertise that exact free-slot count and the main consumes one slot per durable handoff, allowing one readiness notification to fill the pool. If an initialized child exits unexpectedly, the worker immediately advertises the freed capacity while failure reports retry; the replacement is spawned lazily by the next dispatch, and a pre-startup crash does not trigger a respawn loop. `pooledRunnerCount`, `pooledRunnerConcurrency`, and `pooledRunnerMaxJobs` must be finite positive integers; the RSS and lifetime limits must be finite positive numbers. A child is recycled after an acknowledged job when it reaches `pooledRunnerMaxJobs` (default: `100`), `pooledRunnerMaxRssBytes` (default: `536870912`, or 512 MiB), or `pooledRunnerMaxLifetimeMs` (default: `3600000`, or one hour). `execution_mode` is the single source of truth for a job's runtime — pooled rows persist as `execution_mode = "pooled"` directly. See [execution modes and pooled runners](docs/background-jobs.md#execution-modes-and-pooled-runners).
 
 Cold pooled jobs share one atomic model-bootstrap phase. If that phase fails, all
 waiting jobs receive the failure; a later job in the surviving child cannot run
@@ -2595,7 +2595,7 @@ Each job must define exactly one of `every` or `cron`. Cron times are evaluated
 
 ## Persistence and retries
 
-Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the job is marked as handed off with a unique lease id and the worker reports completion or failure back to the main process. If that worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. When the **main** itself restarts (every deploy), a reconnecting worker's still-active handoffs are adopted into its new socket on `hello` so a later disconnect releases them instead of leaving them stuck until the orphan sweep; the main never time-reclaims a disconnected worker's jobs, so a gracefully-draining old-release worker is not double-run. For rolling upgrades, upgrade workers before the main process: a lease-aware main keeps legacy workers connected for old reports but dispatches new jobs only to workers advertising lease-reporting support. See [docs/background-jobs.md](docs/background-jobs.md#worker-disconnect-recovery).
+Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the main generates a unique lease id before asking the adapter to mark the job handed off, and the worker reports completion or failure back to the main process. If that persistence call has an ambiguous result, only the exact caller-generated lease is conditionally returned; failed recovery is retained for the dispatch error-retry path, so worker admission and concurrency do not remain stranded and a newer lease is never reclaimed. Custom adapters must persist a supplied `markHandedOff({handoffId})` exactly; built-in adapters continue generating one for legacy direct callers that omit it. If a worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. When the **main** itself restarts (every deploy), a reconnecting worker's still-active handoffs are adopted into its new socket on `hello` so a later disconnect releases them instead of leaving them stuck until the orphan sweep; the main never time-reclaims a disconnected worker's jobs, so a gracefully-draining old-release worker is not double-run. For rolling upgrades, upgrade workers before the main process: a lease-aware main keeps legacy workers connected for old reports but dispatches new jobs only to workers advertising lease-reporting support. See [docs/background-jobs.md](docs/background-jobs.md#worker-disconnect-recovery).
 
 Failed jobs are re-queued with backoff and retried up to 10 times by default (10s, 1m, 10m, 1h, then +1h per retry). You can override the retry limit per job:
 

package/build/background-jobs/adapter.js

@@ -83,7 +83,9 @@ export default class BackgroundJobsAdapter {
 
   /**
    * Starts a job by claiming its durable handoff.
-   * @param {{jobId: string, workerId?: string}} _args - Handoff request.
+   * When `handoffId` is supplied, the adapter must persist and return that exact
+   * id so the caller can fence an ambiguous commit acknowledgement.
+   * @param {import("./types.js").BackgroundJobHandoffRequest} _args - Handoff request.
    * @returns {Promise<import("./types.js").BackgroundJobHandoff | null>} - Claimed handoff.
    */
   async markHandedOff(_args) { throw new Error("BackgroundJobsAdapter#markHandedOff is not implemented") }

package/build/background-jobs/json-socket.js

@@ -32,6 +32,10 @@ export default class JsonSocket extends EventEmitter {
     this.availablePooledSlots = 0
     /** Whether the worker/main pair uses consumable pooled-capacity credits. */
     this.usesPooledCapacityCredits = false
+    /** Whether this worker has permanently stopped accepting new handoffs. */
+    this.isDraining = false
+    /** Monotonic generation of the worker's latest readiness advertisement. */
+    this.readinessVersion = 0
     /**
      * Narrows the runtime value to the documented type.
      * @type {boolean} */

package/build/background-jobs/local-adapter.js

@@ -107,7 +107,7 @@ export default class LocalBackgroundJobsAdapter extends BackgroundJobsAdapter {
 
   /**
    * Claims one queued local job.
-   * @param {{jobId: string, workerId?: string}} args - Claim request.
+   * @param {import("./types.js").BackgroundJobHandoffRequest} args - Claim request. A supplied handoff id is persisted exactly.
    * @returns {Promise<import("./types.js").BackgroundJobHandoff | null>} - Handoff.
    */
   async markHandedOff(args) { return await this.store.markHandedOff(args) }

package/build/background-jobs/local-store.js

@@ -693,10 +693,10 @@ export default class LocalBackgroundJobsStore {
 
   /**
    * Atomically reserves concurrency and claims one queued job.
-   * @param {{jobId: string, workerId?: string}} args - Claim request.
+   * @param {import("./types.js").BackgroundJobHandoffRequest} args - Claim request. A supplied handoff id is persisted exactly.
    * @returns {Promise<import("./types.js").BackgroundJobHandoff | null>} - Fenced claim.
    */
-  async markHandedOff({jobId, workerId}) {
+  async markHandedOff({jobId, handoffId = new UUID(4).format(), workerId}) {
     await this.ensureReady()
 
     return await this._withDb(async (connection) => await this._mutate(connection, async (db) => {
@@ -706,7 +706,6 @@ export default class LocalBackgroundJobsStore {
       if (job.concurrencyKey && !(await this._reserveConcurrency(db, job.concurrencyKey))) return null
 
       const handedOffAtMs = this.clock.now()
-      const handoffId = new UUID(4).format()
       const affectedRows = await this._updateAffectedRows(db, {
         conditions: {id: jobId, status: "queued"},
         data: {handed_off_at_ms: handedOffAtMs, handoff_id: handoffId, status: "handed_off", worker_id: workerId || "local"},

package/build/background-jobs/main.js

@@ -1,5 +1,6 @@
 // @ts-check
 
+import { randomUUID } from "crypto"
 import net from "net"
 import JsonSocket from "./json-socket.js"
 import BackgroundJobsScheduler from "./scheduler.js"
@@ -92,6 +93,12 @@ export default class BackgroundJobsMain {
      * Active durable handoffs keyed by the exact worker socket that received them.
      * @type {Map<JsonSocket, Map<string, string>>} */
     this.workerHandoffs = new Map()
+    /**
+     * Exact caller-generated leases whose claim outcome was ambiguous or whose
+     * pre-dispatch release has not yet been acknowledged. Retained until a
+     * fenced return succeeds (including an exact no-op).
+     * @type {Map<string, string>} */
+    this.pendingHandoffRecoveries = new Map()
     /**
      * Handoff-adoption queries started by worker hello messages. Shutdown must
      * wait for these before closing the configuration's database pools.
@@ -667,6 +674,7 @@ export default class BackgroundJobsMain {
    * @returns {void}
    */
   _handleWorkerReady({jsonSocket, message}) {
+    jsonSocket.readinessVersion += 1
     jsonSocket.acceptsSpawnedJobs = message.acceptsSpawned !== false && message.acceptsForked !== false
     jsonSocket.acceptsForkedJobs = message.acceptsForked !== false
     jsonSocket.acceptsPooledJobs = message.acceptsPooled === true
@@ -676,7 +684,7 @@ export default class BackgroundJobsMain {
       ? availablePooledSlots
       : 0
     jsonSocket.acceptsInlineJobs = message.acceptsInline !== false
-    if (jsonSocket.supportsHandoffIdReporting) {
+    if (jsonSocket.supportsHandoffIdReporting && !jsonSocket.isDraining) {
       this.readyWorkers.add(jsonSocket)
     } else {
       this.readyWorkers.delete(jsonSocket)
@@ -694,6 +702,7 @@ export default class BackgroundJobsMain {
     // The worker is shutting down gracefully. Stop dispatching new jobs
     // to it but keep the connection in `workers` so any in-flight job
     // it's still draining can report its result.
+    jsonSocket.isDraining = true
     this.readyWorkers.delete(jsonSocket)
   }
 
@@ -1230,6 +1239,8 @@ export default class BackgroundJobsMain {
    * Runs clear error retry timer.
    * @returns {void} */
   _clearErrorRetryTimer() {
+    if (this.pendingHandoffRecoveries.size > 0) return
+
     for (const worker of this.workerHandoffs.keys()) {
       if (!this.workers.has(worker)) return
     }
@@ -1296,12 +1307,20 @@ export default class BackgroundJobsMain {
   }
 
   /**
-   * Retries failed disconnected-socket releases before draining queued work.
+   * Retries failed pre-dispatch and disconnected-socket releases before
+   * draining queued work.
    * @returns {Promise<void>} - Resolves after retry work.
    */
   async _retryAfterError() {
     if (this._stopped) return
 
+    try {
+      await this._retryPendingHandoffRecoveries()
+    } catch {
+      this._scheduleErrorRetry()
+      return
+    }
+
     try {
       for (const worker of this.workerHandoffs.keys()) {
         if (!this.workers.has(worker)) await this._releaseWorkerHandoffs(worker)
@@ -1328,30 +1347,46 @@ export default class BackgroundJobsMain {
       const worker = this.readyWorkerForJob(job)
       if (!worker) return
 
-      this.readyWorkers.delete(worker)
+      const admission = this._consumeWorkerAdmission({job, worker})
+      const requestedHandoffId = randomUUID()
+      let handoff
 
-      if (job.executionMode === "pooled" && worker.usesPooledCapacityCredits && worker.availablePooledSlots > 0) {
-        worker.availablePooledSlots -= 1
-        if (worker.availablePooledSlots > 0) this.readyWorkers.add(worker)
-      }
+      try {
+        handoff = await this.store.markHandedOff({handoffId: requestedHandoffId, jobId: job.id, workerId: worker.workerId})
+      } catch (error) {
+        this._rememberHandoffRecovery({handoffId: requestedHandoffId, jobId: job.id})
+        this._restoreWorkerAdmission({...admission, worker})
 
-      const handoff = await this.store.markHandedOff({jobId: job.id, workerId: worker.workerId})
+        try {
+          await this._recoverHandoff({handoffId: requestedHandoffId, jobId: job.id})
+        } catch (recoveryError) {
+          this._reportHandoffRecoveryError({error: recoveryError, handoffId: requestedHandoffId, jobId: job.id})
+        }
+
+        throw error
+      }
 
       if (!handoff) {
-        if (job.executionMode === "pooled" && worker.usesPooledCapacityCredits) worker.availablePooledSlots += 1
-        if (this.workers.has(worker)) this.readyWorkers.add(worker)
+        this._restoreWorkerAdmission({...admission, worker})
         continue
       }
 
       const handoffs = this.workerHandoffs.get(worker)
 
-      if (!handoffs || !this.workers.has(worker)) {
-        await this.store.markReturnedToQueue({handoffId: handoff.handoffId, jobId: job.id})
+      if (!handoffs || !this.workers.has(worker) || worker.isDraining) {
+        this._rememberHandoffRecovery({handoffId: handoff.handoffId, jobId: job.id})
+        try {
+          await this._recoverHandoff({handoffId: handoff.handoffId, jobId: job.id})
+        } catch (recoveryError) {
+          this._reportHandoffRecoveryError({error: recoveryError, handoffId: handoff.handoffId, jobId: job.id})
+          throw recoveryError
+        }
         this._notifyEnqueued()
         this._redrainQueued = true
         continue
       }
 
+      this._finalizeWorkerAdmission({...admission, job, worker})
       handoffs.set(job.id, handoff.handoffId)
 
       try {
@@ -1382,6 +1417,125 @@ export default class BackgroundJobsMain {
     }
   }
 
+  /**
+   * Consumes one advertised worker admission while persistence is in flight.
+   * @param {object} args - Admission details.
+   * @param {import("./types.js").BackgroundJobRow} args.job - Selected job.
+   * @param {JsonSocket} args.worker - Selected worker socket.
+   * @returns {{pooledCreditConsumed: boolean, readinessVersion: number}} - Reversible admission debit.
+   */
+  _consumeWorkerAdmission({job, worker}) {
+    let pooledCreditConsumed = false
+
+    this.readyWorkers.delete(worker)
+
+    if (job.executionMode === "pooled" && worker.usesPooledCapacityCredits && worker.availablePooledSlots > 0) {
+      pooledCreditConsumed = true
+      worker.availablePooledSlots -= 1
+      if (worker.availablePooledSlots > 0) this.readyWorkers.add(worker)
+    }
+
+    return {pooledCreditConsumed, readinessVersion: worker.readinessVersion}
+  }
+
+  /**
+   * Restores an admission that never reached a worker. A newer readiness
+   * advertisement is already authoritative, so its pooled count is not changed.
+   * @param {object} args - Admission details.
+   * @param {boolean} args.pooledCreditConsumed - Whether a pooled credit was debited.
+   * @param {number} args.readinessVersion - Readiness generation at debit time.
+   * @param {JsonSocket} args.worker - Selected worker socket.
+   * @returns {void}
+   */
+  _restoreWorkerAdmission({pooledCreditConsumed, readinessVersion, worker}) {
+    if (this._stopped || !this.workers.has(worker) || worker.isDraining) return
+
+    if (pooledCreditConsumed && worker.readinessVersion === readinessVersion) {
+      worker.availablePooledSlots += 1
+    }
+
+    if (worker.supportsHandoffIdReporting) this.readyWorkers.add(worker)
+  }
+
+  /**
+   * Applies a successful pooled admission to a readiness advertisement that
+   * arrived while persistence was in flight and replaced the earlier debit.
+   * @param {object} args - Admission details.
+   * @param {import("./types.js").BackgroundJobRow} args.job - Selected job.
+   * @param {boolean} args.pooledCreditConsumed - Whether a pooled credit was debited.
+   * @param {number} args.readinessVersion - Readiness generation at debit time.
+   * @param {JsonSocket} args.worker - Selected worker socket.
+   * @returns {void}
+   */
+  _finalizeWorkerAdmission({job, pooledCreditConsumed, readinessVersion, worker}) {
+    if (!pooledCreditConsumed || job.executionMode !== "pooled") return
+    if (worker.readinessVersion === readinessVersion || !worker.usesPooledCapacityCredits) return
+    if (worker.availablePooledSlots <= 0) return
+
+    worker.availablePooledSlots -= 1
+    if (worker.availablePooledSlots === 0) this.readyWorkers.delete(worker)
+  }
+
+  /**
+   * Retains an exact lease for idempotent pre-dispatch recovery.
+   * @param {{handoffId: string, jobId: string}} args - Exact recovery fence.
+   * @returns {void}
+   */
+  _rememberHandoffRecovery({handoffId, jobId}) {
+    this.pendingHandoffRecoveries.set(handoffId, jobId)
+  }
+
+  /**
+   * Returns one exact lease and forgets it only after the adapter acknowledges
+   * the fenced transition or confirms it was already absent.
+   * @param {{handoffId: string, jobId: string}} args - Exact recovery fence.
+   * @returns {Promise<void>} - Resolves after durable recovery settles.
+   */
+  async _recoverHandoff({handoffId, jobId}) {
+    await this.store.markReturnedToQueue({handoffId, jobId})
+
+    if (this.pendingHandoffRecoveries.get(handoffId) === jobId) {
+      this.pendingHandoffRecoveries.delete(handoffId)
+    }
+  }
+
+  /**
+   * Replays retained exact-ID recoveries through the dispatcher's existing
+   * transient-error retry lifecycle.
+   * @returns {Promise<void>} - Resolves after every retained recovery settles.
+   */
+  async _retryPendingHandoffRecoveries() {
+    for (const [handoffId, jobId] of [...this.pendingHandoffRecoveries]) {
+      try {
+        await this._recoverHandoff({handoffId, jobId})
+      } catch (error) {
+        this._reportHandoffRecoveryError({error, handoffId, jobId})
+        throw error
+      }
+    }
+  }
+
+  /**
+   * Surfaces a failed exact-ID recovery without dropping its retry ledger entry.
+   * @param {object} args - Recovery failure.
+   * @param {ReturnType<typeof JSON.parse>} args.error - Adapter failure.
+   * @param {string} args.handoffId - Exact lease fence.
+   * @param {string} args.jobId - Job id.
+   * @returns {void}
+   */
+  _reportHandoffRecoveryError({error, handoffId, jobId}) {
+    const normalizedError = error instanceof Error ? error : new Error(String(error))
+    const payload = {
+      context: {handoffId, jobId, stage: "background-job-handoff-admission-recovery"},
+      error: normalizedError
+    }
+    const errorEvents = this.configuration.getErrorEvents()
+
+    this.logger.error(() => ["Failed to recover an ambiguous background job handoff:", normalizedError])
+    errorEvents.emit("framework-error", payload)
+    errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
+  }
+
   /**
    * Runs next available job for ready workers.
    * @returns {Promise<import("./types.js").BackgroundJobRow | null>} - Next queued job matching ready worker capacity.

package/build/background-jobs/pooled-runner-child.js

@@ -141,3 +141,4 @@ function handleMessage(message) {
 process.on("message", (message) => handleMessage(message))
 process.once("disconnect", () => void shutdownRunner(0))
 for (const signal of ["SIGTERM", "SIGINT"]) process.once(signal, () => void shutdownRunner(1))
+if (process.send) process.send({type: "ready"})

package/build/background-jobs/store.js

@@ -916,10 +916,11 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
    * Runs mark handed off.
    * @param {object} args - Options.
    * @param {string} args.jobId - Job id.
+   * @param {string} [args.handoffId] - Caller-selected exact lease id. Generated for legacy direct callers when omitted.
    * @param {string} [args.workerId] - Worker id.
    * @returns {Promise<import("./types.js").BackgroundJobHandoff | null>} - Claimed handoff lease, or null when no longer queued.
    */
-  async markHandedOff({jobId, workerId}) {
+  async markHandedOff({jobId, handoffId = randomUUID(), workerId}) {
     await this.ensureReady()
 
     const handedOffAtMs = Date.now()
@@ -928,8 +929,6 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
       const queuedJob = await this._getJobRowById(db, jobId)
       if (!queuedJob || queuedJob.status !== "queued") return null
       if (queuedJob.concurrencyKey && !(await this._reserveConcurrency(db, queuedJob.concurrencyKey))) return null
-      const handoffId = randomUUID()
-
       const affectedRows = await this._updateAffectedRows(db, {
         tableName: JOBS_TABLE,
         data: {

package/build/background-jobs/types.js

@@ -43,6 +43,12 @@
  * @property {string} handoffId - Unique handoff lease id.
  * @property {number} handedOffAtMs - Time handed to a worker in ms.
  */
+/**
+ * @typedef {object} BackgroundJobHandoffRequest
+ * @property {string} jobId - Job to claim.
+ * @property {string} [handoffId] - Exact caller-selected lease id. Adapters must persist and return this id when supplied; built-in adapters generate one when omitted for legacy direct callers.
+ * @property {string} [workerId] - Worker claiming the job.
+ */
 /**
  * @typedef {object} BackgroundJobOptions
  * @property {BackgroundJobExecutionMode} [executionMode] - How the job should run. Node defaults to `"pooled"` (a warm, reused local runner process). Browser/Expo local dispatch defaults to and only accepts `"inline"`. `"forked"` runs a Node job in a fresh `child_process.fork()` child, and `"spawned"` in a detached CLI runner.

package/build/background-jobs/worker.js

@@ -194,8 +194,10 @@ export default class BackgroundJobsWorker {
     this.inflightPooledJobs = new Set()
     /** @type {Set<import("node:child_process").ChildProcess>} */
     this.pooledChildren = new Set()
-    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, inflight: Map<string, {payload: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void, timeoutTimer?: ReturnType<typeof setTimeout> | null}>, lastDispatchSeq: number, retiring: boolean, settling?: boolean, timeoutSigkillTimer?: ReturnType<typeof setTimeout> | null}>} */
+    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, inflight: Map<string, {payload: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void, pooledJob?: Promise<void>, timeoutTimer?: ReturnType<typeof setTimeout> | null}>, lastDispatchSeq: number, retiring: boolean, started?: boolean, settling?: boolean, timeoutSigkillTimer?: ReturnType<typeof setTimeout> | null}>} */
     this.pooledChildStates = new Map()
+    /** @type {WeakSet<Promise<void>>} */
+    this._pooledStartupFailureJobs = new WeakSet()
     // Monotonic dispatch counter for round-robin child selection: each dispatch stamps
     // the chosen child, and selection prefers the child dispatched least recently.
     this._pooledDispatchSeq = 0
@@ -600,16 +602,16 @@ export default class BackgroundJobsWorker {
   }
 
   /**
-   * Tells main we're ready for the next job — but only if we haven't been
-   * asked to drain. Once we've sent `draining` we don't want to take more
-   * work.
+   * Advertises current worker capacity unless the worker is draining.
+   * @param {object} [options] - Advertisement options.
+   * @param {boolean} [options.revokePooledAdmission] - Revoke pooled credits while preserving other execution modes.
    * @returns {void}
    */
-  _sendReadyIfRunning() {
+  _sendReadyIfRunning({revokePooledAdmission = false} = {}) {
     if (this.shouldStop) return
     if (!this.jsonSocket) return
 
-    const readyMessage = this._readyMessage()
+    const readyMessage = this._readyMessage({revokePooledAdmission})
 
     if (!readyMessage) return
     this.jsonSocket.send(readyMessage)
@@ -617,21 +619,24 @@ export default class BackgroundJobsWorker {
 
   /**
    * Runs ready message.
+   * @param {object} [options] - Advertisement options.
+   * @param {boolean} [options.revokePooledAdmission] - Revoke pooled credits while preserving other execution modes.
    * @returns {import("./types.js").BackgroundJobSocketMessage | null} - Ready message or null when the worker has no capacity.
    */
-  _readyMessage() {
+  _readyMessage({revokePooledAdmission = false} = {}) {
     const acceptsProcessJob = this.inflightProcessJobs.size < this.maxConcurrentForkedJobs
     const acceptsInline = this.inflightInlineJobs.size < this.maxConcurrentInlineJobs
-    const acceptsPooled = this._availablePooledSlots() > 0
+    const availablePooledSlots = revokePooledAdmission ? 0 : this._availablePooledSlots()
+    const acceptsPooled = availablePooledSlots > 0
 
-    if (!acceptsProcessJob && !acceptsInline && !acceptsPooled) return null
+    if (!revokePooledAdmission && !acceptsProcessJob && !acceptsInline && !acceptsPooled) return null
 
     return {
       type: "ready",
       acceptsForked: acceptsProcessJob,
       acceptsInline,
       acceptsPooled,
-      availablePooledSlots: this._availablePooledSlots(),
+      availablePooledSlots,
       acceptsSpawned: acceptsProcessJob
     }
   }
@@ -646,7 +651,7 @@ export default class BackgroundJobsWorker {
     let inflight
     inflight = pooledJob.finally(() => {
       this.inflightPooledJobs.delete(inflight)
-      if (!this.shouldStop) this._sendReadyIfRunning()
+      if (!this.shouldStop && !this._pooledStartupFailureJobs.has(pooledJob)) this._sendReadyIfRunning()
     })
     this.inflightPooledJobs.add(inflight)
   }
@@ -689,16 +694,22 @@ export default class BackgroundJobsWorker {
     // Stamp the round-robin cursor so the next dispatch prefers a different child.
     state.lastDispatchSeq = ++this._pooledDispatchSeq
 
-    return new Promise((resolve) => {
-      const timeoutTimer = this._armPooledJobTimeout({child, payload})
+    /**
+     * Resolves the pooled job promise.
+     * @type {(value: void) => void}
+     */
+    let resolvePooledJob = () => {}
+    const pooledJob = new Promise((resolve) => { resolvePooledJob = resolve })
+    const timeoutTimer = this._armPooledJobTimeout({child, payload})
 
-      state.inflight.set(payload.id, {payload, resolve, timeoutTimer})
-      try {
-        child.send({type: "job", payload, sharedTransactionBroker: this._pooledJobSharedTransactionBrokerConfig()})
-      } catch (error) {
-        void this._handlePooledChildFailure({child, error})
-      }
-    })
+    state.inflight.set(payload.id, {payload, resolve: resolvePooledJob, pooledJob, timeoutTimer})
+    try {
+      child.send({type: "job", payload, sharedTransactionBroker: this._pooledJobSharedTransactionBrokerConfig()})
+    } catch (error) {
+      void this._handlePooledChildFailure({child, error})
+    }
+
+    return pooledJob
   }
 
   /**
@@ -769,7 +780,8 @@ export default class BackgroundJobsWorker {
    * (SIGTERM, then SIGKILL after the grace) — a hung JS job cannot be cancelled
    * any other way. The non-clean exit flows through `_handlePooledChildFailure`,
    * which reports every in-flight job on the child failed (so they requeue) and
-   * drops it from tracking; capacity is refilled on the next dispatch.
+   * drops it from tracking; the failure path immediately re-advertises the
+   * resulting capacity once the runner has completed startup.
    * @param {object} args - Options.
    * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
    * @param {string} args.jobId - Job id that overran.
@@ -810,7 +822,7 @@ export default class BackgroundJobsWorker {
     })
     this.pooledChildren.add(child)
     this.inflightProcessChildren.add(child)
-    this.pooledChildStates.set(child, {createdAtMs: Date.now(), jobsRun: 0, inflight: new Map(), lastDispatchSeq: 0, retiring: false})
+    this.pooledChildStates.set(child, {createdAtMs: Date.now(), jobsRun: 0, inflight: new Map(), lastDispatchSeq: 0, retiring: false, started: false})
     child.on("message", (message) => this._handlePooledChildMessage({child, message}))
     child.once("exit", (code, signal) => this._handlePooledChildFailure({child, error: new Error(`Pooled background job runner exited: code=${code} signal=${signal || "none"}`)}))
     child.once("error", (error) => this._handlePooledChildFailure({child, error}))
@@ -829,7 +841,12 @@ export default class BackgroundJobsWorker {
     if (!message || typeof message !== "object") return
     const record = /** @type {{type?: ReturnType<typeof JSON.parse>, jobId?: ReturnType<typeof JSON.parse>, acknowledged?: ReturnType<typeof JSON.parse>, rssBytes?: ReturnType<typeof JSON.parse>, error?: ReturnType<typeof JSON.parse>}} */ (message)
     const state = this.pooledChildStates.get(child)
+    if (record.type === "ready") {
+      if (state) state.started = true
+      return
+    }
     if (record.type !== "job-outcome" || !state || state.settling || typeof record.jobId !== "string") return
+    state.started = true
     const entry = state.inflight.get(record.jobId)
     if (!entry) return
 
@@ -907,9 +924,11 @@ export default class BackgroundJobsWorker {
   /**
    * Removes an exited/unhealthy pooled child and reports every job that was
    * in-flight on it as failed — a process-level crash's blast radius is the
-   * child's whole in-flight set. Capacity is refilled lazily on the next
-   * dispatch (a spawnable slot is still advertised), avoiding a tight respawn
-   * loop when a child crashes on startup.
+   * child's whole in-flight set. Once the child has completed startup, its
+   * freed capacity is advertised immediately; the replacement itself is still
+   * spawned lazily by the next dispatch. A child that exits before its startup
+   * handshake does not re-announce, avoiding a tight respawn loop on startup
+   * failure.
    * @param {object} args - Failure details.
    * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
    * @param {ReturnType<typeof JSON.parse>} args.error - Failure.
@@ -934,7 +953,7 @@ export default class BackgroundJobsWorker {
     if (state) state.inflight.clear()
     this.pooledChildStates.delete(child)
 
-    await Promise.allSettled(entries.map(async (entry) => {
+    const failureReports = entries.map(async (entry) => {
       await this._reportJobResult({
         jobId: entry.payload.id,
         status: "failed",
@@ -944,7 +963,26 @@ export default class BackgroundJobsWorker {
         workerId: entry.payload.workerId || this.workerId
       })
       if (entry.resolve) entry.resolve(undefined)
-    }))
+    })
+
+    // Start every fallback report before announcing capacity so the main cannot
+    // observe a replacement slot before the failed jobs' reports are in flight.
+    // The report promises remain tracked below; a slow retry must not hold the
+    // newly freed runner capacity hostage.
+    if (state && state.started !== false) {
+      this._sendReadyIfRunning()
+    } else if (state) {
+      for (const entry of entries) {
+        if (entry.pooledJob) this._pooledStartupFailureJobs.add(entry.pooledJob)
+      }
+      // A previous ready message may still have unconsumed pooled credits at the
+      // main. Revoke them authoritatively without suppressing valid inline or
+      // process-runner readiness; otherwise queued jobs can trigger a startup
+      // crash loop using the stale credits.
+      this._sendReadyIfRunning({revokePooledAdmission: true})
+    }
+
+    await Promise.allSettled(failureReports)
   }
 
   /**

package/build/src/background-jobs/adapter.d.ts

@@ -81,13 +81,12 @@ export default class BackgroundJobsAdapter {
     getJob(_jobId: string): Promise<import("./types.js").BackgroundJobRow | null>;
     /**
      * Starts a job by claiming its durable handoff.
-     * @param {{jobId: string, workerId?: string}} _args - Handoff request.
+     * When `handoffId` is supplied, the adapter must persist and return that exact
+     * id so the caller can fence an ambiguous commit acknowledgement.
+     * @param {import("./types.js").BackgroundJobHandoffRequest} _args - Handoff request.
      * @returns {Promise<import("./types.js").BackgroundJobHandoff | null>} - Claimed handoff.
      */
-    markHandedOff(_args: {
-        jobId: string;
-        workerId?: string;
-    }): Promise<import("./types.js").BackgroundJobHandoff | null>;
+    markHandedOff(_args: import("./types.js").BackgroundJobHandoffRequest): Promise<import("./types.js").BackgroundJobHandoff | null>;
     /**
      * Marks a handed-off job successful.
      * @param {{jobId: string, handoffId?: string, workerId?: string, handedOffAtMs?: number}} _args - Completion report.

package/build/src/background-jobs/adapter.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/adapter.js"],"names":[],"mappings":"AAEA;;;;GAIG;AACH,MAAM,CAAC,OAAO,OAAO,qBAAqB;IACxC;;;OAGG;IACG,WAAW,IAFJ,OAAO,CAAC,IAAI,CAAC,CAEqE;IAE/F;;;OAGG;IACG,KAAK,IAFE,OAAO,CAAC,IAAI,CAAC,CAEV;IAEhB;;;OAGG;IACG,MAAM,IAFC,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,CAAC,CAI9D;IAED;;;;;OAKG;IACG,qBAAqB,CAAC,KAAK,EAHtB;QAAC,GAAG,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,6BAA6B,EAAE,OAAO,CAAC,CAAA;KAG7C,GAFpB,OAAO,CAAC,IAAI,CAAC,CAEW;IAErC;;;OAGG;IACG,yBAAyB,IAFlB,OAAO,CAAC,IAAI,CAAC,CAEiG;IAE3H;;;;OAIG;IACG,OAAO,CAAC,KAAK,EAHR;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGzG,GAFN,OAAO,CAAC,MAAM,CAAC,CAEgE;IAE5F;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,EAHjB;QAAC,WAAW,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGrH,GAFf,OAAO,CAAC,OAAO,YAAY,EAAE,8BAA8B,CAAC,CAEqC;IAE9G;;;;OAIG;IACG,eAAe,CAAC,YAAY,EAHvB,MAGuB,GAFrB,OAAO,CAAC,OAAO,YAAY,EAAE,+BAA+B,CAAC,CAEyC;IAEnH;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,aAAa,CAAC,EAAE,OAAO,YAAY,EAAE,0BAA0B,GAAG,OAAO,YAAY,EAAE,0BAA0B,EAAE,CAAA;KAG9F,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEiD;IAEnH;;;OAGG;IACG,gBAAgB,IAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEuC;IAEzG;;;;OAIG;IACG,MAAM,CAAC,MAAM,EAHR,MAGQ,GAFN,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEyB;IAE3F;;;;OAIG;IACG,aAAa,CAAC,KAAK,EAHd;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAA;KAGnB,GAFZ,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,GAAG,IAAI,CAAC,CAEkC;IAExG;;;;OAIG;IACG,aAAa,CAAC,KAAK,EAHd;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG/D,GAFZ,OAAO,CAAC,OAAO,CAAC,CAE2E;IAExG;;;;OAIG;IACG,eAAe,CAAC,KAAK,EAHhB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG9E,GAFd,OAAO,CAAC,OAAO,CAAC,CAE+E;IAE5G;;;;OAIG;IACG,mBAAmB,CAAC,KAAK,EAHpB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAGb,GAFlB,OAAO,CAAC,IAAI,CAAC,CAE0F;IAEpH;;;;OAIG;IACG,sBAAsB,CAAC,KAAK,EAHvB;QAAC,QAAQ,EAAE,MAAM,CAAA;KAGM,GAFrB,OAAO,CAAC,KAAK,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAAC,CAAC,CAAC,CAE2D;IAE1H;;;;OAIG;IACG,UAAU,CAAC,KAAK,EAHX;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAGxG,GAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEgC;IAElG;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,eAAe,CAAC,EAAE,MAAM,CAAA;KAGH,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,EAAE,CAAC,CAEsD;IAEnH;;;;OAIG;IACG,iBAAiB,CAAC,KAAK,AAH1B,CACA,EADQ;QAAC,cAAc,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,WAAW,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAA;KAGzD,GAFrB,OAAO,CAAC,MAAM,CAAC,CAEyF;CACtH"}
+{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/adapter.js"],"names":[],"mappings":"AAEA;;;;GAIG;AACH,MAAM,CAAC,OAAO,OAAO,qBAAqB;IACxC;;;OAGG;IACG,WAAW,IAFJ,OAAO,CAAC,IAAI,CAAC,CAEqE;IAE/F;;;OAGG;IACG,KAAK,IAFE,OAAO,CAAC,IAAI,CAAC,CAEV;IAEhB;;;OAGG;IACG,MAAM,IAFC,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,CAAC,CAI9D;IAED;;;;;OAKG;IACG,qBAAqB,CAAC,KAAK,EAHtB;QAAC,GAAG,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,6BAA6B,EAAE,OAAO,CAAC,CAAA;KAG7C,GAFpB,OAAO,CAAC,IAAI,CAAC,CAEW;IAErC;;;OAGG;IACG,yBAAyB,IAFlB,OAAO,CAAC,IAAI,CAAC,CAEiG;IAE3H;;;;OAIG;IACG,OAAO,CAAC,KAAK,EAHR;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGzG,GAFN,OAAO,CAAC,MAAM,CAAC,CAEgE;IAE5F;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,EAHjB;QAAC,WAAW,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGrH,GAFf,OAAO,CAAC,OAAO,YAAY,EAAE,8BAA8B,CAAC,CAEqC;IAE9G;;;;OAIG;IACG,eAAe,CAAC,YAAY,EAHvB,MAGuB,GAFrB,OAAO,CAAC,OAAO,YAAY,EAAE,+BAA+B,CAAC,CAEyC;IAEnH;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,aAAa,CAAC,EAAE,OAAO,YAAY,EAAE,0BAA0B,GAAG,OAAO,YAAY,EAAE,0BAA0B,EAAE,CAAA;KAG9F,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEiD;IAEnH;;;OAGG;IACG,gBAAgB,IAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEuC;IAEzG;;;;OAIG;IACG,MAAM,CAAC,MAAM,EAHR,MAGQ,GAFN,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEyB;IAE3F;;;;;;OAMG;IACG,aAAa,CAAC,KAAK,EAHd,OAAO,YAAY,EAAE,2BAGP,GAFZ,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,GAAG,IAAI,CAAC,CAEkC;IAExG;;;;OAIG;IACG,aAAa,CAAC,KAAK,EAHd;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG/D,GAFZ,OAAO,CAAC,OAAO,CAAC,CAE2E;IAExG;;;;OAIG;IACG,eAAe,CAAC,KAAK,EAHhB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG9E,GAFd,OAAO,CAAC,OAAO,CAAC,CAE+E;IAE5G;;;;OAIG;IACG,mBAAmB,CAAC,KAAK,EAHpB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAGb,GAFlB,OAAO,CAAC,IAAI,CAAC,CAE0F;IAEpH;;;;OAIG;IACG,sBAAsB,CAAC,KAAK,EAHvB;QAAC,QAAQ,EAAE,MAAM,CAAA;KAGM,GAFrB,OAAO,CAAC,KAAK,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAAC,CAAC,CAAC,CAE2D;IAE1H;;;;OAIG;IACG,UAAU,CAAC,KAAK,EAHX;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAGxG,GAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEgC;IAElG;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,eAAe,CAAC,EAAE,MAAM,CAAA;KAGH,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,EAAE,CAAC,CAEsD;IAEnH;;;;OAIG;IACG,iBAAiB,CAAC,KAAK,AAH1B,CACA,EADQ;QAAC,cAAc,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,WAAW,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAA;KAGzD,GAFrB,OAAO,CAAC,MAAM,CAAC,CAEyF;CACtH"}