velocious 1.0.628 → 1.0.629

package/README.md

@@ -2626,7 +2626,7 @@ Each job must define exactly one of `every` or `cron`. Cron times are evaluated
 
 ## Persistence and retries
 
-Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the main generates a unique lease id before asking the adapter to mark the job handed off, and the worker reports completion or failure back to the main process. If that persistence call has an ambiguous result, only the exact caller-generated lease is conditionally returned; failed recovery is retained for the dispatch error-retry path, so worker admission and concurrency do not remain stranded and a newer lease is never reclaimed. Custom adapters must persist a supplied `markHandedOff({handoffId})` exactly; built-in adapters continue generating one for legacy direct callers that omit it. If a worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. When the **main** itself restarts (every deploy), a reconnecting worker's still-active handoffs are adopted into its new socket on `hello` so a later disconnect releases them instead of leaving them stuck until the orphan sweep; the main never time-reclaims a disconnected worker's jobs, so a gracefully-draining old-release worker is not double-run. For rolling upgrades, upgrade workers before the main process: a lease-aware main keeps legacy workers connected for old reports but dispatches new jobs only to workers advertising lease-reporting support. See [docs/background-jobs.md](docs/background-jobs.md#worker-disconnect-recovery).
+Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the main generates a unique lease id before asking the adapter to mark the job handed off, and the worker reports completion or failure back to the main process. If that persistence call has an ambiguous result, only the exact caller-generated lease is conditionally returned; failed recovery is retained for the dispatch error-retry path, so worker admission and concurrency do not remain stranded and a newer lease is never reclaimed. Custom adapters must persist a supplied `markHandedOff({handoffId})` exactly; built-in adapters continue generating one for legacy direct callers that omit it. If a worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. When the **main** itself restarts (every deploy), it snapshots exact pre-start leases and gives surviving workers 30 seconds to reconnect and adopt their active handoffs. At that deadline it waits up to one additional reconnect-grace interval for adoption queries that were already in flight, preventing a normal slow query from racing reclaim without letting a stuck query defer cleanup forever. A worker is excluded from startup reclaim only after its adoption query succeeds while that same socket is still connected; query failures and disconnects during adoption remain reclaimable. The main then orphans only unchanged startup snapshots belonging to worker ids that did not successfully adopt, using the normal retry/count/concurrency/event lifecycle and immediately waking queued work. A successfully adopted worker, a lease created after startup, and any newer re-handoff are fenced out. `BackgroundJobsMain` accepts `workerReconnectGraceMs` for tests and operational tuning; it must be an integer from 0 through 2,147,483,647 ms and invalid values throw. This startup cleanup does not lower the separate two-hour age sweep. For rolling upgrades, upgrade workers before the main process: a lease-aware main keeps legacy workers connected for old reports but dispatches new jobs only to workers advertising lease-reporting support. See [docs/background-jobs.md](docs/background-jobs.md#worker-disconnect-recovery).
 
 Failed jobs are re-queued with backoff and retried up to 10 times by default (10s, 1m, 10m, 1h, then +1h per retry). You can override the retry limit per job:
 

package/build/background-jobs/adapter.js

@@ -118,6 +118,22 @@ export default class BackgroundJobsAdapter {
    */
   async handedOffJobsForWorker(_args) { throw new Error("BackgroundJobsAdapter#handedOffJobsForWorker is not implemented") }
 
+  /**
+   * Snapshots exact active handoffs before a new main generation accepts worker
+   * reconnects. Adapters that do not persist worker leases may return none.
+   * @returns {Promise<import("./types.js").BackgroundJobHandoffSnapshot[]>} - Exact active handoffs.
+   */
+  async snapshotHandedOffJobs() { return [] }
+
+  /**
+   * Applies orphan failure semantics to unchanged exact handoff snapshots.
+   * Adapters that return startup snapshots must implement the matching fenced
+   * transition.
+   * @param {{handoffs: import("./types.js").BackgroundJobHandoffSnapshot[], error: ReturnType<typeof JSON.parse>}} _args - Exact leases and orphan reason.
+   * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+   */
+  async markOrphanedHandoffs(_args) { return [] }
+
   /**
    * Marks a handed-off job failed or retryable.
    * @param {{jobId: string, error: ReturnType<typeof JSON.parse>, handoffId?: string, workerId?: string, handedOffAtMs?: number}} _args - Failure report.

package/build/background-jobs/main.js

@@ -34,6 +34,24 @@ const MAX_TIMER_MS = 2_147_483_647 // ~24.8 days
 const WORKER_STALE_TIMEOUT_MS = 60000
 /** How often the main scans workers for staleness. */
 const WORKER_LIVENESS_SWEEP_MS = 15000
+/** Grace for workers from the previous main generation to reconnect and adopt leases. */
+const WORKER_RECONNECT_GRACE_MS = 30000
+const WORKER_RECONNECT_GRACE_VALIDATION_MESSAGE = `workerReconnectGraceMs must be an integer between 0 and ${MAX_TIMER_MS}`
+
+/**
+ * Resolves a startup reconnect grace without allowing Node's timer overflow to
+ * turn an intentionally long grace into an immediate reclaim.
+ * @param {number | undefined} workerReconnectGraceMs - Requested reconnect grace.
+ * @returns {number} - Valid timer delay.
+ */
+function normalizeWorkerReconnectGraceMs(workerReconnectGraceMs) {
+  if (workerReconnectGraceMs === undefined) return WORKER_RECONNECT_GRACE_MS
+  if (!Number.isInteger(workerReconnectGraceMs) || workerReconnectGraceMs < 0 || workerReconnectGraceMs > MAX_TIMER_MS) {
+    throw new TypeError(WORKER_RECONNECT_GRACE_VALIDATION_MESSAGE)
+  }
+
+  return workerReconnectGraceMs
+}
 /**
  * Worker execution mode capabilities.
  * @type {WorkerExecutionModeCapability[]} */
@@ -61,10 +79,11 @@ export default class BackgroundJobsMain {
    * @param {number} [args.port] - Port.
    * @param {number} [args.workerStaleTimeoutMs] - Override how long a silent worker may go before being dropped (default 60000ms).
    * @param {number} [args.workerLivenessSweepMs] - Override how often stale workers are swept for (default 15000ms).
+   * @param {number} [args.workerReconnectGraceMs] - Integer from 0 through 2,147,483,647 overriding how long previous-generation workers may reconnect before exact startup leases are reclaimed (default 30000ms).
    * @param {boolean} [args.closeDatabaseConnectionsOnStop] - Whether stop owns closing the configuration's database pools (default true).
    * @param {() => void | Promise<void>} [args.onStopped] - Lifecycle hook invoked after the main process finishes stopping.
    */
-  constructor({configuration, host, port, workerStaleTimeoutMs, workerLivenessSweepMs, closeDatabaseConnectionsOnStop = true, onStopped}) {
+  constructor({configuration, host, port, workerStaleTimeoutMs, workerLivenessSweepMs, workerReconnectGraceMs, closeDatabaseConnectionsOnStop = true, onStopped}) {
     this.configuration = configuration
     this.closeDatabaseConnectionsOnStop = closeDatabaseConnectionsOnStop
     this.onStopped = onStopped
@@ -78,6 +97,7 @@ export default class BackgroundJobsMain {
     // long is treated as wedged/dead: its leases are released and it is dropped.
     this.workerStaleTimeoutMs = typeof workerStaleTimeoutMs === "number" && workerStaleTimeoutMs >= 1 ? workerStaleTimeoutMs : WORKER_STALE_TIMEOUT_MS
     this.workerLivenessSweepMs = typeof workerLivenessSweepMs === "number" && workerLivenessSweepMs >= 1 ? workerLivenessSweepMs : WORKER_LIVENESS_SWEEP_MS
+    this.workerReconnectGraceMs = normalizeWorkerReconnectGraceMs(workerReconnectGraceMs)
     /** @type {import("./adapter.js").default | undefined} */
     this.adapter = undefined
     this.logger = new Logger(this)
@@ -104,6 +124,17 @@ export default class BackgroundJobsMain {
      * wait for these before closing the configuration's database pools.
      * @type {Set<Promise<void>>} */
     this.inflightWorkerHandoffAdoptions = new Set()
+    /**
+     * Worker ids whose handoffs were successfully adopted by a still-live
+     * connection in this main generation.
+     * @type {Set<string>}
+     */
+    this.reconnectedWorkerIds = new Set()
+    /** @type {import("./types.js").BackgroundJobHandoffSnapshot[]} */
+    this.startupHandoffSnapshot = []
+    /** @type {Promise<void>[]} */
+    this._startupHandoffAdoptionsAtDeadline = []
+    this._startupHandoffGraceElapsed = false
     /**
      * Narrows the runtime value to the documented type.
      * @type {net.Server | undefined} */
@@ -128,6 +159,10 @@ export default class BackgroundJobsMain {
      * Narrows the runtime value to the documented type.
      * @type {ReturnType<typeof setInterval> | undefined} */
     this._workerStaleTimer = undefined
+    /** @type {ReturnType<typeof setTimeout> | undefined} */
+    this._startupHandoffReclaimTimer = undefined
+    /** @type {Promise<void> | undefined} */
+    this._startupHandoffReclaimPromise = undefined
     /**
      * Narrows the runtime value to the documented type.
      * @type {BackgroundJobsScheduler | undefined} */
@@ -178,6 +213,11 @@ export default class BackgroundJobsMain {
   async start() {
     this._stopped = false
     this.stopPromise = undefined
+    this.reconnectedWorkerIds.clear()
+    this.startupHandoffSnapshot = []
+    this._startupHandoffAdoptionsAtDeadline = []
+    this._startupHandoffGraceElapsed = false
+    this._startupHandoffReclaimPromise = undefined
     this.configuration.setCurrent()
 
     try {
@@ -194,6 +234,7 @@ export default class BackgroundJobsMain {
       // across processes with a database advisory lock, so concurrently started
       // mains cannot interleave them.
       await this.store.reconcileQueueConcurrency()
+      this.startupHandoffSnapshot = await this.store.snapshotHandedOffJobs()
       const server = net.createServer((socket) => this._handleConnection(socket))
       this.server = server
 
@@ -208,6 +249,7 @@ export default class BackgroundJobsMain {
       }
 
       this._setupDispatchTriggers()
+      this._setupStartupHandoffReclaim()
 
       this._orphanTimer = setInterval(() => {
         void this._sweepOrphans()
@@ -297,7 +339,11 @@ export default class BackgroundJobsMain {
             try {
               await this._drainWorkerHandoffAdoptions()
             } finally {
-              await this._stopBeaconAndServer()
+              try {
+                await this._drainStartupHandoffReclaim()
+              } finally {
+                await this._stopBeaconAndServer()
+              }
             }
           }
         }
@@ -325,11 +371,13 @@ export default class BackgroundJobsMain {
     if (this._errorRetryTimer) clearTimeout(this._errorRetryTimer)
     if (this._orphanTimer) clearInterval(this._orphanTimer)
     if (this._workerStaleTimer) clearInterval(this._workerStaleTimer)
+    if (this._startupHandoffReclaimTimer) clearTimeout(this._startupHandoffReclaimTimer)
     this._pollTimer = undefined
     this._scheduledTimer = undefined
     this._errorRetryTimer = undefined
     this._orphanTimer = undefined
     this._workerStaleTimer = undefined
+    this._startupHandoffReclaimTimer = undefined
   }
 
   /**
@@ -431,6 +479,123 @@ export default class BackgroundJobsMain {
     beaconClient.on("connect", this._beaconConnectHandler)
   }
 
+  /**
+   * Arms the bounded adoption grace only when startup found exact persisted
+   * handoffs. The timer is unrefed so an otherwise-finished process is never
+   * retained solely to perform this cleanup.
+   * @returns {void}
+   */
+  _setupStartupHandoffReclaim() {
+    if (this.startupHandoffSnapshot.length === 0) return
+
+    this._startupHandoffReclaimTimer = setTimeout(() => {
+      this._startupHandoffReclaimTimer = undefined
+      this._startupHandoffAdoptionsAtDeadline = [...this.inflightWorkerHandoffAdoptions]
+      this._startupHandoffGraceElapsed = true
+      void this._startStartupHandoffReclaim()
+    }, this.workerReconnectGraceMs)
+    this._startupHandoffReclaimTimer.unref()
+  }
+
+  /**
+   * Starts one tracked startup-reclaim pass, coalescing lifecycle and retry
+   * callers so shutdown can wait for durable mutation before closing pools.
+   * @returns {Promise<void>} - Resolves after this pass settles.
+   */
+  _startStartupHandoffReclaim() {
+    if (this._startupHandoffReclaimPromise) return this._startupHandoffReclaimPromise
+
+    const reclaim = this._reclaimDisconnectedStartupHandoffs()
+
+    this._startupHandoffReclaimPromise = reclaim
+    const clearReclaim = () => {
+      if (this._startupHandoffReclaimPromise === reclaim) {
+        this._startupHandoffReclaimPromise = undefined
+      }
+    }
+    void reclaim.then(clearReclaim, clearReclaim)
+
+    return reclaim
+  }
+
+  /**
+   * Waits for an already-started startup reclaim before adapter shutdown.
+   * @returns {Promise<void>} - Resolves when no pass remains.
+   */
+  async _drainStartupHandoffReclaim() {
+    while (this._startupHandoffReclaimPromise) {
+      await this._startupHandoffReclaimPromise
+    }
+  }
+
+  /**
+   * Orphans only startup-snapshotted leases whose stable worker id has not been
+   * observed by this main generation. Store fencing rejects completed,
+   * returned, replaced, and re-handed-off rows.
+   * @returns {Promise<void>} - Resolves after reclaim or retained retry state.
+   */
+  async _reclaimDisconnectedStartupHandoffs() {
+    if (this._stopped || !this._startupHandoffGraceElapsed) return
+    if (this.startupHandoffSnapshot.length === 0) return
+
+    await this._waitForStartupHandoffAdoptionsAtDeadline()
+    if (this._stopped) return
+
+    const handoffs = this.startupHandoffSnapshot.filter(({workerId}) => !this.reconnectedWorkerIds.has(workerId))
+
+    if (handoffs.length === 0) {
+      this.startupHandoffSnapshot = []
+      return
+    }
+
+    let orphanedJobs
+
+    try {
+      orphanedJobs = await this.store.markOrphanedHandoffs({
+        error: "Job orphaned after its pre-restart worker did not reconnect",
+        handoffs
+      })
+    } catch (error) {
+      this._reportStartupHandoffReclaimError(error)
+      this._scheduleErrorRetry()
+      return
+    }
+
+    this.startupHandoffSnapshot = []
+    await this._handleOrphanedJobs({
+      jobs: orphanedJobs,
+      warning: "Reclaimed background jobs from workers absent after main restart grace"
+    })
+  }
+
+  /**
+   * Lets adoption queries already running at the reconnect deadline settle
+   * before worker ids are filtered. A second bounded grace prevents a stuck
+   * adapter query from deferring startup reclaim forever.
+   * @returns {Promise<void>} - Resolves when the deadline set settles or times out.
+   */
+  async _waitForStartupHandoffAdoptionsAtDeadline() {
+    const adoptions = this._startupHandoffAdoptionsAtDeadline
+
+    this._startupHandoffAdoptionsAtDeadline = []
+    if (adoptions.length === 0) return
+
+    /** @type {ReturnType<typeof setTimeout> | undefined} */
+    let timer
+    const waitLimit = new Promise((resolve) => {
+      // This lifecycle deadline must not keep the main process alive; the
+      // generic timeout helper intentionally uses a referenced timer.
+      timer = setTimeout(resolve, this.workerReconnectGraceMs)
+      timer.unref()
+    })
+
+    try {
+      await Promise.race([Promise.all(adoptions), waitLimit])
+    } finally {
+      if (timer) clearTimeout(timer)
+    }
+  }
+
   /**
    * Publishes a dispatch wake-up on the Beacon channel. No-op in polling
    * mode or when Beacon is not connected; in those cases the direct
@@ -583,6 +748,7 @@ export default class BackgroundJobsMain {
       for (const {jobId, handoffId} of handoffs) {
         map.set(jobId, handoffId)
       }
+      this.reconnectedWorkerIds.add(workerId)
     } catch (error) {
       this._reportHandoffAdoptError(error)
     }
@@ -820,6 +986,22 @@ export default class BackgroundJobsMain {
     errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
   }
 
+  /**
+   * Reports an unexpected startup-snapshot reclaim failure while retaining the
+   * snapshot for the dispatcher's existing transient-error retry lifecycle.
+   * @param {ReturnType<typeof JSON.parse>} error - Reclaim failure.
+   * @returns {void}
+   */
+  _reportStartupHandoffReclaimError(error) {
+    const normalizedError = error instanceof Error ? error : new Error(String(error))
+    const payload = {context: {stage: "background-job-startup-handoff-reclaim"}, error: normalizedError}
+    const errorEvents = this.configuration.getErrorEvents()
+
+    this.logger.error(() => ["Failed to reclaim disconnected startup handoffs:", normalizedError])
+    errorEvents.emit("framework-error", payload)
+    errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
+  }
+
   /**
    * Runs handle enqueue.
    * @param {object} args - Options.
@@ -1237,6 +1419,7 @@ export default class BackgroundJobsMain {
    * @returns {void} */
   _clearErrorRetryTimer() {
     if (this.pendingHandoffRecoveries.size > 0) return
+    if (this._startupHandoffGraceElapsed && this.startupHandoffSnapshot.length > 0) return
 
     for (const worker of this.workerHandoffs.keys()) {
       if (!this.workers.has(worker)) return
@@ -1311,6 +1494,11 @@ export default class BackgroundJobsMain {
   async _retryAfterError() {
     if (this._stopped) return
 
+    if (this._startupHandoffGraceElapsed && this.startupHandoffSnapshot.length > 0) {
+      await this._startStartupHandoffReclaim()
+      if (this.startupHandoffSnapshot.length > 0) return
+    }
+
     try {
       await this._retryPendingHandoffRecoveries()
     } catch {
@@ -1650,31 +1838,40 @@ export default class BackgroundJobsMain {
     try {
       const orphanedJobs = await this.store.markOrphanedJobs()
 
-      if (orphanedJobs.length > 0) {
-        this.logger.warn(() => ["Marked orphaned background jobs", orphanedJobs.length])
-        // Reclaimed orphans become `queued` again — wake the dispatcher first so
-        // an application event handler that throws below cannot strand them
-        // queued until the next external enqueue/reconnect.
-        this._notifyEnqueued()
-        // Emit an event per orphaned job so applications can react to a dead
-        // worker's specific job (e.g. targeted recovery) instead of only polling
-        // for its aftermath. Emit before awaiting the drain so a blocked
-        // dispatcher cannot delay application recovery. Isolate each so one
-        // throwing handler can't suppress the events for the rest.
-        for (const job of orphanedJobs) {
-          try {
-            this._emitBackgroundJobOrphaned({job})
-          } catch (error) {
-            this.logger.error(() => ["A background-job-orphaned event handler threw:", error])
-          }
-        }
-        await this._drain()
-      }
+      await this._handleOrphanedJobs({jobs: orphanedJobs, warning: "Marked orphaned background jobs"})
     } catch (error) {
       this.logger.error(() => ["Failed to mark orphaned jobs:", error])
     }
   }
 
+  /**
+   * Publishes the common post-orphan lifecycle: wake queued retries, emit one
+   * isolated event per accepted transition, and drain so released concurrency
+   * can immediately admit other work.
+   * @param {object} args - Options.
+   * @param {import("./types.js").BackgroundJobRow[]} args.jobs - Accepted orphan transitions.
+   * @param {string} args.warning - Lifecycle log message.
+   * @returns {Promise<void>} - Resolves after the resulting drain.
+   */
+  async _handleOrphanedJobs({jobs, warning}) {
+    if (jobs.length === 0) return
+
+    this.logger.warn(() => [warning, jobs.length])
+    // Reclaimed orphans can become `queued` again — wake the dispatcher first
+    // so an application event handler that throws below cannot strand them.
+    this._notifyEnqueued()
+    // Emit before awaiting the drain so a blocked dispatcher cannot delay
+    // application recovery. Isolate handlers so one cannot suppress the rest.
+    for (const job of jobs) {
+      try {
+        this._emitBackgroundJobOrphaned({job})
+      } catch (error) {
+        this.logger.error(() => ["A background-job-orphaned event handler threw:", error])
+      }
+    }
+    await this._drain()
+  }
+
   /**
    * Drops workers that have gone silent past `workerStaleTimeoutMs` (no
    * heartbeat, ready, or report). A wedged worker keeps its socket open, so the

package/build/background-jobs/store.js

@@ -43,6 +43,13 @@ import {
  * @property {number | null} timeoutMs - Per-job timeout override, or null when omitted.
  */
 
+/**
+ * BackgroundJobOrphanSelection type.
+ * @typedef {object} BackgroundJobOrphanSelection
+ * @property {Record<string, ReturnType<typeof JSON.parse>>} conditions - Exact update fence.
+ * @property {import("./types.js").BackgroundJobRow} job - Selected active handoff.
+ */
+
 /**
  * BackgroundJobTransactionSerializationOptions type.
  * @typedef {object} BackgroundJobTransactionSerializationOptions
@@ -1096,6 +1103,81 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
     return handoffs
   }
 
+  /**
+   * Snapshots exact, lease-aware active handoffs before a new main generation
+   * starts accepting worker reconnects. Legacy rows without a complete worker,
+   * lease, and timestamp identity stay owned by the age-based orphan sweep.
+   * @returns {Promise<import("./types.js").BackgroundJobHandoffSnapshot[]>} - Exact startup handoffs.
+   */
+  async snapshotHandedOffJobs() {
+    await this.ensureReady()
+
+    const rows = await this._withDb(async (db) => await db
+      .newQuery()
+      .from(JOBS_TABLE)
+      .where({status: "handed_off"})
+      .order("created_at_ms ASC")
+      .order("id ASC")
+      .results())
+    /** @type {import("./types.js").BackgroundJobHandoffSnapshot[]} */
+    const handoffs = []
+
+    for (const row of rows) {
+      const job = this._normalizeJobRow(row)
+
+      if (!job.handoffId || !job.workerId || typeof job.handedOffAtMs !== "number") continue
+
+      handoffs.push({
+        handedOffAtMs: job.handedOffAtMs,
+        handoffId: job.handoffId,
+        jobId: job.id,
+        workerId: job.workerId
+      })
+    }
+
+    return handoffs
+  }
+
+  /**
+   * Reclaims only unchanged exact handoffs selected by a main-generation startup
+   * snapshot. The ordinary orphan failure path owns retries, terminal status,
+   * count transitions, schedule ownership, and concurrency release.
+   * @param {object} args - Options.
+   * @param {import("./types.js").BackgroundJobHandoffSnapshot[]} args.handoffs - Exact startup snapshots.
+   * @param {ReturnType<typeof JSON.parse>} args.error - Orphan reason.
+   * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+   */
+  async markOrphanedHandoffs({handoffs, error}) {
+    await this.ensureReady()
+
+    return await this._serializedCountMutation(async (db) => {
+      /** @type {BackgroundJobOrphanSelection[]} */
+      const selections = []
+
+      for (const handoff of handoffs) {
+        const job = await this._getJobRowById(db, handoff.jobId)
+
+        if (!job || job.status !== "handed_off") continue
+        if (job.handoffId !== handoff.handoffId) continue
+        if (job.workerId !== handoff.workerId) continue
+        if (job.handedOffAtMs !== handoff.handedOffAtMs) continue
+
+        selections.push({
+          conditions: {
+            handed_off_at_ms: handoff.handedOffAtMs,
+            handoff_id: handoff.handoffId,
+            id: handoff.jobId,
+            status: "handed_off",
+            worker_id: handoff.workerId
+          },
+          job
+        })
+      }
+
+      return await this._markOrphanSelections({db, error, selections})
+    })
+  }
+
   /**
    * Runs mark failed.
    * @param {object} args - Options.
@@ -1141,8 +1223,8 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
 
       const rows = await query.results()
 
-      /** @type {import("./types.js").BackgroundJobRow[]} */
-      const orphanedJobs = []
+      /** @type {BackgroundJobOrphanSelection[]} */
+      const selections = []
 
       for (const row of rows) {
         const job = this._normalizeJobRow(row)
@@ -1160,28 +1242,55 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
         //      wrongly release the concurrency reservation of — that new lease.
         // `handed_off_at_ms` is always set on a handed-off row (and the SELECT
         // required it `<= cutoff`), so it is a reliable null-safe lease pin.
-        const orphanedJob = await this._applyFailure({
-          db,
-          job,
-          error: "Job orphaned after timeout",
-          markOrphaned: true,
-          conditions: {id: job.id, status: "handed_off", handed_off_at_ms: job.handedOffAtMs}
+        selections.push({
+          conditions: {id: job.id, status: "handed_off", handed_off_at_ms: job.handedOffAtMs},
+          job
         })
-
-        if (orphanedJob) orphanedJobs.push(orphanedJob)
       }
 
-      const statusCounts = this._statusCounts(orphanedJobs)
-      const deltas = this._emptyCountBuckets()
+      return await this._markOrphanSelections({
+        db,
+        error: "Job orphaned after timeout",
+        selections
+      })
+    })
+  }
+
+  /**
+   * Applies the common fenced orphan transition and records one aggregate count
+   * delta for the accepted rows.
+   * @param {object} args - Options.
+   * @param {import("../database/drivers/base.js").default} args.db - Transaction connection.
+   * @param {ReturnType<typeof JSON.parse>} args.error - Orphan reason.
+   * @param {BackgroundJobOrphanSelection[]} args.selections - Selected handoffs and exact fences.
+   * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+   */
+  async _markOrphanSelections({db, error, selections}) {
+    /** @type {import("./types.js").BackgroundJobRow[]} */
+    const orphanedJobs = []
+
+    for (const {conditions, job} of selections) {
+      const orphanedJob = await this._applyFailure({
+        conditions,
+        db,
+        error,
+        job,
+        markOrphaned: true
+      })
 
-      for (const [status, count] of Object.entries(statusCounts)) {
-        deltas.handed_off -= count
-        deltas[status] += count
-      }
-      await this._recordCountDelta(db, deltas)
+      if (orphanedJob) orphanedJobs.push(orphanedJob)
+    }
 
-      return orphanedJobs
-    })
+    const statusCounts = this._statusCounts(orphanedJobs)
+    const deltas = this._emptyCountBuckets()
+
+    for (const [status, count] of Object.entries(statusCounts)) {
+      deltas.handed_off -= count
+      deltas[status] += count
+    }
+    await this._recordCountDelta(db, deltas)
+
+    return orphanedJobs
   }
 
   /**

package/build/background-jobs/types.js

@@ -43,6 +43,13 @@
  * @property {string} handoffId - Unique handoff lease id.
  * @property {number} handedOffAtMs - Time handed to a worker in ms.
  */
+/**
+ * @typedef {object} BackgroundJobHandoffSnapshot
+ * @property {string} jobId - Job holding the lease.
+ * @property {string} handoffId - Exact durable lease id.
+ * @property {string} workerId - Stable worker id that received the lease.
+ * @property {number} handedOffAtMs - Time handed to the worker in ms.
+ */
 /**
  * @typedef {object} BackgroundJobHandoffRequest
  * @property {string} jobId - Job to claim.

package/build/src/background-jobs/adapter.d.ts

@@ -130,6 +130,23 @@ export default class BackgroundJobsAdapter {
         jobId: string;
         handoffId: string;
     }>>;
+    /**
+     * Snapshots exact active handoffs before a new main generation accepts worker
+     * reconnects. Adapters that do not persist worker leases may return none.
+     * @returns {Promise<import("./types.js").BackgroundJobHandoffSnapshot[]>} - Exact active handoffs.
+     */
+    snapshotHandedOffJobs(): Promise<import("./types.js").BackgroundJobHandoffSnapshot[]>;
+    /**
+     * Applies orphan failure semantics to unchanged exact handoff snapshots.
+     * Adapters that return startup snapshots must implement the matching fenced
+     * transition.
+     * @param {{handoffs: import("./types.js").BackgroundJobHandoffSnapshot[], error: ReturnType<typeof JSON.parse>}} _args - Exact leases and orphan reason.
+     * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+     */
+    markOrphanedHandoffs(_args: {
+        handoffs: import("./types.js").BackgroundJobHandoffSnapshot[];
+        error: ReturnType<typeof JSON.parse>;
+    }): Promise<import("./types.js").BackgroundJobRow[]>;
     /**
      * Marks a handed-off job failed or retryable.
      * @param {{jobId: string, error: ReturnType<typeof JSON.parse>, handoffId?: string, workerId?: string, handedOffAtMs?: number}} _args - Failure report.

package/build/src/background-jobs/adapter.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/adapter.js"],"names":[],"mappings":"AAEA;;;;GAIG;AACH,MAAM,CAAC,OAAO,OAAO,qBAAqB;IACxC;;;OAGG;IACG,WAAW,IAFJ,OAAO,CAAC,IAAI,CAAC,CAEqE;IAE/F;;;OAGG;IACG,KAAK,IAFE,OAAO,CAAC,IAAI,CAAC,CAEV;IAEhB;;;OAGG;IACG,MAAM,IAFC,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,CAAC,CAI9D;IAED;;;;;OAKG;IACG,qBAAqB,CAAC,KAAK,EAHtB;QAAC,GAAG,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,6BAA6B,EAAE,OAAO,CAAC,CAAA;KAG7C,GAFpB,OAAO,CAAC,IAAI,CAAC,CAEW;IAErC;;;OAGG;IACG,yBAAyB,IAFlB,OAAO,CAAC,IAAI,CAAC,CAEiG;IAE3H;;;;OAIG;IACG,OAAO,CAAC,KAAK,EAHR;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGzG,GAFN,OAAO,CAAC,MAAM,CAAC,CAEgE;IAE5F;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,EAHjB;QAAC,WAAW,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGrH,GAFf,OAAO,CAAC,OAAO,YAAY,EAAE,8BAA8B,CAAC,CAEqC;IAE9G;;;;OAIG;IACG,eAAe,CAAC,YAAY,EAHvB,MAGuB,GAFrB,OAAO,CAAC,OAAO,YAAY,EAAE,+BAA+B,CAAC,CAEyC;IAEnH;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,aAAa,CAAC,EAAE,OAAO,YAAY,EAAE,0BAA0B,GAAG,OAAO,YAAY,EAAE,0BAA0B,EAAE,CAAA;KAG9F,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEiD;IAEnH;;;OAGG;IACG,gBAAgB,IAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEuC;IAEzG;;;;OAIG;IACG,MAAM,CAAC,MAAM,EAHR,MAGQ,GAFN,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEyB;IAE3F;;;;;;OAMG;IACG,aAAa,CAAC,KAAK,EAHd,OAAO,YAAY,EAAE,2BAGP,GAFZ,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,GAAG,IAAI,CAAC,CAEkC;IAExG;;;;OAIG;IACG,aAAa,CAAC,KAAK,EAHd;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG/D,GAFZ,OAAO,CAAC,OAAO,CAAC,CAE2E;IAExG;;;;OAIG;IACG,eAAe,CAAC,KAAK,EAHhB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG9E,GAFd,OAAO,CAAC,OAAO,CAAC,CAE+E;IAE5G;;;;OAIG;IACG,mBAAmB,CAAC,KAAK,EAHpB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAGb,GAFlB,OAAO,CAAC,IAAI,CAAC,CAE0F;IAEpH;;;;OAIG;IACG,sBAAsB,CAAC,KAAK,EAHvB;QAAC,QAAQ,EAAE,MAAM,CAAA;KAGM,GAFrB,OAAO,CAAC,KAAK,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAAC,CAAC,CAAC,CAE2D;IAE1H;;;;OAIG;IACG,UAAU,CAAC,KAAK,EAHX;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAGxG,GAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEgC;IAElG;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,eAAe,CAAC,EAAE,MAAM,CAAA;KAGH,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,EAAE,CAAC,CAEsD;IAEnH;;;;OAIG;IACG,iBAAiB,CAAC,KAAK,AAH1B,CACA,EADQ;QAAC,cAAc,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,WAAW,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAA;KAGzD,GAFrB,OAAO,CAAC,MAAM,CAAC,CAEyF;CACtH"}
+{"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../../../src/background-jobs/adapter.js"],"names":[],"mappings":"AAEA;;;;GAIG;AACH,MAAM,CAAC,OAAO,OAAO,qBAAqB;IACxC;;;OAGG;IACG,WAAW,IAFJ,OAAO,CAAC,IAAI,CAAC,CAEqE;IAE/F;;;OAGG;IACG,KAAK,IAFE,OAAO,CAAC,IAAI,CAAC,CAEV;IAEhB;;;OAGG;IACG,MAAM,IAFC,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,CAAC,CAI9D;IAED;;;;;OAKG;IACG,qBAAqB,CAAC,KAAK,EAHtB;QAAC,GAAG,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,6BAA6B,EAAE,OAAO,CAAC,CAAA;KAG7C,GAFpB,OAAO,CAAC,IAAI,CAAC,CAEW;IAErC;;;OAGG;IACG,yBAAyB,IAFlB,OAAO,CAAC,IAAI,CAAC,CAEiG;IAE3H;;;;OAIG;IACG,OAAO,CAAC,KAAK,EAHR;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGzG,GAFN,OAAO,CAAC,MAAM,CAAC,CAEgE;IAE5F;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,EAHjB;QAAC,WAAW,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,IAAI,EAAE,KAAK,CAAC,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC;QAAC,OAAO,CAAC,EAAE,OAAO,YAAY,EAAE,oBAAoB,CAAA;KAGrH,GAFf,OAAO,CAAC,OAAO,YAAY,EAAE,8BAA8B,CAAC,CAEqC;IAE9G;;;;OAIG;IACG,eAAe,CAAC,YAAY,EAHvB,MAGuB,GAFrB,OAAO,CAAC,OAAO,YAAY,EAAE,+BAA+B,CAAC,CAEyC;IAEnH;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,aAAa,CAAC,EAAE,OAAO,YAAY,EAAE,0BAA0B,GAAG,OAAO,YAAY,EAAE,0BAA0B,EAAE,CAAA;KAG9F,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEiD;IAEnH;;;OAGG;IACG,gBAAgB,IAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEuC;IAEzG;;;;OAIG;IACG,MAAM,CAAC,MAAM,EAHR,MAGQ,GAFN,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEyB;IAE3F;;;;;;OAMG;IACG,aAAa,CAAC,KAAK,EAHd,OAAO,YAAY,EAAE,2BAGP,GAFZ,OAAO,CAAC,OAAO,YAAY,EAAE,oBAAoB,GAAG,IAAI,CAAC,CAEkC;IAExG;;;;OAIG;IACG,aAAa,CAAC,KAAK,EAHd;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG/D,GAFZ,OAAO,CAAC,OAAO,CAAC,CAE2E;IAExG;;;;OAIG;IACG,eAAe,CAAC,KAAK,EAHhB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAG9E,GAFd,OAAO,CAAC,OAAO,CAAC,CAE+E;IAE5G;;;;OAIG;IACG,mBAAmB,CAAC,KAAK,EAHpB;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAGb,GAFlB,OAAO,CAAC,IAAI,CAAC,CAE0F;IAEpH;;;;OAIG;IACG,sBAAsB,CAAC,KAAK,EAHvB;QAAC,QAAQ,EAAE,MAAM,CAAA;KAGM,GAFrB,OAAO,CAAC,KAAK,CAAC;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,SAAS,EAAE,MAAM,CAAA;KAAC,CAAC,CAAC,CAE2D;IAE1H;;;;OAIG;IACG,qBAAqB,IAFd,OAAO,CAAC,OAAO,YAAY,EAAE,4BAA4B,EAAE,CAAC,CAE9B;IAE3C;;;;;;OAMG;IACG,oBAAoB,CAAC,KAAK,EAHrB;QAAC,QAAQ,EAAE,OAAO,YAAY,EAAE,4BAA4B,EAAE,CAAC;QAAC,KAAK,EAAE,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAA;KAG/E,GAFnB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,EAAE,CAAC,CAEd;IAE/C;;;;OAIG;IACG,UAAU,CAAC,KAAK,EAHX;QAAC,KAAK,EAAE,MAAM,CAAC;QAAC,KAAK,EAAE,UAAU,CAAC,OAAO,IAAI,CAAC,KAAK,CAAC,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAC;QAAC,QAAQ,CAAC,EAAE,MAAM,CAAC;QAAC,aAAa,CAAC,EAAE,MAAM,CAAA;KAGxG,GAFT,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,GAAG,IAAI,CAAC,CAEgC;IAElG;;;;OAIG;IACG,gBAAgB,CAAC,KAAK,AAHzB,CACA,EADQ;QAAC,eAAe,CAAC,EAAE,MAAM,CAAA;KAGH,GAFpB,OAAO,CAAC,OAAO,YAAY,EAAE,gBAAgB,EAAE,CAAC,CAEsD;IAEnH;;;;OAIG;IACG,iBAAiB,CAAC,KAAK,AAH1B,CACA,EADQ;QAAC,cAAc,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,WAAW,CAAC,EAAE,MAAM,GAAG,IAAI,CAAC;QAAC,SAAS,CAAC,EAAE,MAAM,CAAA;KAGzD,GAFrB,OAAO,CAAC,MAAM,CAAC,CAEyF;CACtH"}