velocious 1.0.628 → 1.0.630

package/README.md

@@ -34,7 +34,7 @@
 * Cross-process broadcast bus for `broadcastToChannel` via `velocious beacon`, including background job runner processes (see [docs/beacon.md](docs/beacon.md))
 * Configurable HTTP server worker handlers plus backpressured, descriptor-only file responses with completion callbacks (see [docs/http-server.md](docs/http-server.md))
 * Default-on buffered HTTP response compression with Brotli/gzip content negotiation, global and per-response opt-outs, and HEAD-correct representation headers (see [docs/http-server.md](docs/http-server.md#response-compression))
-* Background jobs with Node SQL/TCP workers plus a Browser/Expo local SQLite store and in-process dispatcher, failure events, and authorized database-scoped dashboard count snapshots/deltas (see [docs/background-jobs.md](docs/background-jobs.md), [docs/local-background-jobs.md](docs/local-background-jobs.md), and [docs/background-jobs-dashboard.md](docs/background-jobs-dashboard.md))
+* Background jobs with Node SQL/TCP workers plus a Browser/Expo local SQLite store and in-process dispatcher, including failure events and authorized database-scoped dashboard count snapshots/deltas. Release-directory integrations have a required release-scoped jobs-main/worker generation and asynchronous retirement compliance target that current startup adoption does not implement end to end (see [docs/background-jobs.md](docs/background-jobs.md), [docs/local-background-jobs.md](docs/local-background-jobs.md), and [docs/background-jobs-dashboard.md](docs/background-jobs-dashboard.md))
 * Durable one-off background-job scheduling with exact epoch timestamps (see [docs/scheduled-background-job-enqueue.md](docs/scheduled-background-job-enqueue.md))
 * Rails-style request and database query logging (see [docs/logging.md](docs/logging.md))
 * EJS-backed mailers with delivery, queueing, and payload rendering support (see [docs/mailers.md](docs/mailers.md))
@@ -1133,6 +1133,8 @@ FrontendModelBase.setAutoload(false)
 
 Scoped frontend queries (e.g. `Task.where(...).preload([name]).toArray()` from user code) bypass cohort batching by design, same as the backend. Siblings with locally set state from `.setRelationship()` / `.build()` are preserved across cohort batches.
 
+Backend relationship `build(...)` / `create(...)` helpers and generated singular builders with a concrete target use that model's generated write-attribute type. Model-valued relationship attributes are accepted, while unknown and invalid attributes fail type checking. Targetless polymorphic `belongsTo` builders remain generic because no single target write contract exists. See [docs/relationships.md](docs/relationships.md#building-related-records).
+
 ## Through relationships
 
 Use the `through` option on `hasMany` to define a relationship that traverses an intermediate (join) table:
@@ -2310,6 +2312,15 @@ Create the file `src/routes/testing/another-action.ejs` and so something like th
 
 Velocious includes a simple background jobs system inspired by Sidekiq.
 
+In a release-directory production topology, one jobs generation is a
+same-release `background-jobs-main` and worker pool on its own endpoint. Start the
+complete candidate generation before activation. A retired main stops schedules,
+new dispatch, and new handoffs but remains with its old workers to supervise the
+handoffs it already owns until every worker drains. Old workers never transfer to
+the new main. Deploy and HTTP/WebSocket drain completion are independent of this
+potentially hours-long lifecycle. See [release-generation
+draining](docs/background-jobs.md#release-generation-draining).
+
 Jobs can opt into cross-worker durable concurrency limits by pairing a non-empty `concurrencyKey` with a positive-integer `maxConcurrency` in their background-job options, or by deriving the key in a hydrated job instance's non-static `concurrencyKey()` method. Explicit enqueue options win. The first cap registered for a key is stable; conflicting caps are rejected. See [durable concurrency limits](docs/background-jobs.md#durable-concurrency-limits).
 
 Production apps can listen for `background-job-failed` (or its `all-error` mirror) to report accepted failed attempts, including retry and terminal-state metadata, and for `background-job-orphaned` to react to a specific job the main process reclaimed after its worker died mid-run — e.g. enqueue a targeted recovery for the work it left behind, instead of only polling for the aftermath. Orphan handlers run before the sweep waits for reclaimed jobs to be dispatched, so a stalled dispatcher does not delay application recovery. See [docs/background-jobs.md](docs/background-jobs.md#failure-events).
@@ -2419,7 +2430,7 @@ VELOCIOUS_BACKGROUND_JOBS_WORKER_SHUTDOWN_TIMEOUT_MS=indefinite
 VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS=5400000
 ```
 
-`VELOCIOUS_BACKGROUND_JOBS_WORKER_SHUTDOWN_TIMEOUT_MS` (default: `indefinite`) bounds how long a `background-jobs-worker` waits for in-flight jobs on `SIGTERM`/`SIGINT` before terminating any forked or spawned child runners still running, so they are not orphaned across a deploy. The default waits for jobs to finish and never interrupts a running job; set a positive number of milliseconds for a finite cap (keep it shorter than your process supervisor's graceful-stop window so the worker reaps its own runners first). See [docs/background-jobs.md](docs/background-jobs.md#worker-shutdown-and-process-job-draining).
+`VELOCIOUS_BACKGROUND_JOBS_WORKER_SHUTDOWN_TIMEOUT_MS` (default: `indefinite`) bounds how long a `background-jobs-worker` waits for in-flight jobs on `SIGTERM`/`SIGINT` before terminating any forked or spawned child runners still running. The default waits for jobs to finish and never interrupts a running job; a positive finite cap is a per-worker shutdown control for an explicitly requested process stop, not the normal deploy-completion mechanism. During release retirement, the old jobs-main and workers may drain for hours after deploy returns. See [docs/background-jobs.md](docs/background-jobs.md#worker-shutdown-and-process-job-draining).
 
 `maxConcurrentInlineJobs` (default: `4`) caps how many `executionMode: "inline"` jobs a single `background-jobs-worker` process runs in parallel. Concurrency is at the JS event-loop level: every job in flight shares the worker's process and DB connection pool, so the cap should fit the pool, not the CPU count. Forking remains the right tool when you need memory isolation across long-running jobs or want to use more cores; select it with `executionMode: "forked"`.
 
@@ -2626,7 +2637,7 @@ Each job must define exactly one of `every` or `cron`. Cron times are evaluated
 
 ## Persistence and retries
 
-Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the main generates a unique lease id before asking the adapter to mark the job handed off, and the worker reports completion or failure back to the main process. If that persistence call has an ambiguous result, only the exact caller-generated lease is conditionally returned; failed recovery is retained for the dispatch error-retry path, so worker admission and concurrency do not remain stranded and a newer lease is never reclaimed. Custom adapters must persist a supplied `markHandedOff({handoffId})` exactly; built-in adapters continue generating one for legacy direct callers that omit it. If a worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. When the **main** itself restarts (every deploy), a reconnecting worker's still-active handoffs are adopted into its new socket on `hello` so a later disconnect releases them instead of leaving them stuck until the orphan sweep; the main never time-reclaims a disconnected worker's jobs, so a gracefully-draining old-release worker is not double-run. For rolling upgrades, upgrade workers before the main process: a lease-aware main keeps legacy workers connected for old reports but dispatches new jobs only to workers advertising lease-reporting support. See [docs/background-jobs.md](docs/background-jobs.md#worker-disconnect-recovery).
+Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the main generates a unique lease id before asking the adapter to mark the job handed off, and the worker reports completion or failure back to the main process. If that persistence call has an ambiguous result, only the exact caller-generated lease is conditionally returned; failed recovery is retained for the dispatch error-retry path, so worker admission and concurrency do not remain stranded and a newer lease is never reclaimed. Custom adapters must persist a supplied `markHandedOff({handoffId})` exactly; built-in adapters continue generating one for legacy direct callers that omit it. If a worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. Startup reconnection/adoption is an abnormal crash/legacy-recovery facility, not the normal deploy topology: during ordinary release retirement the old main remains alive and owns its old workers, and they must not reconnect to the new main. A production integration that restarts jobs-main every deploy and depends on worker adoption is not compliant with the release-generation contract. See [release-generation draining](docs/background-jobs.md#release-generation-draining) and [worker disconnect recovery](docs/background-jobs.md#worker-disconnect-recovery).
 
 Failed jobs are re-queued with backoff and retried up to 10 times by default (10s, 1m, 10m, 1h, then +1h per retry). You can override the retry limit per job:
 

package/build/background-jobs/adapter.js

@@ -118,6 +118,22 @@ export default class BackgroundJobsAdapter {
    */
   async handedOffJobsForWorker(_args) { throw new Error("BackgroundJobsAdapter#handedOffJobsForWorker is not implemented") }
 
+  /**
+   * Snapshots exact active handoffs before a new main generation accepts worker
+   * reconnects. Adapters that do not persist worker leases may return none.
+   * @returns {Promise<import("./types.js").BackgroundJobHandoffSnapshot[]>} - Exact active handoffs.
+   */
+  async snapshotHandedOffJobs() { return [] }
+
+  /**
+   * Applies orphan failure semantics to unchanged exact handoff snapshots.
+   * Adapters that return startup snapshots must implement the matching fenced
+   * transition.
+   * @param {{handoffs: import("./types.js").BackgroundJobHandoffSnapshot[], error: ReturnType<typeof JSON.parse>}} _args - Exact leases and orphan reason.
+   * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+   */
+  async markOrphanedHandoffs(_args) { return [] }
+
   /**
    * Marks a handed-off job failed or retryable.
    * @param {{jobId: string, error: ReturnType<typeof JSON.parse>, handoffId?: string, workerId?: string, handedOffAtMs?: number}} _args - Failure report.

package/build/background-jobs/main.js

@@ -34,6 +34,24 @@ const MAX_TIMER_MS = 2_147_483_647 // ~24.8 days
 const WORKER_STALE_TIMEOUT_MS = 60000
 /** How often the main scans workers for staleness. */
 const WORKER_LIVENESS_SWEEP_MS = 15000
+/** Grace for workers from the previous main generation to reconnect and adopt leases. */
+const WORKER_RECONNECT_GRACE_MS = 30000
+const WORKER_RECONNECT_GRACE_VALIDATION_MESSAGE = `workerReconnectGraceMs must be an integer between 0 and ${MAX_TIMER_MS}`
+
+/**
+ * Resolves a startup reconnect grace without allowing Node's timer overflow to
+ * turn an intentionally long grace into an immediate reclaim.
+ * @param {number | undefined} workerReconnectGraceMs - Requested reconnect grace.
+ * @returns {number} - Valid timer delay.
+ */
+function normalizeWorkerReconnectGraceMs(workerReconnectGraceMs) {
+  if (workerReconnectGraceMs === undefined) return WORKER_RECONNECT_GRACE_MS
+  if (!Number.isInteger(workerReconnectGraceMs) || workerReconnectGraceMs < 0 || workerReconnectGraceMs > MAX_TIMER_MS) {
+    throw new TypeError(WORKER_RECONNECT_GRACE_VALIDATION_MESSAGE)
+  }
+
+  return workerReconnectGraceMs
+}
 /**
  * Worker execution mode capabilities.
  * @type {WorkerExecutionModeCapability[]} */
@@ -61,10 +79,11 @@ export default class BackgroundJobsMain {
    * @param {number} [args.port] - Port.
    * @param {number} [args.workerStaleTimeoutMs] - Override how long a silent worker may go before being dropped (default 60000ms).
    * @param {number} [args.workerLivenessSweepMs] - Override how often stale workers are swept for (default 15000ms).
+   * @param {number} [args.workerReconnectGraceMs] - Integer from 0 through 2,147,483,647 overriding how long previous-generation workers may reconnect before exact startup leases are reclaimed (default 30000ms).
    * @param {boolean} [args.closeDatabaseConnectionsOnStop] - Whether stop owns closing the configuration's database pools (default true).
    * @param {() => void | Promise<void>} [args.onStopped] - Lifecycle hook invoked after the main process finishes stopping.
    */
-  constructor({configuration, host, port, workerStaleTimeoutMs, workerLivenessSweepMs, closeDatabaseConnectionsOnStop = true, onStopped}) {
+  constructor({configuration, host, port, workerStaleTimeoutMs, workerLivenessSweepMs, workerReconnectGraceMs, closeDatabaseConnectionsOnStop = true, onStopped}) {
     this.configuration = configuration
     this.closeDatabaseConnectionsOnStop = closeDatabaseConnectionsOnStop
     this.onStopped = onStopped
@@ -78,6 +97,7 @@ export default class BackgroundJobsMain {
     // long is treated as wedged/dead: its leases are released and it is dropped.
     this.workerStaleTimeoutMs = typeof workerStaleTimeoutMs === "number" && workerStaleTimeoutMs >= 1 ? workerStaleTimeoutMs : WORKER_STALE_TIMEOUT_MS
     this.workerLivenessSweepMs = typeof workerLivenessSweepMs === "number" && workerLivenessSweepMs >= 1 ? workerLivenessSweepMs : WORKER_LIVENESS_SWEEP_MS
+    this.workerReconnectGraceMs = normalizeWorkerReconnectGraceMs(workerReconnectGraceMs)
     /** @type {import("./adapter.js").default | undefined} */
     this.adapter = undefined
     this.logger = new Logger(this)
@@ -104,6 +124,17 @@ export default class BackgroundJobsMain {
      * wait for these before closing the configuration's database pools.
      * @type {Set<Promise<void>>} */
     this.inflightWorkerHandoffAdoptions = new Set()
+    /**
+     * Worker ids whose handoffs were successfully adopted by a still-live
+     * connection in this main generation.
+     * @type {Set<string>}
+     */
+    this.reconnectedWorkerIds = new Set()
+    /** @type {import("./types.js").BackgroundJobHandoffSnapshot[]} */
+    this.startupHandoffSnapshot = []
+    /** @type {Promise<void>[]} */
+    this._startupHandoffAdoptionsAtDeadline = []
+    this._startupHandoffGraceElapsed = false
     /**
      * Narrows the runtime value to the documented type.
      * @type {net.Server | undefined} */
@@ -128,6 +159,10 @@ export default class BackgroundJobsMain {
      * Narrows the runtime value to the documented type.
      * @type {ReturnType<typeof setInterval> | undefined} */
     this._workerStaleTimer = undefined
+    /** @type {ReturnType<typeof setTimeout> | undefined} */
+    this._startupHandoffReclaimTimer = undefined
+    /** @type {Promise<void> | undefined} */
+    this._startupHandoffReclaimPromise = undefined
     /**
      * Narrows the runtime value to the documented type.
      * @type {BackgroundJobsScheduler | undefined} */
@@ -178,6 +213,11 @@ export default class BackgroundJobsMain {
   async start() {
     this._stopped = false
     this.stopPromise = undefined
+    this.reconnectedWorkerIds.clear()
+    this.startupHandoffSnapshot = []
+    this._startupHandoffAdoptionsAtDeadline = []
+    this._startupHandoffGraceElapsed = false
+    this._startupHandoffReclaimPromise = undefined
     this.configuration.setCurrent()
 
     try {
@@ -194,6 +234,7 @@ export default class BackgroundJobsMain {
       // across processes with a database advisory lock, so concurrently started
       // mains cannot interleave them.
       await this.store.reconcileQueueConcurrency()
+      this.startupHandoffSnapshot = await this.store.snapshotHandedOffJobs()
       const server = net.createServer((socket) => this._handleConnection(socket))
       this.server = server
 
@@ -208,6 +249,7 @@ export default class BackgroundJobsMain {
       }
 
       this._setupDispatchTriggers()
+      this._setupStartupHandoffReclaim()
 
       this._orphanTimer = setInterval(() => {
         void this._sweepOrphans()
@@ -297,7 +339,11 @@ export default class BackgroundJobsMain {
             try {
               await this._drainWorkerHandoffAdoptions()
             } finally {
-              await this._stopBeaconAndServer()
+              try {
+                await this._drainStartupHandoffReclaim()
+              } finally {
+                await this._stopBeaconAndServer()
+              }
             }
           }
         }
@@ -325,11 +371,13 @@ export default class BackgroundJobsMain {
     if (this._errorRetryTimer) clearTimeout(this._errorRetryTimer)
     if (this._orphanTimer) clearInterval(this._orphanTimer)
     if (this._workerStaleTimer) clearInterval(this._workerStaleTimer)
+    if (this._startupHandoffReclaimTimer) clearTimeout(this._startupHandoffReclaimTimer)
     this._pollTimer = undefined
     this._scheduledTimer = undefined
     this._errorRetryTimer = undefined
     this._orphanTimer = undefined
     this._workerStaleTimer = undefined
+    this._startupHandoffReclaimTimer = undefined
   }
 
   /**
@@ -431,6 +479,123 @@ export default class BackgroundJobsMain {
     beaconClient.on("connect", this._beaconConnectHandler)
   }
 
+  /**
+   * Arms the bounded adoption grace only when startup found exact persisted
+   * handoffs. The timer is unrefed so an otherwise-finished process is never
+   * retained solely to perform this cleanup.
+   * @returns {void}
+   */
+  _setupStartupHandoffReclaim() {
+    if (this.startupHandoffSnapshot.length === 0) return
+
+    this._startupHandoffReclaimTimer = setTimeout(() => {
+      this._startupHandoffReclaimTimer = undefined
+      this._startupHandoffAdoptionsAtDeadline = [...this.inflightWorkerHandoffAdoptions]
+      this._startupHandoffGraceElapsed = true
+      void this._startStartupHandoffReclaim()
+    }, this.workerReconnectGraceMs)
+    this._startupHandoffReclaimTimer.unref()
+  }
+
+  /**
+   * Starts one tracked startup-reclaim pass, coalescing lifecycle and retry
+   * callers so shutdown can wait for durable mutation before closing pools.
+   * @returns {Promise<void>} - Resolves after this pass settles.
+   */
+  _startStartupHandoffReclaim() {
+    if (this._startupHandoffReclaimPromise) return this._startupHandoffReclaimPromise
+
+    const reclaim = this._reclaimDisconnectedStartupHandoffs()
+
+    this._startupHandoffReclaimPromise = reclaim
+    const clearReclaim = () => {
+      if (this._startupHandoffReclaimPromise === reclaim) {
+        this._startupHandoffReclaimPromise = undefined
+      }
+    }
+    void reclaim.then(clearReclaim, clearReclaim)
+
+    return reclaim
+  }
+
+  /**
+   * Waits for an already-started startup reclaim before adapter shutdown.
+   * @returns {Promise<void>} - Resolves when no pass remains.
+   */
+  async _drainStartupHandoffReclaim() {
+    while (this._startupHandoffReclaimPromise) {
+      await this._startupHandoffReclaimPromise
+    }
+  }
+
+  /**
+   * Orphans only startup-snapshotted leases whose stable worker id has not been
+   * observed by this main generation. Store fencing rejects completed,
+   * returned, replaced, and re-handed-off rows.
+   * @returns {Promise<void>} - Resolves after reclaim or retained retry state.
+   */
+  async _reclaimDisconnectedStartupHandoffs() {
+    if (this._stopped || !this._startupHandoffGraceElapsed) return
+    if (this.startupHandoffSnapshot.length === 0) return
+
+    await this._waitForStartupHandoffAdoptionsAtDeadline()
+    if (this._stopped) return
+
+    const handoffs = this.startupHandoffSnapshot.filter(({workerId}) => !this.reconnectedWorkerIds.has(workerId))
+
+    if (handoffs.length === 0) {
+      this.startupHandoffSnapshot = []
+      return
+    }
+
+    let orphanedJobs
+
+    try {
+      orphanedJobs = await this.store.markOrphanedHandoffs({
+        error: "Job orphaned after its pre-restart worker did not reconnect",
+        handoffs
+      })
+    } catch (error) {
+      this._reportStartupHandoffReclaimError(error)
+      this._scheduleErrorRetry()
+      return
+    }
+
+    this.startupHandoffSnapshot = []
+    await this._handleOrphanedJobs({
+      jobs: orphanedJobs,
+      warning: "Reclaimed background jobs from workers absent after main restart grace"
+    })
+  }
+
+  /**
+   * Lets adoption queries already running at the reconnect deadline settle
+   * before worker ids are filtered. A second bounded grace prevents a stuck
+   * adapter query from deferring startup reclaim forever.
+   * @returns {Promise<void>} - Resolves when the deadline set settles or times out.
+   */
+  async _waitForStartupHandoffAdoptionsAtDeadline() {
+    const adoptions = this._startupHandoffAdoptionsAtDeadline
+
+    this._startupHandoffAdoptionsAtDeadline = []
+    if (adoptions.length === 0) return
+
+    /** @type {ReturnType<typeof setTimeout> | undefined} */
+    let timer
+    const waitLimit = new Promise((resolve) => {
+      // This lifecycle deadline must not keep the main process alive; the
+      // generic timeout helper intentionally uses a referenced timer.
+      timer = setTimeout(resolve, this.workerReconnectGraceMs)
+      timer.unref()
+    })
+
+    try {
+      await Promise.race([Promise.all(adoptions), waitLimit])
+    } finally {
+      if (timer) clearTimeout(timer)
+    }
+  }
+
   /**
    * Publishes a dispatch wake-up on the Beacon channel. No-op in polling
    * mode or when Beacon is not connected; in those cases the direct
@@ -583,6 +748,7 @@ export default class BackgroundJobsMain {
       for (const {jobId, handoffId} of handoffs) {
         map.set(jobId, handoffId)
       }
+      this.reconnectedWorkerIds.add(workerId)
     } catch (error) {
       this._reportHandoffAdoptError(error)
     }
@@ -820,6 +986,22 @@ export default class BackgroundJobsMain {
     errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
   }
 
+  /**
+   * Reports an unexpected startup-snapshot reclaim failure while retaining the
+   * snapshot for the dispatcher's existing transient-error retry lifecycle.
+   * @param {ReturnType<typeof JSON.parse>} error - Reclaim failure.
+   * @returns {void}
+   */
+  _reportStartupHandoffReclaimError(error) {
+    const normalizedError = error instanceof Error ? error : new Error(String(error))
+    const payload = {context: {stage: "background-job-startup-handoff-reclaim"}, error: normalizedError}
+    const errorEvents = this.configuration.getErrorEvents()
+
+    this.logger.error(() => ["Failed to reclaim disconnected startup handoffs:", normalizedError])
+    errorEvents.emit("framework-error", payload)
+    errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
+  }
+
   /**
    * Runs handle enqueue.
    * @param {object} args - Options.
@@ -1237,6 +1419,7 @@ export default class BackgroundJobsMain {
    * @returns {void} */
   _clearErrorRetryTimer() {
     if (this.pendingHandoffRecoveries.size > 0) return
+    if (this._startupHandoffGraceElapsed && this.startupHandoffSnapshot.length > 0) return
 
     for (const worker of this.workerHandoffs.keys()) {
       if (!this.workers.has(worker)) return
@@ -1311,6 +1494,11 @@ export default class BackgroundJobsMain {
   async _retryAfterError() {
     if (this._stopped) return
 
+    if (this._startupHandoffGraceElapsed && this.startupHandoffSnapshot.length > 0) {
+      await this._startStartupHandoffReclaim()
+      if (this.startupHandoffSnapshot.length > 0) return
+    }
+
     try {
       await this._retryPendingHandoffRecoveries()
     } catch {
@@ -1650,31 +1838,40 @@ export default class BackgroundJobsMain {
     try {
       const orphanedJobs = await this.store.markOrphanedJobs()
 
-      if (orphanedJobs.length > 0) {
-        this.logger.warn(() => ["Marked orphaned background jobs", orphanedJobs.length])
-        // Reclaimed orphans become `queued` again — wake the dispatcher first so
-        // an application event handler that throws below cannot strand them
-        // queued until the next external enqueue/reconnect.
-        this._notifyEnqueued()
-        // Emit an event per orphaned job so applications can react to a dead
-        // worker's specific job (e.g. targeted recovery) instead of only polling
-        // for its aftermath. Emit before awaiting the drain so a blocked
-        // dispatcher cannot delay application recovery. Isolate each so one
-        // throwing handler can't suppress the events for the rest.
-        for (const job of orphanedJobs) {
-          try {
-            this._emitBackgroundJobOrphaned({job})
-          } catch (error) {
-            this.logger.error(() => ["A background-job-orphaned event handler threw:", error])
-          }
-        }
-        await this._drain()
-      }
+      await this._handleOrphanedJobs({jobs: orphanedJobs, warning: "Marked orphaned background jobs"})
     } catch (error) {
       this.logger.error(() => ["Failed to mark orphaned jobs:", error])
     }
   }
 
+  /**
+   * Publishes the common post-orphan lifecycle: wake queued retries, emit one
+   * isolated event per accepted transition, and drain so released concurrency
+   * can immediately admit other work.
+   * @param {object} args - Options.
+   * @param {import("./types.js").BackgroundJobRow[]} args.jobs - Accepted orphan transitions.
+   * @param {string} args.warning - Lifecycle log message.
+   * @returns {Promise<void>} - Resolves after the resulting drain.
+   */
+  async _handleOrphanedJobs({jobs, warning}) {
+    if (jobs.length === 0) return
+
+    this.logger.warn(() => [warning, jobs.length])
+    // Reclaimed orphans can become `queued` again — wake the dispatcher first
+    // so an application event handler that throws below cannot strand them.
+    this._notifyEnqueued()
+    // Emit before awaiting the drain so a blocked dispatcher cannot delay
+    // application recovery. Isolate handlers so one cannot suppress the rest.
+    for (const job of jobs) {
+      try {
+        this._emitBackgroundJobOrphaned({job})
+      } catch (error) {
+        this.logger.error(() => ["A background-job-orphaned event handler threw:", error])
+      }
+    }
+    await this._drain()
+  }
+
   /**
    * Drops workers that have gone silent past `workerStaleTimeoutMs` (no
    * heartbeat, ready, or report). A wedged worker keeps its socket open, so the

package/build/background-jobs/store.js

@@ -43,6 +43,13 @@ import {
  * @property {number | null} timeoutMs - Per-job timeout override, or null when omitted.
  */
 
+/**
+ * BackgroundJobOrphanSelection type.
+ * @typedef {object} BackgroundJobOrphanSelection
+ * @property {Record<string, ReturnType<typeof JSON.parse>>} conditions - Exact update fence.
+ * @property {import("./types.js").BackgroundJobRow} job - Selected active handoff.
+ */
+
 /**
  * BackgroundJobTransactionSerializationOptions type.
  * @typedef {object} BackgroundJobTransactionSerializationOptions
@@ -1096,6 +1103,81 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
     return handoffs
   }
 
+  /**
+   * Snapshots exact, lease-aware active handoffs before a new main generation
+   * starts accepting worker reconnects. Legacy rows without a complete worker,
+   * lease, and timestamp identity stay owned by the age-based orphan sweep.
+   * @returns {Promise<import("./types.js").BackgroundJobHandoffSnapshot[]>} - Exact startup handoffs.
+   */
+  async snapshotHandedOffJobs() {
+    await this.ensureReady()
+
+    const rows = await this._withDb(async (db) => await db
+      .newQuery()
+      .from(JOBS_TABLE)
+      .where({status: "handed_off"})
+      .order("created_at_ms ASC")
+      .order("id ASC")
+      .results())
+    /** @type {import("./types.js").BackgroundJobHandoffSnapshot[]} */
+    const handoffs = []
+
+    for (const row of rows) {
+      const job = this._normalizeJobRow(row)
+
+      if (!job.handoffId || !job.workerId || typeof job.handedOffAtMs !== "number") continue
+
+      handoffs.push({
+        handedOffAtMs: job.handedOffAtMs,
+        handoffId: job.handoffId,
+        jobId: job.id,
+        workerId: job.workerId
+      })
+    }
+
+    return handoffs
+  }
+
+  /**
+   * Reclaims only unchanged exact handoffs selected by a main-generation startup
+   * snapshot. The ordinary orphan failure path owns retries, terminal status,
+   * count transitions, schedule ownership, and concurrency release.
+   * @param {object} args - Options.
+   * @param {import("./types.js").BackgroundJobHandoffSnapshot[]} args.handoffs - Exact startup snapshots.
+   * @param {ReturnType<typeof JSON.parse>} args.error - Orphan reason.
+   * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+   */
+  async markOrphanedHandoffs({handoffs, error}) {
+    await this.ensureReady()
+
+    return await this._serializedCountMutation(async (db) => {
+      /** @type {BackgroundJobOrphanSelection[]} */
+      const selections = []
+
+      for (const handoff of handoffs) {
+        const job = await this._getJobRowById(db, handoff.jobId)
+
+        if (!job || job.status !== "handed_off") continue
+        if (job.handoffId !== handoff.handoffId) continue
+        if (job.workerId !== handoff.workerId) continue
+        if (job.handedOffAtMs !== handoff.handedOffAtMs) continue
+
+        selections.push({
+          conditions: {
+            handed_off_at_ms: handoff.handedOffAtMs,
+            handoff_id: handoff.handoffId,
+            id: handoff.jobId,
+            status: "handed_off",
+            worker_id: handoff.workerId
+          },
+          job
+        })
+      }
+
+      return await this._markOrphanSelections({db, error, selections})
+    })
+  }
+
   /**
    * Runs mark failed.
    * @param {object} args - Options.
@@ -1141,8 +1223,8 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
 
       const rows = await query.results()
 
-      /** @type {import("./types.js").BackgroundJobRow[]} */
-      const orphanedJobs = []
+      /** @type {BackgroundJobOrphanSelection[]} */
+      const selections = []
 
       for (const row of rows) {
         const job = this._normalizeJobRow(row)
@@ -1160,28 +1242,55 @@ export default class BackgroundJobsStore extends BackgroundJobsAdapter {
         //      wrongly release the concurrency reservation of — that new lease.
         // `handed_off_at_ms` is always set on a handed-off row (and the SELECT
         // required it `<= cutoff`), so it is a reliable null-safe lease pin.
-        const orphanedJob = await this._applyFailure({
-          db,
-          job,
-          error: "Job orphaned after timeout",
-          markOrphaned: true,
-          conditions: {id: job.id, status: "handed_off", handed_off_at_ms: job.handedOffAtMs}
+        selections.push({
+          conditions: {id: job.id, status: "handed_off", handed_off_at_ms: job.handedOffAtMs},
+          job
         })
-
-        if (orphanedJob) orphanedJobs.push(orphanedJob)
       }
 
-      const statusCounts = this._statusCounts(orphanedJobs)
-      const deltas = this._emptyCountBuckets()
+      return await this._markOrphanSelections({
+        db,
+        error: "Job orphaned after timeout",
+        selections
+      })
+    })
+  }
+
+  /**
+   * Applies the common fenced orphan transition and records one aggregate count
+   * delta for the accepted rows.
+   * @param {object} args - Options.
+   * @param {import("../database/drivers/base.js").default} args.db - Transaction connection.
+   * @param {ReturnType<typeof JSON.parse>} args.error - Orphan reason.
+   * @param {BackgroundJobOrphanSelection[]} args.selections - Selected handoffs and exact fences.
+   * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+   */
+  async _markOrphanSelections({db, error, selections}) {
+    /** @type {import("./types.js").BackgroundJobRow[]} */
+    const orphanedJobs = []
+
+    for (const {conditions, job} of selections) {
+      const orphanedJob = await this._applyFailure({
+        conditions,
+        db,
+        error,
+        job,
+        markOrphaned: true
+      })
 
-      for (const [status, count] of Object.entries(statusCounts)) {
-        deltas.handed_off -= count
-        deltas[status] += count
-      }
-      await this._recordCountDelta(db, deltas)
+      if (orphanedJob) orphanedJobs.push(orphanedJob)
+    }
 
-      return orphanedJobs
-    })
+    const statusCounts = this._statusCounts(orphanedJobs)
+    const deltas = this._emptyCountBuckets()
+
+    for (const [status, count] of Object.entries(statusCounts)) {
+      deltas.handed_off -= count
+      deltas[status] += count
+    }
+    await this._recordCountDelta(db, deltas)
+
+    return orphanedJobs
   }
 
   /**

package/build/background-jobs/types.js

@@ -43,6 +43,13 @@
  * @property {string} handoffId - Unique handoff lease id.
  * @property {number} handedOffAtMs - Time handed to a worker in ms.
  */
+/**
+ * @typedef {object} BackgroundJobHandoffSnapshot
+ * @property {string} jobId - Job holding the lease.
+ * @property {string} handoffId - Exact durable lease id.
+ * @property {string} workerId - Stable worker id that received the lease.
+ * @property {number} handedOffAtMs - Time handed to the worker in ms.
+ */
 /**
  * @typedef {object} BackgroundJobHandoffRequest
  * @property {string} jobId - Job to claim.

package/build/database/record/instance-relationships/base.js

@@ -55,7 +55,7 @@ export default class VelociousDatabaseRecordBaseInstanceRelationship {
   /**
    * Runs build.
    * @abstract
-   * @param {Record<string, ReturnType<typeof JSON.parse>>} attributes - Attributes.
+   * @param {ConstructorParameters<TMC>[0]} attributes - Target model write attributes.
    * @returns {InstanceType<TMC>} - The build.
    */
   build(attributes) { // eslint-disable-line no-unused-vars

package/build/database/record/instance-relationships/belongs-to.js

@@ -19,7 +19,7 @@ export default class VelociousDatabaseRecordBelongsToInstanceRelationship extend
 
   /**
    * Runs build.
-   * @param {Record<string, ReturnType<typeof JSON.parse>>} data - Data payload.
+   * @param {ConstructorParameters<TMC>[0]} data - Target model write attributes.
    * @returns {InstanceType<TMC>} - The build.
    */
   build(data) {