velocious 1.0.627 → 1.0.629

package/README.md

@@ -2310,7 +2310,7 @@ Create the file `src/routes/testing/another-action.ejs` and so something like th
 
 Velocious includes a simple background jobs system inspired by Sidekiq.
 
-Jobs can opt into cross-worker durable concurrency limits by pairing a non-empty `concurrencyKey` with a positive-integer `maxConcurrency` in their background-job options. The first cap registered for a key is stable; conflicting caps are rejected. See [durable concurrency limits](docs/background-jobs.md#durable-concurrency-limits).
+Jobs can opt into cross-worker durable concurrency limits by pairing a non-empty `concurrencyKey` with a positive-integer `maxConcurrency` in their background-job options, or by deriving the key in a hydrated job instance's non-static `concurrencyKey()` method. Explicit enqueue options win. The first cap registered for a key is stable; conflicting caps are rejected. See [durable concurrency limits](docs/background-jobs.md#durable-concurrency-limits).
 
 Production apps can listen for `background-job-failed` (or its `all-error` mirror) to report accepted failed attempts, including retry and terminal-state metadata, and for `background-job-orphaned` to react to a specific job the main process reclaimed after its worker died mid-run — e.g. enqueue a targeted recovery for the work it left behind, instead of only polling for the aftermath. Orphan handlers run before the sweep waits for reclaimed jobs to be dispatched, so a stalled dispatcher does not delay application recovery. See [docs/background-jobs.md](docs/background-jobs.md#failure-events).
 
@@ -2626,7 +2626,7 @@ Each job must define exactly one of `every` or `cron`. Cron times are evaluated
 
 ## Persistence and retries
 
-Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the main generates a unique lease id before asking the adapter to mark the job handed off, and the worker reports completion or failure back to the main process. If that persistence call has an ambiguous result, only the exact caller-generated lease is conditionally returned; failed recovery is retained for the dispatch error-retry path, so worker admission and concurrency do not remain stranded and a newer lease is never reclaimed. Custom adapters must persist a supplied `markHandedOff({handoffId})` exactly; built-in adapters continue generating one for legacy direct callers that omit it. If a worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. When the **main** itself restarts (every deploy), a reconnecting worker's still-active handoffs are adopted into its new socket on `hello` so a later disconnect releases them instead of leaving them stuck until the orphan sweep; the main never time-reclaims a disconnected worker's jobs, so a gracefully-draining old-release worker is not double-run. For rolling upgrades, upgrade workers before the main process: a lease-aware main keeps legacy workers connected for old reports but dispatches new jobs only to workers advertising lease-reporting support. See [docs/background-jobs.md](docs/background-jobs.md#worker-disconnect-recovery).
+Jobs are persisted in the configured database (`backgroundJobs.databaseIdentifier`) in an internal `background_jobs` table. When a worker picks a job, the main generates a unique lease id before asking the adapter to mark the job handed off, and the worker reports completion or failure back to the main process. If that persistence call has an ambiguous result, only the exact caller-generated lease is conditionally returned; failed recovery is retained for the dispatch error-retry path, so worker admission and concurrency do not remain stranded and a newer lease is never reclaimed. Custom adapters must persist a supplied `markHandedOff({handoffId})` exactly; built-in adapters continue generating one for legacy direct callers that omit it. If a worker socket disconnects unexpectedly, only the leases handed to that exact socket are immediately returned to the queue; late reports are fenced by lease id so they cannot mutate a newer attempt. This recovery is at-least-once and may repeat application side effects if the disconnected attempt had already started them. Gracefully draining workers keep their leases while they finish. When the **main** itself restarts (every deploy), it snapshots exact pre-start leases and gives surviving workers 30 seconds to reconnect and adopt their active handoffs. At that deadline it waits up to one additional reconnect-grace interval for adoption queries that were already in flight, preventing a normal slow query from racing reclaim without letting a stuck query defer cleanup forever. A worker is excluded from startup reclaim only after its adoption query succeeds while that same socket is still connected; query failures and disconnects during adoption remain reclaimable. The main then orphans only unchanged startup snapshots belonging to worker ids that did not successfully adopt, using the normal retry/count/concurrency/event lifecycle and immediately waking queued work. A successfully adopted worker, a lease created after startup, and any newer re-handoff are fenced out. `BackgroundJobsMain` accepts `workerReconnectGraceMs` for tests and operational tuning; it must be an integer from 0 through 2,147,483,647 ms and invalid values throw. This startup cleanup does not lower the separate two-hour age sweep. For rolling upgrades, upgrade workers before the main process: a lease-aware main keeps legacy workers connected for old reports but dispatches new jobs only to workers advertising lease-reporting support. See [docs/background-jobs.md](docs/background-jobs.md#worker-disconnect-recovery).
 
 Failed jobs are re-queued with backoff and retried up to 10 times by default (10s, 1m, 10m, 1h, then +1h per retry). You can override the retry limit per job:
 

package/build/background-jobs/adapter.js

@@ -118,6 +118,22 @@ export default class BackgroundJobsAdapter {
    */
   async handedOffJobsForWorker(_args) { throw new Error("BackgroundJobsAdapter#handedOffJobsForWorker is not implemented") }
 
+  /**
+   * Snapshots exact active handoffs before a new main generation accepts worker
+   * reconnects. Adapters that do not persist worker leases may return none.
+   * @returns {Promise<import("./types.js").BackgroundJobHandoffSnapshot[]>} - Exact active handoffs.
+   */
+  async snapshotHandedOffJobs() { return [] }
+
+  /**
+   * Applies orphan failure semantics to unchanged exact handoff snapshots.
+   * Adapters that return startup snapshots must implement the matching fenced
+   * transition.
+   * @param {{handoffs: import("./types.js").BackgroundJobHandoffSnapshot[], error: ReturnType<typeof JSON.parse>}} _args - Exact leases and orphan reason.
+   * @returns {Promise<import("./types.js").BackgroundJobRow[]>} - Accepted transitions.
+   */
+  async markOrphanedHandoffs(_args) { return [] }
+
   /**
    * Marks a handed-off job failed or retryable.
    * @param {{jobId: string, error: ReturnType<typeof JSON.parse>, handoffId?: string, workerId?: string, handedOffAtMs?: number}} _args - Failure report.

package/build/background-jobs/job-runner.js

@@ -93,6 +93,14 @@ export default async function runJobPayload(payload, {closeConnections = true, m
   await registry.load()
   const JobClass = registry.getJobByName(payload.jobName)
   const jobInstance = new JobClass()
+  const jobArgs = payload.args || []
+  jobInstance._setBackgroundJobContext({
+    args: jobArgs,
+    jobClass: JobClass,
+    jobName: payload.jobName,
+    options: payload.options || {},
+    payload
+  })
   /**
    * Perform.
    * @type {(...args: Array<ReturnType<typeof JSON.parse>>) => Promise<void>} */
@@ -107,7 +115,7 @@ export default async function runJobPayload(payload, {closeConnections = true, m
   try {
     try {
       await configuration.withConnections({databaseIdentifiers: JobClass.databaseIdentifiers, name: `Background job runner: ${payload.jobName}`}, async () => {
-        await perform.apply(jobInstance, payload.args || [])
+        await perform.apply(jobInstance, jobArgs)
       })
     } catch (error) {
       if (error instanceof BackgroundJobRescheduleSignal) {

package/build/background-jobs/local-dispatcher.js

@@ -184,7 +184,29 @@ export default class LocalBackgroundJobsDispatcher {
         configuration: this.configuration,
         JobClass,
         jobArgs: job.args,
-        name: `Local background job: ${job.jobName}`
+        jobOptions: {
+          concurrencyKey: job.concurrencyKey || undefined,
+          executionMode: job.executionMode,
+          maxConcurrency: job.maxConcurrency ?? undefined,
+          maxRetries: job.maxRetries ?? undefined,
+          queue: job.queue,
+          scheduledAtMs: job.scheduledAtMs ?? undefined,
+          timeoutMs: job.timeoutMs ?? undefined
+        },
+        name: `Local background job: ${job.jobName}`,
+        payload: {
+          args: job.args,
+          handedOffAtMs: handoff.handedOffAtMs,
+          handoffId: handoff.handoffId,
+          id: job.id,
+          jobName: job.jobName,
+          options: {
+            concurrencyKey: job.concurrencyKey || undefined,
+            executionMode: job.executionMode,
+            maxConcurrency: job.maxConcurrency ?? undefined,
+            queue: job.queue
+          }
+        }
       })
     } catch (error) {
       if (error instanceof BackgroundJobRescheduleSignal) {

package/build/background-jobs/main.js

@@ -34,6 +34,24 @@ const MAX_TIMER_MS = 2_147_483_647 // ~24.8 days
 const WORKER_STALE_TIMEOUT_MS = 60000
 /** How often the main scans workers for staleness. */
 const WORKER_LIVENESS_SWEEP_MS = 15000
+/** Grace for workers from the previous main generation to reconnect and adopt leases. */
+const WORKER_RECONNECT_GRACE_MS = 30000
+const WORKER_RECONNECT_GRACE_VALIDATION_MESSAGE = `workerReconnectGraceMs must be an integer between 0 and ${MAX_TIMER_MS}`
+
+/**
+ * Resolves a startup reconnect grace without allowing Node's timer overflow to
+ * turn an intentionally long grace into an immediate reclaim.
+ * @param {number | undefined} workerReconnectGraceMs - Requested reconnect grace.
+ * @returns {number} - Valid timer delay.
+ */
+function normalizeWorkerReconnectGraceMs(workerReconnectGraceMs) {
+  if (workerReconnectGraceMs === undefined) return WORKER_RECONNECT_GRACE_MS
+  if (!Number.isInteger(workerReconnectGraceMs) || workerReconnectGraceMs < 0 || workerReconnectGraceMs > MAX_TIMER_MS) {
+    throw new TypeError(WORKER_RECONNECT_GRACE_VALIDATION_MESSAGE)
+  }
+
+  return workerReconnectGraceMs
+}
 /**
  * Worker execution mode capabilities.
  * @type {WorkerExecutionModeCapability[]} */
@@ -61,10 +79,11 @@ export default class BackgroundJobsMain {
    * @param {number} [args.port] - Port.
    * @param {number} [args.workerStaleTimeoutMs] - Override how long a silent worker may go before being dropped (default 60000ms).
    * @param {number} [args.workerLivenessSweepMs] - Override how often stale workers are swept for (default 15000ms).
+   * @param {number} [args.workerReconnectGraceMs] - Integer from 0 through 2,147,483,647 overriding how long previous-generation workers may reconnect before exact startup leases are reclaimed (default 30000ms).
    * @param {boolean} [args.closeDatabaseConnectionsOnStop] - Whether stop owns closing the configuration's database pools (default true).
    * @param {() => void | Promise<void>} [args.onStopped] - Lifecycle hook invoked after the main process finishes stopping.
    */
-  constructor({configuration, host, port, workerStaleTimeoutMs, workerLivenessSweepMs, closeDatabaseConnectionsOnStop = true, onStopped}) {
+  constructor({configuration, host, port, workerStaleTimeoutMs, workerLivenessSweepMs, workerReconnectGraceMs, closeDatabaseConnectionsOnStop = true, onStopped}) {
     this.configuration = configuration
     this.closeDatabaseConnectionsOnStop = closeDatabaseConnectionsOnStop
     this.onStopped = onStopped
@@ -78,6 +97,7 @@ export default class BackgroundJobsMain {
     // long is treated as wedged/dead: its leases are released and it is dropped.
     this.workerStaleTimeoutMs = typeof workerStaleTimeoutMs === "number" && workerStaleTimeoutMs >= 1 ? workerStaleTimeoutMs : WORKER_STALE_TIMEOUT_MS
     this.workerLivenessSweepMs = typeof workerLivenessSweepMs === "number" && workerLivenessSweepMs >= 1 ? workerLivenessSweepMs : WORKER_LIVENESS_SWEEP_MS
+    this.workerReconnectGraceMs = normalizeWorkerReconnectGraceMs(workerReconnectGraceMs)
     /** @type {import("./adapter.js").default | undefined} */
     this.adapter = undefined
     this.logger = new Logger(this)
@@ -104,6 +124,17 @@ export default class BackgroundJobsMain {
      * wait for these before closing the configuration's database pools.
      * @type {Set<Promise<void>>} */
     this.inflightWorkerHandoffAdoptions = new Set()
+    /**
+     * Worker ids whose handoffs were successfully adopted by a still-live
+     * connection in this main generation.
+     * @type {Set<string>}
+     */
+    this.reconnectedWorkerIds = new Set()
+    /** @type {import("./types.js").BackgroundJobHandoffSnapshot[]} */
+    this.startupHandoffSnapshot = []
+    /** @type {Promise<void>[]} */
+    this._startupHandoffAdoptionsAtDeadline = []
+    this._startupHandoffGraceElapsed = false
     /**
      * Narrows the runtime value to the documented type.
      * @type {net.Server | undefined} */
@@ -128,6 +159,10 @@ export default class BackgroundJobsMain {
      * Narrows the runtime value to the documented type.
      * @type {ReturnType<typeof setInterval> | undefined} */
     this._workerStaleTimer = undefined
+    /** @type {ReturnType<typeof setTimeout> | undefined} */
+    this._startupHandoffReclaimTimer = undefined
+    /** @type {Promise<void> | undefined} */
+    this._startupHandoffReclaimPromise = undefined
     /**
      * Narrows the runtime value to the documented type.
      * @type {BackgroundJobsScheduler | undefined} */
@@ -178,6 +213,11 @@ export default class BackgroundJobsMain {
   async start() {
     this._stopped = false
     this.stopPromise = undefined
+    this.reconnectedWorkerIds.clear()
+    this.startupHandoffSnapshot = []
+    this._startupHandoffAdoptionsAtDeadline = []
+    this._startupHandoffGraceElapsed = false
+    this._startupHandoffReclaimPromise = undefined
     this.configuration.setCurrent()
 
     try {
@@ -194,6 +234,7 @@ export default class BackgroundJobsMain {
       // across processes with a database advisory lock, so concurrently started
       // mains cannot interleave them.
       await this.store.reconcileQueueConcurrency()
+      this.startupHandoffSnapshot = await this.store.snapshotHandedOffJobs()
       const server = net.createServer((socket) => this._handleConnection(socket))
       this.server = server
 
@@ -208,6 +249,7 @@ export default class BackgroundJobsMain {
       }
 
       this._setupDispatchTriggers()
+      this._setupStartupHandoffReclaim()
 
       this._orphanTimer = setInterval(() => {
         void this._sweepOrphans()
@@ -223,10 +265,7 @@ export default class BackgroundJobsMain {
           await this.store.enqueue({
             jobName: jobClass.jobName(),
             args,
-            // Fold in the job class's static `queue` (as performLater* do) so a
-            // scheduled job with `static queue = "..."` lands on its queue and
-            // honors the configured cap without every schedule repeating it.
-            options: jobClass._withQueue(options)
+            options: jobClass._withJobContext({jobArgs: args, jobOptions: options})
           })
           this._notifyEnqueued()
           // Persistence is the scheduler enqueue boundary. Dispatch remains
@@ -300,7 +339,11 @@ export default class BackgroundJobsMain {
             try {
               await this._drainWorkerHandoffAdoptions()
             } finally {
-              await this._stopBeaconAndServer()
+              try {
+                await this._drainStartupHandoffReclaim()
+              } finally {
+                await this._stopBeaconAndServer()
+              }
             }
           }
         }
@@ -328,11 +371,13 @@ export default class BackgroundJobsMain {
     if (this._errorRetryTimer) clearTimeout(this._errorRetryTimer)
     if (this._orphanTimer) clearInterval(this._orphanTimer)
     if (this._workerStaleTimer) clearInterval(this._workerStaleTimer)
+    if (this._startupHandoffReclaimTimer) clearTimeout(this._startupHandoffReclaimTimer)
     this._pollTimer = undefined
     this._scheduledTimer = undefined
     this._errorRetryTimer = undefined
     this._orphanTimer = undefined
     this._workerStaleTimer = undefined
+    this._startupHandoffReclaimTimer = undefined
   }
 
   /**
@@ -434,6 +479,123 @@ export default class BackgroundJobsMain {
     beaconClient.on("connect", this._beaconConnectHandler)
   }
 
+  /**
+   * Arms the bounded adoption grace only when startup found exact persisted
+   * handoffs. The timer is unrefed so an otherwise-finished process is never
+   * retained solely to perform this cleanup.
+   * @returns {void}
+   */
+  _setupStartupHandoffReclaim() {
+    if (this.startupHandoffSnapshot.length === 0) return
+
+    this._startupHandoffReclaimTimer = setTimeout(() => {
+      this._startupHandoffReclaimTimer = undefined
+      this._startupHandoffAdoptionsAtDeadline = [...this.inflightWorkerHandoffAdoptions]
+      this._startupHandoffGraceElapsed = true
+      void this._startStartupHandoffReclaim()
+    }, this.workerReconnectGraceMs)
+    this._startupHandoffReclaimTimer.unref()
+  }
+
+  /**
+   * Starts one tracked startup-reclaim pass, coalescing lifecycle and retry
+   * callers so shutdown can wait for durable mutation before closing pools.
+   * @returns {Promise<void>} - Resolves after this pass settles.
+   */
+  _startStartupHandoffReclaim() {
+    if (this._startupHandoffReclaimPromise) return this._startupHandoffReclaimPromise
+
+    const reclaim = this._reclaimDisconnectedStartupHandoffs()
+
+    this._startupHandoffReclaimPromise = reclaim
+    const clearReclaim = () => {
+      if (this._startupHandoffReclaimPromise === reclaim) {
+        this._startupHandoffReclaimPromise = undefined
+      }
+    }
+    void reclaim.then(clearReclaim, clearReclaim)
+
+    return reclaim
+  }
+
+  /**
+   * Waits for an already-started startup reclaim before adapter shutdown.
+   * @returns {Promise<void>} - Resolves when no pass remains.
+   */
+  async _drainStartupHandoffReclaim() {
+    while (this._startupHandoffReclaimPromise) {
+      await this._startupHandoffReclaimPromise
+    }
+  }
+
+  /**
+   * Orphans only startup-snapshotted leases whose stable worker id has not been
+   * observed by this main generation. Store fencing rejects completed,
+   * returned, replaced, and re-handed-off rows.
+   * @returns {Promise<void>} - Resolves after reclaim or retained retry state.
+   */
+  async _reclaimDisconnectedStartupHandoffs() {
+    if (this._stopped || !this._startupHandoffGraceElapsed) return
+    if (this.startupHandoffSnapshot.length === 0) return
+
+    await this._waitForStartupHandoffAdoptionsAtDeadline()
+    if (this._stopped) return
+
+    const handoffs = this.startupHandoffSnapshot.filter(({workerId}) => !this.reconnectedWorkerIds.has(workerId))
+
+    if (handoffs.length === 0) {
+      this.startupHandoffSnapshot = []
+      return
+    }
+
+    let orphanedJobs
+
+    try {
+      orphanedJobs = await this.store.markOrphanedHandoffs({
+        error: "Job orphaned after its pre-restart worker did not reconnect",
+        handoffs
+      })
+    } catch (error) {
+      this._reportStartupHandoffReclaimError(error)
+      this._scheduleErrorRetry()
+      return
+    }
+
+    this.startupHandoffSnapshot = []
+    await this._handleOrphanedJobs({
+      jobs: orphanedJobs,
+      warning: "Reclaimed background jobs from workers absent after main restart grace"
+    })
+  }
+
+  /**
+   * Lets adoption queries already running at the reconnect deadline settle
+   * before worker ids are filtered. A second bounded grace prevents a stuck
+   * adapter query from deferring startup reclaim forever.
+   * @returns {Promise<void>} - Resolves when the deadline set settles or times out.
+   */
+  async _waitForStartupHandoffAdoptionsAtDeadline() {
+    const adoptions = this._startupHandoffAdoptionsAtDeadline
+
+    this._startupHandoffAdoptionsAtDeadline = []
+    if (adoptions.length === 0) return
+
+    /** @type {ReturnType<typeof setTimeout> | undefined} */
+    let timer
+    const waitLimit = new Promise((resolve) => {
+      // This lifecycle deadline must not keep the main process alive; the
+      // generic timeout helper intentionally uses a referenced timer.
+      timer = setTimeout(resolve, this.workerReconnectGraceMs)
+      timer.unref()
+    })
+
+    try {
+      await Promise.race([Promise.all(adoptions), waitLimit])
+    } finally {
+      if (timer) clearTimeout(timer)
+    }
+  }
+
   /**
    * Publishes a dispatch wake-up on the Beacon channel. No-op in polling
    * mode or when Beacon is not connected; in those cases the direct
@@ -586,6 +748,7 @@ export default class BackgroundJobsMain {
       for (const {jobId, handoffId} of handoffs) {
         map.set(jobId, handoffId)
       }
+      this.reconnectedWorkerIds.add(workerId)
     } catch (error) {
       this._reportHandoffAdoptError(error)
     }
@@ -823,6 +986,22 @@ export default class BackgroundJobsMain {
     errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
   }
 
+  /**
+   * Reports an unexpected startup-snapshot reclaim failure while retaining the
+   * snapshot for the dispatcher's existing transient-error retry lifecycle.
+   * @param {ReturnType<typeof JSON.parse>} error - Reclaim failure.
+   * @returns {void}
+   */
+  _reportStartupHandoffReclaimError(error) {
+    const normalizedError = error instanceof Error ? error : new Error(String(error))
+    const payload = {context: {stage: "background-job-startup-handoff-reclaim"}, error: normalizedError}
+    const errorEvents = this.configuration.getErrorEvents()
+
+    this.logger.error(() => ["Failed to reclaim disconnected startup handoffs:", normalizedError])
+    errorEvents.emit("framework-error", payload)
+    errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
+  }
+
   /**
    * Runs handle enqueue.
    * @param {object} args - Options.
@@ -1240,6 +1419,7 @@ export default class BackgroundJobsMain {
    * @returns {void} */
   _clearErrorRetryTimer() {
     if (this.pendingHandoffRecoveries.size > 0) return
+    if (this._startupHandoffGraceElapsed && this.startupHandoffSnapshot.length > 0) return
 
     for (const worker of this.workerHandoffs.keys()) {
       if (!this.workers.has(worker)) return
@@ -1314,6 +1494,11 @@ export default class BackgroundJobsMain {
   async _retryAfterError() {
     if (this._stopped) return
 
+    if (this._startupHandoffGraceElapsed && this.startupHandoffSnapshot.length > 0) {
+      await this._startStartupHandoffReclaim()
+      if (this.startupHandoffSnapshot.length > 0) return
+    }
+
     try {
       await this._retryPendingHandoffRecoveries()
     } catch {
@@ -1400,7 +1585,12 @@ export default class BackgroundJobsMain {
             workerId: worker.workerId,
             handedOffAtMs: handoff.handedOffAtMs,
             options: {
+              concurrencyKey: job.concurrencyKey || undefined,
               executionMode: job.executionMode,
+              maxConcurrency: job.maxConcurrency ?? undefined,
+              maxRetries: job.maxRetries ?? undefined,
+              queue: job.queue,
+              scheduledAtMs: job.scheduledAtMs ?? undefined,
               ...(job.timeoutMs === null ? {} : {timeoutMs: job.timeoutMs})
             }
           }
@@ -1648,31 +1838,40 @@ export default class BackgroundJobsMain {
     try {
       const orphanedJobs = await this.store.markOrphanedJobs()
 
-      if (orphanedJobs.length > 0) {
-        this.logger.warn(() => ["Marked orphaned background jobs", orphanedJobs.length])
-        // Reclaimed orphans become `queued` again — wake the dispatcher first so
-        // an application event handler that throws below cannot strand them
-        // queued until the next external enqueue/reconnect.
-        this._notifyEnqueued()
-        // Emit an event per orphaned job so applications can react to a dead
-        // worker's specific job (e.g. targeted recovery) instead of only polling
-        // for its aftermath. Emit before awaiting the drain so a blocked
-        // dispatcher cannot delay application recovery. Isolate each so one
-        // throwing handler can't suppress the events for the rest.
-        for (const job of orphanedJobs) {
-          try {
-            this._emitBackgroundJobOrphaned({job})
-          } catch (error) {
-            this.logger.error(() => ["A background-job-orphaned event handler threw:", error])
-          }
-        }
-        await this._drain()
-      }
+      await this._handleOrphanedJobs({jobs: orphanedJobs, warning: "Marked orphaned background jobs"})
     } catch (error) {
       this.logger.error(() => ["Failed to mark orphaned jobs:", error])
     }
   }
 
+  /**
+   * Publishes the common post-orphan lifecycle: wake queued retries, emit one
+   * isolated event per accepted transition, and drain so released concurrency
+   * can immediately admit other work.
+   * @param {object} args - Options.
+   * @param {import("./types.js").BackgroundJobRow[]} args.jobs - Accepted orphan transitions.
+   * @param {string} args.warning - Lifecycle log message.
+   * @returns {Promise<void>} - Resolves after the resulting drain.
+   */
+  async _handleOrphanedJobs({jobs, warning}) {
+    if (jobs.length === 0) return
+
+    this.logger.warn(() => [warning, jobs.length])
+    // Reclaimed orphans can become `queued` again — wake the dispatcher first
+    // so an application event handler that throws below cannot strand them.
+    this._notifyEnqueued()
+    // Emit before awaiting the drain so a blocked dispatcher cannot delay
+    // application recovery. Isolate handlers so one cannot suppress the rest.
+    for (const job of jobs) {
+      try {
+        this._emitBackgroundJobOrphaned({job})
+      } catch (error) {
+        this.logger.error(() => ["A background-job-orphaned event handler threw:", error])
+      }
+    }
+    await this._drain()
+  }
+
   /**
    * Drops workers that have gone silent past `workerStaleTimeoutMs` (no
    * heartbeat, ready, or report). A wedged worker keeps its socket open, so the

package/build/background-jobs/perform-job.js

@@ -6,11 +6,20 @@
  * @param {import("../configuration.js").default} args.configuration - Active configuration.
  * @param {typeof import("./platform-job.js").default} args.JobClass - Job class.
  * @param {Array<ReturnType<typeof JSON.parse>>} args.jobArgs - Job arguments.
+ * @param {import("./types.js").BackgroundJobOptions} [args.jobOptions] - Resolved runtime options.
  * @param {string} args.name - Connection-scope label.
+ * @param {import("./types.js").BackgroundJobPayload} [args.payload] - Persisted runner payload.
  * @returns {Promise<void>} - Resolves after performance.
  */
-export default async function performBackgroundJob({configuration, JobClass, jobArgs, name}) {
+export default async function performBackgroundJob({configuration, JobClass, jobArgs, jobOptions = {}, name, payload}) {
   const jobInstance = new JobClass()
+  jobInstance._setBackgroundJobContext({
+    args: jobArgs,
+    jobClass: JobClass,
+    jobName: JobClass.jobName(),
+    options: jobOptions,
+    ...(payload ? {payload} : {})
+  })
   /**
    * Narrows the generic subclass's runtime method to serialized job arguments.
    * @type {(...args: Array<ReturnType<typeof JSON.parse>>) => Promise<void>}

package/build/background-jobs/platform-job.js

@@ -14,6 +14,11 @@ import {cancelScheduledBackgroundJob, enqueueBackgroundJob, replaceScheduledBack
  * @template {Array<ReturnType<typeof JSON.parse>>} [TArgs=[]]
  */
 export default class VelociousJob {
+  constructor() {
+    /** @type {import("./types.js").BackgroundJobContext | undefined} */
+    this._backgroundJobContext = undefined
+  }
+
   /**
    * Database identifiers checked out while this job performs. Set an explicit
    * list to avoid holding unrelated configured database connections, or `[]`
@@ -82,6 +87,63 @@ export default class VelociousJob {
     return merged
   }
 
+  /**
+   * Resolves class-derived enqueue options on a hydrated job instance. Explicit
+   * per-enqueue options take precedence over the instance concurrency key.
+   * @param {object} args - Job context.
+   * @param {Array<ReturnType<typeof JSON.parse>>} args.jobArgs - Job arguments.
+   * @param {import("./types.js").BackgroundJobOptions | undefined} args.jobOptions - Job options.
+   * @returns {import("./types.js").BackgroundJobOptions} - Resolved job options.
+   */
+  static _withJobContext({jobArgs, jobOptions}) {
+    const options = this._withQueue(jobOptions)
+
+    if (options.concurrencyKey !== undefined) return options
+
+    const jobInstance = new this()
+    jobInstance._setBackgroundJobContext({
+      args: jobArgs,
+      jobClass: this,
+      jobName: this.jobName(),
+      options
+    })
+    const concurrencyKey = jobInstance.concurrencyKey()
+
+    if (concurrencyKey !== undefined) options.concurrencyKey = concurrencyKey
+
+    return options
+  }
+
+  /**
+   * Sets the complete context available to this hydrated job instance.
+   * Framework enqueue/runner boundaries own this method.
+   * @param {import("./types.js").BackgroundJobContext} context - Job context.
+   * @returns {void}
+   */
+  _setBackgroundJobContext(context) {
+    this._backgroundJobContext = context
+  }
+
+  /**
+   * Returns this hydrated job's complete enqueue or runner context.
+   * @returns {import("./types.js").BackgroundJobContext} - Job context.
+   */
+  backgroundJobContext() {
+    if (!this._backgroundJobContext) throw new Error("Background job context is not hydrated")
+
+    return this._backgroundJobContext
+  }
+
+  /**
+   * Override to derive a durable concurrency key from `backgroundJobContext()`.
+   * Pair the derived key with `maxConcurrency` in enqueue options. An explicit
+   * per-enqueue `concurrencyKey` takes precedence and skips this method.
+   * @returns {string | undefined} - Derived concurrency key, or undefined for none.
+   */
+  concurrencyKey() {
+    return undefined
+  }
+
   /**
    * Runs perform later.
    * @param {...ReturnType<typeof JSON.parse>} args - Job args.

package/build/background-jobs/runtime.js

@@ -52,9 +52,10 @@ export async function enqueueBackgroundJob({JobClass, jobArgs, jobOptions}) {
  * @returns {Promise<string>} - Durable job id or ephemeral inline performance id.
  */
 export async function enqueueBackgroundJobForConfiguration({configuration, JobClass, jobArgs, jobOptions}) {
+  const resolvedJobOptions = JobClass._withJobContext({jobArgs, jobOptions})
 
   if (configuration.getBackgroundJobsConfig().mode === "inline") {
-    validateInlineOptions(jobOptions)
+    validateInlineOptions(resolvedJobOptions)
     configuration.setCurrent()
     await configuration.initialize({type: "background-jobs-inline"})
 
@@ -63,6 +64,7 @@ export async function enqueueBackgroundJobForConfiguration({configuration, JobCl
         configuration,
         JobClass,
         jobArgs,
+        jobOptions: resolvedJobOptions,
         name: `Background job inline mode: ${JobClass.jobName()}`
       })
     } catch (error) {
@@ -81,7 +83,7 @@ export async function enqueueBackgroundJobForConfiguration({configuration, JobCl
   return await client.enqueue({
     jobName: JobClass.jobName(),
     args: jobArgs,
-    options: JobClass._withQueue(jobOptions)
+    options: resolvedJobOptions
   })
 }
 
@@ -117,12 +119,13 @@ export async function replaceScheduledBackgroundJobForConfiguration({configurati
   }
 
   const client = configuration.getEnvironmentHandler().backgroundJobsClient({configuration})
+  const resolvedJobOptions = JobClass._withJobContext({jobArgs, jobOptions})
 
   return await client.replaceScheduled({
     scheduleKey,
     jobName: JobClass.jobName(),
     args: jobArgs,
-    options: JobClass._withQueue(jobOptions)
+    options: resolvedJobOptions
   })
 }