velocious 1.0.527 → 1.0.529

package/README.md

@@ -25,6 +25,7 @@
 * Cross-process broadcast bus for `broadcastToChannel` via `velocious beacon`, including background job runner processes (see [docs/beacon.md](docs/beacon.md))
 * Configurable HTTP server worker handlers plus backpressured, descriptor-only file responses with completion callbacks (see [docs/http-server.md](docs/http-server.md))
 * Background jobs with failure events for production reporting (see [docs/background-jobs.md](docs/background-jobs.md))
+* Durable one-off background-job scheduling with exact epoch timestamps (see [docs/scheduled-background-job-enqueue.md](docs/scheduled-background-job-enqueue.md))
 * Rails-style request and database query logging (see [docs/logging.md](docs/logging.md))
 * EJS-backed mailers with delivery, queueing, and payload rendering support (see [docs/mailers.md](docs/mailers.md))
 * Trusted reverse proxy handling for `request.remoteAddress()` (see [docs/trusted-proxies.md](docs/trusted-proxies.md))
@@ -1975,6 +1976,10 @@ export default new Configuration({
     databaseIdentifier: "default",
     maxConcurrentForkedJobs: 4,
     maxConcurrentInlineJobs: 4,
+    pooledRunnerCount: 4,
+    pooledRunnerMaxJobs: 100,
+    pooledRunnerMaxRssBytes: 536870912,
+    pooledRunnerMaxLifetimeMs: 3600000,
     dispatchStrategy: "beacon",
     jobTimeoutMs: null
   }
@@ -1989,6 +1994,10 @@ VELOCIOUS_BACKGROUND_JOBS_PORT=7331
 VELOCIOUS_BACKGROUND_JOBS_DATABASE_IDENTIFIER=default
 VELOCIOUS_BACKGROUND_JOBS_MAX_CONCURRENT_FORKED_JOBS=4
 VELOCIOUS_BACKGROUND_JOBS_MAX_CONCURRENT_INLINE_JOBS=4
+VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_COUNT=4
+VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_MAX_JOBS=100
+VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_MAX_RSS_BYTES=536870912
+VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_MAX_LIFETIME_MS=3600000
 VELOCIOUS_BACKGROUND_JOBS_DISPATCH_STRATEGY=beacon
 VELOCIOUS_BACKGROUND_JOBS_POLL_INTERVAL_MS=1000
 VELOCIOUS_BACKGROUND_JOBS_WORKER_SHUTDOWN_TIMEOUT_MS=indefinite
@@ -1999,6 +2008,8 @@ VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS=5400000
 
 `maxConcurrentInlineJobs` (default: `4`) caps how many `executionMode: "inline"` jobs a single `background-jobs-worker` process runs in parallel. Concurrency is at the JS event-loop level: every job in flight shares the worker's process and DB connection pool, so the cap should fit the pool, not the CPU count. Forking remains the right tool when you need memory isolation across long-running jobs or want to use more cores. The older `forked: false` option still maps to inline mode.
 
+New jobs default to `executionMode: "pooled"`: a worker runs them serially in warm, reusable Node child runners. `pooledRunnerCount` (default: `4`) bounds this independent per-worker capacity. It and `pooledRunnerMaxJobs` must be finite positive integers; the RSS and lifetime limits must be finite positive numbers. A child is recycled after an acknowledged job when it reaches `pooledRunnerMaxJobs` (default: `100`), `pooledRunnerMaxRssBytes` (default: `536870912`, or 512 MiB), or `pooledRunnerMaxLifetimeMs` (default: `3600000`, or one hour). Pooled rows remain readable by older mains during rolling upgrades and run there with legacy one-shot fork behavior. `forked: true` remains an explicit alias for `"forked"`; `forked: false` remains inline. See [execution modes and pooled runners](docs/background-jobs.md#execution-modes-and-pooled-runners).
+
 `maxConcurrentForkedJobs` (default: `4`) caps how many out-of-process `executionMode: "forked"` or `executionMode: "spawned"` jobs one worker may keep in flight. Forked jobs use `child_process.fork()` with an attached IPC channel. After the main process acknowledges their durable status report, forked and spawned one-shot runners exit without waiting for graceful Beacon/database teardown; the OS closes their process-owned resources. A missing or rejected status acknowledgement makes the runner exit as failed instead of reporting clean success. Spawned jobs use the legacy `background-jobs-runner` CLI process via `child_process.spawn()` and are only for callers that intentionally want that spawned behavior.
 
 `jobTimeoutMs` (or `VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS`, milliseconds; default: disabled) is a wall-clock backstop for a single `"forked"` runner. A forked job still running after the timeout is terminated (`SIGTERM`, then `SIGKILL` after the reaping grace) and reported `failed`, so a genuinely-hung runner can't pin a worker — and its whole-app boot and DB connections — indefinitely (notably a retired-release worker draining after a deploy). It's a coarse safety net, not per-job tuning: it applies to every forked runner, so set it well above the longest legitimate forked job. Omit it, or set `null`/`<= 0`, to disable. `"inline"` jobs are not covered — they share the worker's process and can't be killed without killing the worker. See [docs/background-jobs.md](docs/background-jobs.md#forked-job-timeout-hung-runner-backstop).
@@ -2052,7 +2063,7 @@ await MyJob.performLaterWithOptions({
 })
 ```
 
-Schedule a one-off job for a specific epoch timestamp in milliseconds:
+[Schedule a one-off job](docs/scheduled-background-job-enqueue.md) for a specific epoch timestamp in milliseconds:
 
 ```js
 await MyJob.performLaterWithOptions({
@@ -2129,7 +2140,7 @@ Cron fields are: `minute hour day-of-month month day-of-week`. Supported syntax:
 
 Each job must define exactly one of `every` or `cron`. Cron times are evaluated in the **server's local timezone**, at minute granularity.
 
-`background-jobs-main` owns the schedule and enqueues the configured jobs into the normal Velocious background-jobs queue. The HTTP server does not run scheduled jobs itself.
+`background-jobs-main` owns the schedule and enqueues the configured jobs into the normal Velocious background-jobs queue. The HTTP server does not run scheduled jobs itself. Graceful main shutdown waits for scheduled enqueues already in flight before closing database connections, so stopped schedulers cannot write into a later application or test lifecycle.
 
 ## Persistence and retries
 

package/build/background-jobs/forked-runner-child.js

@@ -1,12 +1,13 @@
 // @ts-check
 
 import runJobPayload from "./job-runner.js"
+import setRunnerProcessTitle from "./runner-process-title.js"
 
 // Name the process so `ps`/`top`/`htop` can identify forked job runners at a
 // glance instead of a wall of generic "node" entries. Updated to the specific
 // job name once one arrives (see runJobMessage), so operators can see exactly
 // which jobs are running, how many of each, and which are eating resources.
-process.title = "velocious background-jobs-runner"
+setRunnerProcessTitle()
 
 let finishing = false
 

package/build/background-jobs/job-runner.js

@@ -6,6 +6,17 @@ import BackgroundJobsStatusReporter from "./status-reporter.js"
 
 const BEACON_READY_TIMEOUT_MS = 5000
 
+export class BackgroundJobPerformedFailure extends Error {
+  /**
+   * Creates a performed-job failure after its terminal report is acknowledged.
+   * @param {Error} cause - A job perform error whose failed terminal report was acknowledged.
+   */
+  constructor(cause) {
+    super(cause.message, {cause})
+    this.name = "BackgroundJobPerformedFailure"
+  }
+}
+
 /**
  * Runs report beacon ready error.
  * @param {import("../configuration.js").default} configuration - Configuration.
@@ -96,11 +107,12 @@ export default async function runJobPayload(payload, {closeConnections = true} =
         await perform.apply(jobInstance, payload.args || [])
       })
     } catch (error) {
+      const performedError = error instanceof Error ? error : new Error(String(error))
       if (payload.id) {
         await reporter.reportWithRetry({
           jobId: payload.id,
           status: "failed",
-          error,
+          error: performedError,
           handoffId: payload.handoffId,
           workerId: payload.workerId,
           handedOffAtMs: payload.handedOffAtMs,
@@ -108,7 +120,7 @@ export default async function runJobPayload(payload, {closeConnections = true} =
         })
       }
 
-      throw error
+      throw new BackgroundJobPerformedFailure(performedError)
     }
 
     if (payload.id) {

package/build/background-jobs/json-socket.js

@@ -26,6 +26,8 @@ export default class JsonSocket extends EventEmitter {
      * Narrows the runtime value to the documented type.
      * @type {boolean} */
     this.acceptsForkedJobs = true
+    /** @type {boolean} */
+    this.acceptsPooledJobs = false
     /**
      * Narrows the runtime value to the documented type.
      * @type {boolean} */

package/build/background-jobs/main.js

@@ -6,6 +6,7 @@ import BackgroundJobsScheduler from "./scheduler.js"
 import BackgroundJobsStore from "./store.js"
 import Logger from "../logger.js"
 import PruneTerminalBackgroundJobsJob from "../jobs/prune-terminal-background-jobs.js"
+import VelociousError from "../velocious-error.js"
 
 /**
  * Channel used by `background-jobs-main` to coordinate dispatch wake-ups
@@ -38,6 +39,12 @@ const WORKER_LIVENESS_SWEEP_MS = 15000
 const WORKER_EXECUTION_MODE_CAPABILITIES = [
   {executionMode: "inline", accepts: (worker) => worker.acceptsInlineJobs !== false},
   {executionMode: "forked", accepts: (worker) => worker.acceptsForkedJobs !== false},
+  // Pooled is opt-in: only workers that explicitly advertise `acceptsPooled`
+  // receive pooled jobs. The `=== true` (rather than `!== false`) check keeps a
+  // pre-pooled worker — which never sends the field — out of the pooled-capable
+  // set, so the main never dispatches a pooled job to a worker that cannot run
+  // one. This is the conservative half of the extended readiness protocol.
+  {executionMode: "pooled", accepts: (worker) => worker.acceptsPooledJobs === true},
   {executionMode: "spawned", accepts: (worker) => worker.acceptsSpawnedJobs !== false}
 ]
 const WORKER_EXECUTION_MODE_CAPABILITIES_BY_MODE = new Map(
@@ -80,6 +87,11 @@ export default class BackgroundJobsMain {
      * Active durable handoffs keyed by the exact worker socket that received them.
      * @type {Map<JsonSocket, Map<string, string>>} */
     this.workerHandoffs = new Map()
+    /**
+     * Handoff-adoption queries started by worker hello messages. Shutdown must
+     * wait for these before closing the configuration's database pools.
+     * @type {Set<Promise<void>>} */
+    this.inflightWorkerHandoffAdoptions = new Set()
     /**
      * Narrows the runtime value to the documented type.
      * @type {net.Server | undefined} */
@@ -208,9 +220,12 @@ export default class BackgroundJobsMain {
     this._closeWorkers()
     this._clearTimers()
     this._disconnectBeaconHandlers()
-    this.scheduler?.stop()
-
-    await this._stopBeaconAndServer()
+    await this.scheduler?.stop()
+    try {
+      await this._drainWorkerHandoffAdoptions()
+    } finally {
+      await this._stopBeaconAndServer()
+    }
   }
 
   /**
@@ -416,18 +431,45 @@ export default class BackgroundJobsMain {
     if (message?.type !== "hello") return null
 
     if (message.role === "worker") {
+      if (this._stopped) {
+        jsonSocket.close()
+        return message.role
+      }
+
       jsonSocket.workerId = message.workerId
       jsonSocket.supportsHandoffIdReporting = message.supportsHandoffIdReporting === true
       jsonSocket.supportsHeartbeat = message.supportsHeartbeat === true
       jsonSocket.lastSeenAt = Date.now()
       this.workers.add(jsonSocket)
       this.workerHandoffs.set(jsonSocket, new Map())
-      void this._adoptWorkerHandoffs(jsonSocket)
+      this._trackWorkerHandoffAdoption(jsonSocket)
     }
 
     return message.role
   }
 
+  /**
+   * Tracks a worker handoff-adoption query through shutdown.
+   * @param {JsonSocket} jsonSocket - Reconnecting worker socket.
+   * @returns {void}
+   */
+  _trackWorkerHandoffAdoption(jsonSocket) {
+    const adoption = this._adoptWorkerHandoffs(jsonSocket)
+    this.inflightWorkerHandoffAdoptions.add(adoption)
+    const removeAdoption = () => this.inflightWorkerHandoffAdoptions.delete(adoption)
+    void adoption.then(removeAdoption, removeAdoption)
+  }
+
+  /**
+   * Waits for worker handoff-adoption queries to finish.
+   * @returns {Promise<void>} - Resolves when no adoption query remains.
+   */
+  async _drainWorkerHandoffAdoptions() {
+    while (this.inflightWorkerHandoffAdoptions.size > 0) {
+      await Promise.all([...this.inflightWorkerHandoffAdoptions])
+    }
+  }
+
   /**
    * Adopts a reconnecting worker's still-active `handed_off` jobs into its new
    * socket's handoff map. A fresh main (e.g. after a deploy restart) holds no
@@ -533,6 +575,7 @@ export default class BackgroundJobsMain {
   _handleWorkerReady({jsonSocket, message}) {
     jsonSocket.acceptsSpawnedJobs = message.acceptsSpawned !== false && message.acceptsForked !== false
     jsonSocket.acceptsForkedJobs = message.acceptsForked !== false
+    jsonSocket.acceptsPooledJobs = message.acceptsPooled === true
     jsonSocket.acceptsInlineJobs = message.acceptsInline !== false
     if (jsonSocket.supportsHandoffIdReporting) {
       this.readyWorkers.add(jsonSocket)
@@ -683,7 +726,21 @@ export default class BackgroundJobsMain {
       this._notifyEnqueued()
       await this._drain()
     } catch (error) {
-      this.logger.error(() => ["Failed to enqueue background job:", error])
+      if (error instanceof VelociousError && error.safeToExpose) {
+        jsonSocket.send({type: "enqueue-error", error: error.message})
+        return
+      }
+
+      const normalizedError = error instanceof Error ? error : new Error(String(error))
+      const payload = {
+        context: {jobName: message.jobName, stage: "background-job-enqueue"},
+        error: normalizedError
+      }
+      const errorEvents = this.configuration.getErrorEvents()
+
+      this.logger.error(() => ["Failed to enqueue background job:", normalizedError])
+      errorEvents.emit("framework-error", payload)
+      errorEvents.emit("all-error", {...payload, errorType: "framework-error"})
       jsonSocket.send({type: "enqueue-error", error: "Failed to enqueue job"})
     }
   }
@@ -1110,7 +1167,7 @@ export default class BackgroundJobsMain {
     const executionModes = this.readyWorkerExecutionModes()
 
     if (executionModes.length === 0) return null
-    if (executionModes.length === 3) return await this.store.nextAvailableJob()
+    if (executionModes.length === WORKER_EXECUTION_MODE_CAPABILITIES.length) return await this.store.nextAvailableJob()
 
     return await this.store.nextAvailableJob({executionMode: executionModes})
   }

package/build/background-jobs/pooled-runner-child.js

@@ -0,0 +1,78 @@
+// @ts-check
+
+import runJobPayload, { BackgroundJobPerformedFailure } from "./job-runner.js"
+import setRunnerProcessTitle from "./runner-process-title.js"
+
+setRunnerProcessTitle()
+let running = false
+
+/**
+ * Checks whether an IPC value is a runnable pooled job message.
+ * @param {?} message - IPC message.
+ * @returns {message is {type: "job", payload: import("./types.js").BackgroundJobPayload & {id: string}}} - Whether this is a valid job message.
+ */
+function isJobMessage(message) {
+  if (!message || typeof message !== "object") return false
+  const record = /** @type {{type?: ?, payload?: ?}} */ (message)
+
+  return record.type === "job" && !!record.payload && typeof record.payload === "object" && typeof record.payload.id === "string"
+}
+
+/**
+ * Sends the terminal outcome after the main/DB report has been acknowledged or rejected.
+ * @param {object} args - Outcome.
+ * @param {string} args.jobId - Job id.
+ * @param {boolean} args.acknowledged - Whether the terminal report was acknowledged.
+ * @param {"completed" | "failed"} [args.status] - Acknowledged terminal status.
+ * @param {Error} [args.error] - Reporting error when acknowledgement was not obtained.
+ * @returns {Promise<void>} - Resolves after IPC accepts the message.
+ */
+function sendOutcome({jobId, acknowledged, status, error}) {
+  return new Promise((resolve) => {
+    if (!process.send) {
+      resolve(undefined)
+      return
+    }
+
+    process.send({
+      type: "job-outcome",
+      jobId,
+      acknowledged,
+      status,
+      rssBytes: process.memoryUsage().rss,
+      error: error?.message
+    }, () => resolve(undefined))
+  })
+}
+
+/**
+ * Handles one serial job message.
+ * @param {?} message - IPC message.
+ * @returns {Promise<void>} - Resolves after reporting.
+ */
+async function handleMessage(message) {
+  if (running || !isJobMessage(message)) return
+
+  running = true
+  try {
+    await runJobPayload(message.payload, {closeConnections: false})
+    await sendOutcome({jobId: message.payload.id, acknowledged: true, status: "completed"})
+  } catch (error) {
+    if (error instanceof BackgroundJobPerformedFailure) {
+      await sendOutcome({jobId: message.payload.id, acknowledged: true, status: "failed"})
+    } else {
+      const reportError = error instanceof Error ? error : new Error(String(error))
+      console.error("Pooled background job runner failed before terminal acknowledgement:", reportError)
+      await sendOutcome({jobId: message.payload.id, acknowledged: false, error: reportError})
+      process.exitCode = 1
+      if (process.disconnect) process.disconnect()
+      setImmediate(() => process.exit(1))
+    }
+  } finally {
+    running = false
+  }
+}
+
+process.on("message", (message) => { void handleMessage(message) })
+process.once("disconnect", () => process.exit(0))
+for (const signal of ["SIGTERM", "SIGINT"]) process.once(signal, () => process.exit(1))

package/build/background-jobs/runner-process-title.js

@@ -0,0 +1,6 @@
+// @ts-check
+
+/** Sets the idle base process title shared by child runner entrypoints. */
+export default function setRunnerProcessTitle() {
+  process.title = "velocious background-jobs-runner"
+}

package/build/background-jobs/scheduler.js

@@ -81,6 +81,8 @@ export default class BackgroundJobsScheduler {
      * Narrows the runtime value to the documented type.
      * @type {Array<ReturnType<typeof setTimeout>>} */
     this.timeoutIds = []
+    /** @type {Set<Promise<void>>} - Scheduled enqueues that shutdown must drain. */
+    this.pendingEnqueues = new Set()
     /**
      * Narrows the runtime value to the documented type.
      * @type {boolean} - True between stop() and the next start(); cron self-rescheduler checks this so a stop() during an in-flight enqueue doesn't immediately re-arm.
@@ -113,8 +115,9 @@ export default class BackgroundJobsScheduler {
 
   /**
    * Runs stop.
-   * @returns {void} */
-  stop() {
+   * @returns {Promise<void>} Resolves after in-flight scheduled enqueues finish.
+   */
+  async stop() {
     this.stopped = true
 
     for (const intervalId of this.intervalIds) {
@@ -127,6 +130,8 @@ export default class BackgroundJobsScheduler {
 
     this.intervalIds = []
     this.timeoutIds = []
+
+    await Promise.all(this.pendingEnqueues)
   }
 
   /**
@@ -176,10 +181,10 @@ export default class BackgroundJobsScheduler {
     }
 
     const timeoutId = setTimeout(() => {
-      void this.enqueueScheduledJob({jobConfiguration, jobKey})
+      void this.runScheduledJob({jobConfiguration, jobKey})
 
       const intervalId = setInterval(() => {
-        void this.enqueueScheduledJob({jobConfiguration, jobKey})
+        void this.runScheduledJob({jobConfiguration, jobKey})
       }, intervalMs)
 
       this.intervalIds.push(intervalId)
@@ -214,7 +219,7 @@ export default class BackgroundJobsScheduler {
       const timeoutId = setTimeout(async () => {
         if (this.stopped) return
 
-        await this.enqueueScheduledJob({jobConfiguration, jobKey})
+        await this.runScheduledJob({jobConfiguration, jobKey})
 
         // The await above can yield to a stop() call. Re-check before
         // re-arming so we don't keep firing after shutdown.
@@ -229,6 +234,24 @@ export default class BackgroundJobsScheduler {
     scheduleNext()
   }
 
+  /**
+   * Tracks a scheduled enqueue so stop() cannot return while it can still mutate the store.
+   * @param {object} args - Options.
+   * @param {import("../configuration-types.js").ScheduledBackgroundJobConfiguration} args.jobConfiguration - Job configuration.
+   * @param {string} args.jobKey - Job key.
+   * @returns {Promise<void>} - Resolves after the enqueue attempt finishes.
+   */
+  async runScheduledJob({jobConfiguration, jobKey}) {
+    const pendingEnqueue = this.enqueueScheduledJob({jobConfiguration, jobKey})
+    this.pendingEnqueues.add(pendingEnqueue)
+
+    try {
+      await pendingEnqueue
+    } finally {
+      this.pendingEnqueues.delete(pendingEnqueue)
+    }
+  }
+
   /**
    * Runs enqueue scheduled job.
    * @param {object} args - Options.

package/build/background-jobs/store.js

@@ -3,6 +3,7 @@
 import {randomUUID} from "crypto"
 import Logger from "../logger.js"
 import TableData from "../database/table-data/index.js"
+import VelociousError from "../velocious-error.js"
 import BackgroundJobRecord from "./job-record.js"
 import normalizeBackgroundJobError from "./normalize-error.js"
 
@@ -17,8 +18,25 @@ const ORPHANED_AFTER_MS = 2 * 60 * 60 * 1000
 /**
  * Execution modes.
  * @type {import("./types.js").BackgroundJobExecutionMode[]} */
-const EXECUTION_MODES = ["inline", "forked", "spawned"]
-const DEFAULT_EXECUTION_MODE = "forked"
+const EXECUTION_MODES = ["inline", "forked", "pooled", "spawned"]
+/**
+ * Execution mode for a new enqueue that names neither `executionMode` nor the
+ * legacy `forked` flag. Pooled routes the job to a warm, reused local runner
+ * process — the same isolation as forked without paying a fresh process per job.
+ * @type {import("./types.js").BackgroundJobExecutionMode} */
+const DEFAULT_EXECUTION_MODE = "pooled"
+/**
+ * Execution mode a legacy `forked = true` row (persisted before the
+ * `execution_mode` column existed) backfills to. It must stay `"forked"` so an
+ * upgrade never silently reinterprets already-persisted forked jobs as pooled —
+ * distinct from {@link DEFAULT_EXECUTION_MODE}, which only governs new enqueues.
+ * @type {import("./types.js").BackgroundJobExecutionMode} */
+const LEGACY_FORKED_EXECUTION_MODE = "forked"
+// Pooled jobs persist with the legacy-safe `forked` mode so a pre-pool main can
+// deserialize and run them during a rolling upgrade. Old versions ignore the
+// nullable handoff id on queued rows, making it a backward-compatible marker.
+const POOLED_HANDOFF_ID_PREFIX = "velocious-pooled:"
+const POOLED_QUEUED_HANDOFF_ID = `${POOLED_HANDOFF_ID_PREFIX}queued`
 const DEFAULT_QUEUE = "default"
 // Queue-derived durable concurrency keys are namespaced so they can't collide
 // with explicit caller-supplied concurrencyKeys.
@@ -137,7 +155,7 @@ export default class BackgroundJobsStore {
           job_name: jobName,
           args_json: argsJson,
           forked: executionMode !== "inline",
-          execution_mode: executionMode,
+          execution_mode: executionMode === "pooled" ? LEGACY_FORKED_EXECUTION_MODE : executionMode,
           queue,
           max_retries: maxRetries,
           attempts: 0,
@@ -145,7 +163,8 @@ export default class BackgroundJobsStore {
           scheduled_at_ms: scheduledAtMs,
           created_at_ms: now,
           concurrency_key: concurrency?.concurrencyKey || null,
-          max_concurrency: concurrency?.maxConcurrency || null
+          max_concurrency: concurrency?.maxConcurrency || null,
+          handoff_id: executionMode === "pooled" ? POOLED_QUEUED_HANDOFF_ID : null
         }
       })
     })
@@ -220,11 +239,7 @@ export default class BackgroundJobsStore {
     if (typeof forked === "boolean") {
       query = query.where({forked})
     }
-    if (executionMode) {
-      const executionModes = Array.isArray(executionMode) ? executionMode : [executionMode]
-
-      query = query.where({execution_mode: executionModes})
-    }
+    if (executionMode) query = this._whereExecutionMode({db, executionMode, query})
 
     if (scheduledAtOperator === "<=") {
       const priorityOrder = this._queuePriorityOrderSql(db)
@@ -400,12 +415,14 @@ export default class BackgroundJobsStore {
     await this.ensureReady()
 
     const handedOffAtMs = Date.now()
-    const handoffId = randomUUID()
 
     return await this._withDb(async (db) => await this._transactionResult(db, async () => {
       const queuedJob = await this._getJobRowById(db, jobId)
       if (!queuedJob || queuedJob.status !== "queued") return null
       if (queuedJob.concurrencyKey && !(await this._reserveConcurrency(db, queuedJob.concurrencyKey))) return null
+      const handoffId = queuedJob.executionMode === "pooled"
+        ? `${POOLED_HANDOFF_ID_PREFIX}${randomUUID()}`
+        : randomUUID()
 
       const affectedRows = await this._updateAffectedRows(db, {
         tableName: JOBS_TABLE,
@@ -479,7 +496,7 @@ export default class BackgroundJobsStore {
           status: "queued",
           scheduled_at_ms: Date.now(),
           handed_off_at_ms: null,
-          handoff_id: null,
+          handoff_id: job.executionMode === "pooled" ? POOLED_QUEUED_HANDOFF_ID : null,
           worker_id: null
         },
         conditions: {handoff_id: handoffId, id: jobId, status: "handed_off"}
@@ -737,7 +754,7 @@ export default class BackgroundJobsStore {
     if (scheduledAtMs === undefined) return defaultScheduledAtMs
     if (Number.isSafeInteger(scheduledAtMs) && scheduledAtMs >= 0) return scheduledAtMs
 
-    throw new Error("background job scheduledAtMs must be a non-negative safe integer")
+    throw VelociousError.safe("background job scheduledAtMs must be a non-negative safe integer")
   }
 
   async _ensureSchema() {
@@ -989,7 +1006,7 @@ export default class BackgroundJobsStore {
       const executionModeColumnSql = db.quoteColumn("execution_mode")
 
       await db.query(
-        `UPDATE ${tableNameSql} SET ${executionModeColumnSql} = ${db.quote(DEFAULT_EXECUTION_MODE)} ` +
+        `UPDATE ${tableNameSql} SET ${executionModeColumnSql} = ${db.quote(LEGACY_FORKED_EXECUTION_MODE)} ` +
         `WHERE ${forkedColumnSql} = ${db.quote(true)} AND ${executionModeColumnSql} IS NULL`
       )
       await db.query(
@@ -1080,6 +1097,7 @@ export default class BackgroundJobsStore {
       scheduledAt,
       shouldRetry
     })
+    if (shouldRetry && job.executionMode === "pooled") update.handoff_id = POOLED_QUEUED_HANDOFF_ID
 
     const affectedRows = await this._updateAffectedRows(db, {
       tableName: JOBS_TABLE,
@@ -1188,8 +1206,10 @@ export default class BackgroundJobsStore {
    * @returns {import("./types.js").BackgroundJobRow} - Normalized job row.
    */
   _normalizeJobRow(row) {
-    const executionMode = row.execution_mode
-      ? this._normalizeExecutionModeName(String(row.execution_mode))
+    const persistedExecutionMode = row.execution_mode ? String(row.execution_mode) : null
+    const handoffId = row.handoff_id ? String(row.handoff_id) : null
+    const executionMode = persistedExecutionMode
+      ? this._normalizePersistedExecutionMode({executionMode: persistedExecutionMode, handoffId})
       : this._normalizeExecutionMode({forked: this._normalizeBoolean(row.forked)})
 
     return {
@@ -1205,7 +1225,7 @@ export default class BackgroundJobsStore {
       scheduledAtMs: this._normalizeNumber(row.scheduled_at_ms),
       createdAtMs: this._normalizeNumber(row.created_at_ms),
       handedOffAtMs: this._normalizeNumber(row.handed_off_at_ms),
-      handoffId: row.handoff_id ? String(row.handoff_id) : null,
+      handoffId,
       completedAtMs: this._normalizeNumber(row.completed_at_ms),
       failedAtMs: this._normalizeNumber(row.failed_at_ms),
       orphanedAtMs: this._normalizeNumber(row.orphaned_at_ms),
@@ -1505,7 +1525,11 @@ export default class BackgroundJobsStore {
     if (executionMode) {
       return this._normalizeExecutionModeName(executionMode)
     }
+    // The legacy `forked` flag stays authoritative when supplied: `false` is
+    // inline and `true` is forked (never the pooled default). Only an enqueue
+    // that names neither option falls through to the pooled default.
     if (options?.forked === false) return "inline"
+    if (options?.forked === true) return LEGACY_FORKED_EXECUTION_MODE
 
     return DEFAULT_EXECUTION_MODE
   }
@@ -1523,6 +1547,47 @@ export default class BackgroundJobsStore {
     throw new Error(`Invalid background job executionMode: ${executionMode}`)
   }
 
+  /**
+   * Normalizes a legacy-safe persisted execution mode.
+   * @param {object} args - Options.
+   * @param {string} args.executionMode - Persisted legacy execution mode.
+   * @param {string | null} args.handoffId - Persisted handoff id or pooled marker.
+   * @returns {import("./types.js").BackgroundJobExecutionMode} - Runtime execution mode.
+   */
+  _normalizePersistedExecutionMode({executionMode, handoffId}) {
+    if (executionMode === LEGACY_FORKED_EXECUTION_MODE && handoffId?.startsWith(POOLED_HANDOFF_ID_PREFIX)) return "pooled"
+
+    return this._normalizeExecutionModeName(executionMode)
+  }
+
+  /**
+   * Filters persisted legacy-safe execution modes.
+   * @param {object} args - Options.
+   * @param {import("../database/drivers/base.js").default} args.db - Database connection.
+   * @param {import("./types.js").BackgroundJobExecutionMode | import("./types.js").BackgroundJobExecutionMode[]} args.executionMode - Runtime modes.
+   * @param {import("../database/query/index.js").default} args.query - Query to filter.
+   * @returns {import("../database/query/index.js").default} - Filtered query.
+   */
+  _whereExecutionMode({db, executionMode, query}) {
+    const executionModes = Array.isArray(executionMode) ? executionMode : [executionMode]
+    /** @type {string[]} */
+    const conditions = []
+    const executionModeColumn = db.quoteColumn("execution_mode")
+    const handoffIdColumn = db.quoteColumn("handoff_id")
+
+    for (const mode of executionModes) {
+      if (mode === "pooled") {
+        conditions.push(`(${executionModeColumn} = ${db.quote(LEGACY_FORKED_EXECUTION_MODE)} AND ${handoffIdColumn} = ${db.quote(POOLED_QUEUED_HANDOFF_ID)})`)
+      } else if (mode === LEGACY_FORKED_EXECUTION_MODE) {
+        conditions.push(`(${executionModeColumn} = ${db.quote(mode)} AND (${handoffIdColumn} IS NULL OR ${handoffIdColumn} <> ${db.quote(POOLED_QUEUED_HANDOFF_ID)}))`)
+      } else {
+        conditions.push(`${executionModeColumn} = ${db.quote(mode)}`)
+      }
+    }
+
+    return query.where(`(${conditions.join(" OR ")})`)
+  }
+
   /**
    * Runs parse args.
    * @param {?} value - Input value.

package/build/background-jobs/types.js

@@ -1,7 +1,7 @@
 // @ts-check
 
 /**
- * @typedef {"inline" | "forked" | "spawned"} BackgroundJobExecutionMode
+ * @typedef {"inline" | "forked" | "pooled" | "spawned"} BackgroundJobExecutionMode
  */
 /**
  * @typedef {object} BackgroundJobHandoff
@@ -10,8 +10,8 @@
  */
 /**
  * @typedef {object} BackgroundJobOptions
- * @property {BackgroundJobExecutionMode} [executionMode] - How the job should run. Defaults to `"forked"`.
- * @property {boolean} [forked] - Compatibility alias: `false` maps to `"inline"` and `true` maps to `"forked"`.
+ * @property {BackgroundJobExecutionMode} [executionMode] - How the job should run. Defaults to `"pooled"` (a warm, reused local runner process). `"forked"` runs the job in a fresh `child_process.fork()` child, `"spawned"` in a detached CLI runner, and `"inline"` inside the worker process.
+ * @property {boolean} [forked] - Compatibility alias: `false` maps to `"inline"` and `true` maps to `"forked"`. Omitting both `forked` and `executionMode` uses the default `"pooled"` mode.
  * @property {number} [maxRetries] - Max retries for a failed job before it is marked failed.
  * @property {string} [queue] - Queue name. Defaults to `"default"`. When the queue has a configured cap in `backgroundJobs.queues`, that cap is enforced cluster-wide.
  * @property {string} [concurrencyKey] - Opaque non-empty key used to share a concurrency cap. Overrides any queue-derived cap.
@@ -67,8 +67,8 @@
  * @typedef {"worker" | "client" | "reporter"} BackgroundJobSocketRole
  */
 /**
- * @typedef {{type: "hello", role: BackgroundJobSocketRole, supportsHandoffIdReporting?: boolean, supportsHeartbeat?: boolean, workerId?: string}} BackgroundJobHelloMessage
- * @typedef {{type: "ready", acceptsForked?: boolean, acceptsInline?: boolean, acceptsSpawned?: boolean}} BackgroundJobReadyMessage
+ * @typedef {{type: "hello", role: BackgroundJobSocketRole, supportsHandoffIdReporting?: boolean, supportsHeartbeat?: boolean, supportsPooled?: boolean, workerId?: string}} BackgroundJobHelloMessage
+ * @typedef {{type: "ready", acceptsForked?: boolean, acceptsInline?: boolean, acceptsPooled?: boolean, acceptsSpawned?: boolean}} BackgroundJobReadyMessage
  * @typedef {{type: "draining"}} BackgroundJobDrainingMessage
  * @typedef {{type: "heartbeat", workerId?: string}} BackgroundJobHeartbeatMessage
  * @typedef {{type: "enqueue", jobName: string, args?: Array<?>, options?: BackgroundJobOptions}} BackgroundJobEnqueueMessage