velocious 1.0.529 → 1.0.532

package/README.md

@@ -1977,6 +1977,7 @@ export default new Configuration({
     maxConcurrentForkedJobs: 4,
     maxConcurrentInlineJobs: 4,
     pooledRunnerCount: 4,
+    pooledRunnerConcurrency: 1,
     pooledRunnerMaxJobs: 100,
     pooledRunnerMaxRssBytes: 536870912,
     pooledRunnerMaxLifetimeMs: 3600000,
@@ -2008,11 +2009,11 @@ VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS=5400000
 
 `maxConcurrentInlineJobs` (default: `4`) caps how many `executionMode: "inline"` jobs a single `background-jobs-worker` process runs in parallel. Concurrency is at the JS event-loop level: every job in flight shares the worker's process and DB connection pool, so the cap should fit the pool, not the CPU count. Forking remains the right tool when you need memory isolation across long-running jobs or want to use more cores. The older `forked: false` option still maps to inline mode.
 
-New jobs default to `executionMode: "pooled"`: a worker runs them serially in warm, reusable Node child runners. `pooledRunnerCount` (default: `4`) bounds this independent per-worker capacity. It and `pooledRunnerMaxJobs` must be finite positive integers; the RSS and lifetime limits must be finite positive numbers. A child is recycled after an acknowledged job when it reaches `pooledRunnerMaxJobs` (default: `100`), `pooledRunnerMaxRssBytes` (default: `536870912`, or 512 MiB), or `pooledRunnerMaxLifetimeMs` (default: `3600000`, or one hour). Pooled rows remain readable by older mains during rolling upgrades and run there with legacy one-shot fork behavior. `forked: true` remains an explicit alias for `"forked"`; `forked: false` remains inline. See [execution modes and pooled runners](docs/background-jobs.md#execution-modes-and-pooled-runners).
+New jobs default to `executionMode: "pooled"`: a worker runs them in warm, reusable Node child runners. `pooledRunnerCount` (default: `4`) bounds this independent per-worker pool, and `pooledRunnerConcurrency` (default: `1`) sets how many jobs each child runs at once on its own event loop, so total pooled capacity is `pooledRunnerCount × pooledRunnerConcurrency` — raise concurrency for I/O-bound jobs to get high throughput from a bounded, isolated set of processes. `pooledRunnerCount`, `pooledRunnerConcurrency`, and `pooledRunnerMaxJobs` must be finite positive integers; the RSS and lifetime limits must be finite positive numbers. A child is recycled after an acknowledged job when it reaches `pooledRunnerMaxJobs` (default: `100`), `pooledRunnerMaxRssBytes` (default: `536870912`, or 512 MiB), or `pooledRunnerMaxLifetimeMs` (default: `3600000`, or one hour). Pooled rows remain readable by older mains during rolling upgrades and run there with legacy one-shot fork behavior. `forked: true` remains an explicit alias for `"forked"`; `forked: false` remains inline. See [execution modes and pooled runners](docs/background-jobs.md#execution-modes-and-pooled-runners).
 
 `maxConcurrentForkedJobs` (default: `4`) caps how many out-of-process `executionMode: "forked"` or `executionMode: "spawned"` jobs one worker may keep in flight. Forked jobs use `child_process.fork()` with an attached IPC channel. After the main process acknowledges their durable status report, forked and spawned one-shot runners exit without waiting for graceful Beacon/database teardown; the OS closes their process-owned resources. A missing or rejected status acknowledgement makes the runner exit as failed instead of reporting clean success. Spawned jobs use the legacy `background-jobs-runner` CLI process via `child_process.spawn()` and are only for callers that intentionally want that spawned behavior.
 
-`jobTimeoutMs` (or `VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS`, milliseconds; default: disabled) is a wall-clock backstop for a single `"forked"` runner. A forked job still running after the timeout is terminated (`SIGTERM`, then `SIGKILL` after the reaping grace) and reported `failed`, so a genuinely-hung runner can't pin a worker — and its whole-app boot and DB connections — indefinitely (notably a retired-release worker draining after a deploy). It's a coarse safety net, not per-job tuning: it applies to every forked runner, so set it well above the longest legitimate forked job. Omit it, or set `null`/`<= 0`, to disable. `"inline"` jobs are not covered — they share the worker's process and can't be killed without killing the worker. See [docs/background-jobs.md](docs/background-jobs.md#forked-job-timeout-hung-runner-backstop).
+`jobTimeoutMs` (or `VELOCIOUS_BACKGROUND_JOBS_JOB_TIMEOUT_MS`, milliseconds; default: disabled) is a wall-clock backstop for `"forked"` and `"pooled"` jobs. A job still running after the timeout is terminated (`SIGTERM`, then `SIGKILL` after the reaping grace) and reported `failed`, so a genuinely-hung job can't pin a worker's capacity — and its whole-app boot and DB connections — indefinitely (notably a retired-release worker draining after a deploy). For a **pooled** job the whole child running it is killed, so its concurrent in-flight siblings on that child are also reported `failed` and requeued — a hung JS job can't be cancelled any other way — before a replacement child is spawned. It's a coarse safety net, not per-job tuning: it applies to every forked and pooled job, so set it well above the longest legitimate job. Omit it, or set `null`/`<= 0`, to disable. `"inline"` jobs are not covered — they share the worker's process and can't be killed without killing the worker. See [docs/background-jobs.md](docs/background-jobs.md#job-timeout-hung-runner-backstop).
 
 ### Dispatch strategy
 
@@ -2326,4 +2327,6 @@ npx velocious db:tenants:migrate projectTenant --parallel 20
 
 At runtime, the apartment-style `Tenant` façade (`velocious/build/src/tenants/tenant.js`) is the single entry point: `Tenant.with(tenant, callback)` / `Tenant.current()` to switch into and read a tenant context, `Tenant.each({identifier, callback, parallel?, filter?})` to run a callback within every provider-listed tenant, and `Tenant.drop({identifier, tenant})` (plus the `db:tenants:drop` CLI command) to drop a tenant's database through the provider's `dropDatabase` hook. `Tenant.with` and `Tenant.each` run their callbacks inside `ensureConnections`, so switching into a tenant establishes its database connections (global + tenant) and passes them to the callback — the tenant is immediately queryable without the caller wiring up connections (already-open connections are reused, so nesting does not double-connect). `Tenant.with` is generic over its callback's return type, so a value returned from inside the tenant context comes back to the caller with its type preserved (no cast needed). `Tenant.aggregateAcross({identifier, aggregates, keyColumns, subquery, tenants?, filter?})` runs one aggregate over the same table across many tenant databases and returns the merged result — grouping tenants by server and using a single cross-database `UNION ALL` where the driver supports two-part `` `database`.`table` `` references (MySQL/MariaDB) or one query per tenant otherwise (PostgreSQL/SQLite/MSSQL).
 
+`SchemaCloner` adds a missing auto-increment column and its separate source unique index in one schema alteration, including on MySQL/MariaDB where an auto-increment column must be keyed when it is created.
+
 See [docs/tenant-databases.md](docs/tenant-databases.md) for the full configuration and migration pattern.

package/build/background-jobs/job-runner.js

@@ -78,9 +78,10 @@ function runnerProcessTitle(JobClass, payload) {
  * @param {import("./types.js").BackgroundJobPayload} payload - Payload.
  * @param {object} [options] - Runner options.
  * @param {boolean} [options.closeConnections] - Whether to gracefully close framework connections after the job.
+ * @param {boolean} [options.manageProcessTitle] - Whether to set the per-job process title and restore it afterwards. Off for concurrent pooled runners, where interleaved snapshot/restore of the single process-wide `process.title` would corrupt it; the pooled child owns an aggregate title instead.
  * @returns {Promise<void>} - Resolves when complete.
  */
-export default async function runJobPayload(payload, {closeConnections = true} = {}) {
+export default async function runJobPayload(payload, {closeConnections = true, manageProcessTitle = true} = {}) {
   const configuration = await configurationResolver()
   configuration.setCurrent()
   await configuration.initialize({type: "background-jobs-runner"})
@@ -98,8 +99,9 @@ export default async function runJobPayload(payload, {closeConnections = true} =
 
   // Name the process after the job it is running so `ps`/`top` show what each
   // runner is doing; restored in the `finally` below when the job finishes.
+  // Skipped for concurrent pooled runners, whose child owns an aggregate title.
   const previousTitle = process.title
-  process.title = runnerProcessTitle(JobClass, payload)
+  if (manageProcessTitle) process.title = runnerProcessTitle(JobClass, payload)
 
   try {
     try {
@@ -136,7 +138,7 @@ export default async function runJobPayload(payload, {closeConnections = true} =
   } finally {
     // Restore the runner's base title so a lingering/idle runner (or a reused
     // one) doesn't misreport a finished job as still running.
-    process.title = previousTitle
+    if (manageProcessTitle) process.title = previousTitle
     if (closeConnections) {
       try {
         await configuration.disconnectBeacon()

package/build/background-jobs/pooled-runner-child.js

@@ -3,8 +3,32 @@
 import runJobPayload, { BackgroundJobPerformedFailure } from "./job-runner.js"
 import setRunnerProcessTitle from "./runner-process-title.js"
 
+const BASE_PROCESS_TITLE = "velocious background-jobs-runner"
+
 setRunnerProcessTitle()
-let running = false
+
+/**
+ * Ids of jobs currently running in this child. A pooled child runs up to
+ * `pooledRunnerConcurrency` jobs at once (the worker only dispatches within that
+ * bound); the set dedupes a redelivered job id and lets each job settle
+ * independently.
+ * @type {Set<string>}
+ */
+const runningJobIds = new Set()
+
+/**
+ * Sets an aggregate process title from the current in-flight count. A child runs
+ * jobs concurrently, so a per-job title (which `runJobPayload` would snapshot and
+ * restore around a single job) cannot represent the process — interleaved
+ * completions would leave a stale label. Recomputing from `runningJobIds.size` is
+ * concurrency-safe and honest: `ps`/`top` show how many jobs the child is running.
+ * @returns {void}
+ */
+function updateProcessTitle() {
+  const count = runningJobIds.size
+
+  process.title = count > 0 ? `${BASE_PROCESS_TITLE}: ${count} ${count === 1 ? "job" : "jobs"}` : BASE_PROCESS_TITLE
+}
 
 /**
  * Checks whether an IPC value is a runnable pooled job message.
@@ -46,33 +70,46 @@ function sendOutcome({jobId, acknowledged, status, error}) {
 }
 
 /**
- * Handles one serial job message.
- * @param {?} message - IPC message.
+ * Runs one job concurrently with any siblings and reports its own terminal
+ * outcome. A single job's unexpected failure reports that job for reclamation
+ * (`acknowledged: false`) but does NOT take down the child — its concurrent
+ * siblings keep running. Only a process-level fault (which escapes every
+ * per-job try/catch) ends the child, which the worker sees as an exit and
+ * reclaims for the whole in-flight set.
+ * @param {import("./types.js").BackgroundJobPayload & {id: string}} payload - Job payload.
  * @returns {Promise<void>} - Resolves after reporting.
  */
-async function handleMessage(message) {
-  if (running || !isJobMessage(message)) return
-
-  running = true
+async function runJob(payload) {
   try {
-    await runJobPayload(message.payload, {closeConnections: false})
-    await sendOutcome({jobId: message.payload.id, acknowledged: true, status: "completed"})
+    await runJobPayload(payload, {closeConnections: false, manageProcessTitle: false})
+    await sendOutcome({jobId: payload.id, acknowledged: true, status: "completed"})
   } catch (error) {
     if (error instanceof BackgroundJobPerformedFailure) {
-      await sendOutcome({jobId: message.payload.id, acknowledged: true, status: "failed"})
+      await sendOutcome({jobId: payload.id, acknowledged: true, status: "failed"})
     } else {
       const reportError = error instanceof Error ? error : new Error(String(error))
       console.error("Pooled background job runner failed before terminal acknowledgement:", reportError)
-      await sendOutcome({jobId: message.payload.id, acknowledged: false, error: reportError})
-      process.exitCode = 1
-      if (process.disconnect) process.disconnect()
-      setImmediate(() => process.exit(1))
+      await sendOutcome({jobId: payload.id, acknowledged: false, error: reportError})
     }
   } finally {
-    running = false
+    runningJobIds.delete(payload.id)
+    updateProcessTitle()
   }
 }
 
-process.on("message", (message) => { void handleMessage(message) })
+/**
+ * Handles a job message, starting it alongside any concurrent siblings.
+ * @param {?} message - IPC message.
+ * @returns {void}
+ */
+function handleMessage(message) {
+  if (!isJobMessage(message) || runningJobIds.has(message.payload.id)) return
+
+  runningJobIds.add(message.payload.id)
+  updateProcessTitle()
+  void runJob(message.payload)
+}
+
+process.on("message", (message) => handleMessage(message))
 process.once("disconnect", () => process.exit(0))
 for (const signal of ["SIGTERM", "SIGINT"]) process.once(signal, () => process.exit(1))

package/build/background-jobs/worker.js

@@ -66,14 +66,15 @@ export default class BackgroundJobsWorker {
    * @param {number} [args.maxConcurrentForkedJobs] - Override the process runner concurrency cap from `configuration.getBackgroundJobsConfig()`.
    * @param {number} [args.maxConcurrentInlineJobs] - Override the inline-job concurrency cap from `configuration.getBackgroundJobsConfig()`.
    * @param {number} [args.pooledRunnerCount] - Override the pooled runner count.
+   * @param {number} [args.pooledRunnerConcurrency] - Override the per-runner concurrency.
    * @param {number} [args.pooledRunnerMaxJobs] - Override the per-runner recycle job count.
    * @param {number} [args.pooledRunnerMaxRssBytes] - Override the per-runner recycle RSS limit.
    * @param {number} [args.pooledRunnerMaxLifetimeMs] - Override the per-runner recycle lifetime.
    * @param {number} [args.forkedChildSigkillGraceMs] - Override the grace period between SIGTERM and SIGKILL when reaping lingering process runners on stop.
    * @param {number} [args.heartbeatIntervalMs] - Override the liveness heartbeat interval (default 15000ms).
-   * @param {number} [args.jobTimeoutMs] - Override the forked-job wall-clock timeout from `configuration.getBackgroundJobsConfig()`. `0` disables it.
+   * @param {number} [args.jobTimeoutMs] - Override the wall-clock timeout for forked and pooled jobs from `configuration.getBackgroundJobsConfig()`. `0` disables it.
    */
-  constructor({configuration, host, port, maxConcurrentForkedJobs, maxConcurrentInlineJobs, pooledRunnerCount, pooledRunnerMaxJobs, pooledRunnerMaxRssBytes, pooledRunnerMaxLifetimeMs, forkedChildSigkillGraceMs, heartbeatIntervalMs, jobTimeoutMs} = {}) {
+  constructor({configuration, host, port, maxConcurrentForkedJobs, maxConcurrentInlineJobs, pooledRunnerCount, pooledRunnerConcurrency, pooledRunnerMaxJobs, pooledRunnerMaxRssBytes, pooledRunnerMaxLifetimeMs, forkedChildSigkillGraceMs, heartbeatIntervalMs, jobTimeoutMs} = {}) {
     /**
      * Narrows the runtime value to the documented type.
      * @type {Promise<import("../configuration.js").default>} */
@@ -110,10 +111,12 @@ export default class BackgroundJobsWorker {
      * @type {number} */
     this.maxConcurrentForkedJobs = this.maxConcurrentForkedJobsOverride || 4
     this.pooledRunnerCountOverride = positiveInteger(pooledRunnerCount)
+    this.pooledRunnerConcurrencyOverride = positiveInteger(pooledRunnerConcurrency)
     this.pooledRunnerMaxJobsOverride = positiveInteger(pooledRunnerMaxJobs)
     this.pooledRunnerMaxRssBytesOverride = positiveNumber(pooledRunnerMaxRssBytes)
     this.pooledRunnerMaxLifetimeMsOverride = positiveNumber(pooledRunnerMaxLifetimeMs)
     this.pooledRunnerCount = this.pooledRunnerCountOverride || 4
+    this.pooledRunnerConcurrency = this.pooledRunnerConcurrencyOverride || 1
     this.pooledRunnerMaxJobs = this.pooledRunnerMaxJobsOverride || 100
     this.pooledRunnerMaxRssBytes = this.pooledRunnerMaxRssBytesOverride || 512 * 1024 * 1024
     this.pooledRunnerMaxLifetimeMs = this.pooledRunnerMaxLifetimeMsOverride || 60 * 60 * 1000
@@ -126,7 +129,7 @@ export default class BackgroundJobsWorker {
       ? forkedChildSigkillGraceMs
       : FORKED_CHILD_SIGKILL_GRACE_MS
     /**
-     * Constructor override for the forked-job wall-clock timeout. When unset the
+     * Constructor override for the forked and pooled wall-clock job timeout. When unset the
      * timeout is read from `configuration.getBackgroundJobsConfig().jobTimeoutMs`
      * at fork time (default: disabled).
      * @type {number | undefined}
@@ -183,7 +186,7 @@ export default class BackgroundJobsWorker {
     this.inflightPooledJobs = new Set()
     /** @type {Set<import("node:child_process").ChildProcess>} */
     this.pooledChildren = new Set()
-    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, payload?: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void, settling?: boolean}>} */
+    /** @type {Map<import("node:child_process").ChildProcess, {createdAtMs: number, jobsRun: number, inflight: Map<string, {payload: import("./types.js").BackgroundJobPayload & {id: string}, resolve?: (value: void) => void, timeoutTimer?: ReturnType<typeof setTimeout> | null}>, retiring: boolean, settling?: boolean, timeoutSigkillTimer?: ReturnType<typeof setTimeout> | null}>} */
     this.pooledChildStates = new Map()
   }
 
@@ -210,6 +213,7 @@ export default class BackgroundJobsWorker {
     }
     const poolConfig = this.configuration.getBackgroundJobsConfig()
     if (typeof this.pooledRunnerCountOverride !== "number") this.pooledRunnerCount = poolConfig.pooledRunnerCount
+    if (typeof this.pooledRunnerConcurrencyOverride !== "number") this.pooledRunnerConcurrency = poolConfig.pooledRunnerConcurrency
     if (typeof this.pooledRunnerMaxJobsOverride !== "number") this.pooledRunnerMaxJobs = poolConfig.pooledRunnerMaxJobs
     if (typeof this.pooledRunnerMaxRssBytesOverride !== "number") this.pooledRunnerMaxRssBytes = poolConfig.pooledRunnerMaxRssBytes
     if (typeof this.pooledRunnerMaxLifetimeMsOverride !== "number") this.pooledRunnerMaxLifetimeMs = poolConfig.pooledRunnerMaxLifetimeMs
@@ -582,7 +586,7 @@ export default class BackgroundJobsWorker {
   _readyMessage() {
     const acceptsProcessJob = this.inflightProcessJobs.size < this.maxConcurrentForkedJobs
     const acceptsInline = this.inflightInlineJobs.size < this.maxConcurrentInlineJobs
-    const acceptsPooled = this.inflightPooledJobs.size < this.pooledRunnerCount
+    const acceptsPooled = this._availablePooledSlots() > 0
 
     if (!acceptsProcessJob && !acceptsInline && !acceptsPooled) return null
 
@@ -612,27 +616,112 @@ export default class BackgroundJobsWorker {
   }
 
   /**
-   * Runs a payload in an idle pooled child.
+   * Free pooled slots across the pool: open slots in non-retiring children plus
+   * the slots we could add by spawning more children up to `pooledRunnerCount`.
+   * Retiring children (draining before replacement) never contribute capacity.
+   * @returns {number} - Number of pooled jobs the worker can accept right now.
+   */
+  _availablePooledSlots() {
+    let openInExisting = 0
+    let nonRetiringChildren = 0
+
+    for (const child of this.pooledChildren) {
+      const state = this.pooledChildStates.get(child)
+      if (!state || state.retiring) continue
+      nonRetiringChildren += 1
+      openInExisting += this.pooledRunnerConcurrency - state.inflight.size
+    }
+
+    const spawnableChildren = Math.max(0, this.pooledRunnerCount - nonRetiringChildren)
+
+    return openInExisting + spawnableChildren * this.pooledRunnerConcurrency
+  }
+
+  /**
+   * Runs a payload on a pooled child with a free concurrency slot, spawning a
+   * new child when every non-retiring child is full and the pool is below
+   * `pooledRunnerCount`. Each child runs up to `pooledRunnerConcurrency` jobs at
+   * once on its own event loop.
    * @param {import("./types.js").BackgroundJobPayload & {id: string}} payload - Job payload.
    * @returns {Promise<void>} - Resolves after the durable report.
    */
   _runPooledJob(payload) {
-    let child = [...this.pooledChildren].find((candidate) => !this.pooledChildStates.get(candidate)?.payload)
+    let child
+    for (const candidate of this.pooledChildren) {
+      const state = this.pooledChildStates.get(candidate)
+      if (state && !state.retiring && state.inflight.size < this.pooledRunnerConcurrency) {
+        child = candidate
+        break
+      }
+    }
     if (!child) child = this._createPooledChild()
     const state = this.pooledChildStates.get(child)
     if (!state) throw new Error("Pooled runner state missing")
 
     return new Promise((resolve) => {
-      state.payload = payload
-      state.resolve = resolve
+      const timeoutTimer = this._armPooledJobTimeout({child, jobId: payload.id})
+
+      state.inflight.set(payload.id, {payload, resolve, timeoutTimer})
       try {
         child.send({type: "job", payload})
       } catch (error) {
-        this._handlePooledChildFailure({child, error})
+        void this._handlePooledChildFailure({child, error})
       }
     })
   }
 
+  /**
+   * Arms a per-job wall-clock backstop for a pooled job. A pooled child hosts many
+   * concurrent jobs, so a single genuinely-hung job would otherwise pin its
+   * runner's concurrency slot forever — the lifetime recycle only retires a child
+   * once its in-flight set drains, which a hung job never does. On overrun the
+   * whole child is terminated so the hung job (and its siblings) requeue. Returns
+   * the timer, or null when no timeout is configured.
+   * @param {object} args - Options.
+   * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
+   * @param {string} args.jobId - Job id whose overrun is guarded.
+   * @returns {ReturnType<typeof setTimeout> | null} - The armed timer, or null.
+   */
+  _armPooledJobTimeout({child, jobId}) {
+    const timeoutMs = this._resolveJobTimeoutMs()
+
+    if (!(typeof timeoutMs === "number" && timeoutMs > 0)) return null
+
+    return setTimeout(() => this._onPooledJobTimeout({child, jobId}), timeoutMs)
+  }
+
+  /**
+   * Fired when a pooled job overruns its timeout. Terminates the child running it
+   * (SIGTERM, then SIGKILL after the grace) — a hung JS job cannot be cancelled
+   * any other way. The non-clean exit flows through `_handlePooledChildFailure`,
+   * which reports every in-flight job on the child failed (so they requeue) and
+   * drops it from tracking; capacity is refilled on the next dispatch.
+   * @param {object} args - Options.
+   * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
+   * @param {string} args.jobId - Job id that overran.
+   * @returns {void}
+   */
+  _onPooledJobTimeout({child, jobId}) {
+    const state = this.pooledChildStates.get(child)
+
+    // Already settling/gone, or the job finished in the race with this timer.
+    if (!state || state.settling || !state.inflight.has(jobId)) return
+
+    try {
+      child.kill("SIGTERM")
+    } catch {
+      // Child already exited; nothing to do.
+    }
+
+    state.timeoutSigkillTimer = setTimeout(() => {
+      try {
+        child.kill("SIGKILL")
+      } catch {
+        // Child already exited; nothing to do.
+      }
+    }, this.forkedChildSigkillGraceMs)
+  }
+
   /**
    * Creates a reusable pooled child.
    * @returns {import("node:child_process").ChildProcess} - New pooled child.
@@ -647,7 +736,7 @@ export default class BackgroundJobsWorker {
     })
     this.pooledChildren.add(child)
     this.inflightProcessChildren.add(child)
-    this.pooledChildStates.set(child, {createdAtMs: Date.now(), jobsRun: 0})
+    this.pooledChildStates.set(child, {createdAtMs: Date.now(), jobsRun: 0, inflight: new Map(), retiring: false})
     child.on("message", (message) => this._handlePooledChildMessage({child, message}))
     child.once("exit", (code, signal) => this._handlePooledChildFailure({child, error: new Error(`Pooled background job runner exited: code=${code} signal=${signal || "none"}`)}))
     child.once("error", (error) => this._handlePooledChildFailure({child, error}))
@@ -655,7 +744,8 @@ export default class BackgroundJobsWorker {
   }
 
   /**
-   * Handles a pooled child's durable-report acknowledgement.
+   * Handles a pooled child's per-job durable-report acknowledgement. A child
+   * runs jobs concurrently and reports one `job-outcome` per job id.
    * @param {object} args - Message details.
    * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
    * @param {?} args.message - IPC message.
@@ -665,23 +755,71 @@ export default class BackgroundJobsWorker {
     if (!message || typeof message !== "object") return
     const record = /** @type {{type?: ?, jobId?: ?, acknowledged?: ?, rssBytes?: ?, error?: ?}} */ (message)
     const state = this.pooledChildStates.get(child)
-    if (record.type !== "job-outcome" || !state?.payload || record.jobId !== state.payload.id || state.settling) return
-    if (record.acknowledged !== true) {
-      void this._handlePooledChildFailure({child, error: new Error(typeof record.error === "string" ? record.error : "Pooled runner terminal report was not acknowledged")})
-      return
-    }
-    const resolve = state.resolve
-    state.payload = undefined
-    state.resolve = undefined
+    if (record.type !== "job-outcome" || !state || state.settling || typeof record.jobId !== "string") return
+    const entry = state.inflight.get(record.jobId)
+    if (!entry) return
+
+    if (entry.timeoutTimer) clearTimeout(entry.timeoutTimer)
+    state.inflight.delete(record.jobId)
     state.jobsRun += 1
-    if (resolve) resolve(undefined)
+    const resolve = entry.resolve
+
+    if (record.acknowledged === true) {
+      if (resolve) resolve(undefined)
+    } else {
+      // The child stayed alive but could not confirm this one job's terminal
+      // report; reclaim just this job — its concurrent siblings are unaffected.
+      void this._reportJobResult({
+        jobId: entry.payload.id,
+        status: "failed",
+        error: new Error(typeof record.error === "string" ? record.error : "Pooled runner terminal report was not acknowledged"),
+        handoffId: entry.payload.handoffId,
+        handedOffAtMs: entry.payload.handedOffAtMs,
+        workerId: entry.payload.workerId || this.workerId
+      }).finally(() => { if (resolve) resolve(undefined) })
+    }
+
     const rssBytes = typeof record.rssBytes === "number" ? record.rssBytes : Number.POSITIVE_INFINITY
     const runnerAgeMs = Date.now() - state.createdAtMs
-    if (state.jobsRun >= this.pooledRunnerMaxJobs || rssBytes >= this.pooledRunnerMaxRssBytes || runnerAgeMs >= this.pooledRunnerMaxLifetimeMs || this.shouldStop) this._retirePooledChild(child)
+    if (!state.retiring && (state.jobsRun >= this.pooledRunnerMaxJobs || rssBytes >= this.pooledRunnerMaxRssBytes || runnerAgeMs >= this.pooledRunnerMaxLifetimeMs || this.shouldStop)) {
+      this._beginRetirePooledChild(child)
+    }
+    this._terminateIfDrained(child)
+  }
+
+  /**
+   * Marks a pooled child for retirement and eagerly spawns a single replacement
+   * (1-for-1) so its capacity is restored immediately without waiting for it to
+   * finish draining. The retiring child stops receiving new jobs and is
+   * terminated only once its in-flight set drains, so a long-running job (e.g. a
+   * build) is never cut off.
+   * @param {import("node:child_process").ChildProcess} child - Child to retire.
+   * @returns {void}
+   */
+  _beginRetirePooledChild(child) {
+    const state = this.pooledChildStates.get(child)
+    if (!state || state.retiring) return
+
+    state.retiring = true
+    // Best-effort pre-warm: skip when stopping (no new work) or before the
+    // worker is initialized (no configuration to fork a child from).
+    if (!this.shouldStop && this.configuration) this._createPooledChild()
   }
 
   /**
-   * Retires an idle pooled child.
+   * Terminates a retiring pooled child once it has no in-flight jobs left.
+   * @param {import("node:child_process").ChildProcess} child - Child to check.
+   * @returns {void}
+   */
+  _terminateIfDrained(child) {
+    const state = this.pooledChildStates.get(child)
+    if (!state || !state.retiring || state.inflight.size > 0) return
+
+    this._retirePooledChild(child)
+  }
+
+  /**
+   * Retires a drained pooled child (removes it from tracking, then SIGTERMs it).
    * @param {import("node:child_process").ChildProcess} child - Child process to retire.
    * @returns {void}
    */
@@ -693,7 +831,11 @@ export default class BackgroundJobsWorker {
   }
 
   /**
-   * Removes and reports an unhealthy pooled child.
+   * Removes an exited/unhealthy pooled child and reports every job that was
+   * in-flight on it as failed — a process-level crash's blast radius is the
+   * child's whole in-flight set. Capacity is refilled lazily on the next
+   * dispatch (a spawnable slot is still advertised), avoiding a tight respawn
+   * loop when a child crashes on startup.
    * @param {object} args - Failure details.
    * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
    * @param {?} args.error - Failure.
@@ -702,24 +844,33 @@ export default class BackgroundJobsWorker {
   async _handlePooledChildFailure({child, error}) {
     const state = this.pooledChildStates.get(child)
     if (state?.settling) return
-    if (state) state.settling = true
+    if (state) {
+      state.settling = true
+      // Cancel this child's pending timers before its in-flight set is reported —
+      // the SIGKILL grace from a timeout kill, and every armed per-job backstop.
+      if (state.timeoutSigkillTimer) clearTimeout(state.timeoutSigkillTimer)
+      for (const inflightEntry of state.inflight.values()) {
+        if (inflightEntry.timeoutTimer) clearTimeout(inflightEntry.timeoutTimer)
+      }
+    }
     this.pooledChildren.delete(child)
     this.inflightProcessChildren.delete(child)
-    if (!state?.payload) {
-      this.pooledChildStates.delete(child)
-      return
-    }
-    const payload = state.payload
-    await this._reportJobResult({
-      jobId: payload.id,
-      status: "failed",
-      error,
-      handoffId: payload.handoffId,
-      handedOffAtMs: payload.handedOffAtMs,
-      workerId: payload.workerId || this.workerId
-    })
+
+    const entries = state ? [...state.inflight.values()] : []
+    if (state) state.inflight.clear()
     this.pooledChildStates.delete(child)
-    if (state.resolve) state.resolve(undefined)
+
+    await Promise.allSettled(entries.map(async (entry) => {
+      await this._reportJobResult({
+        jobId: entry.payload.id,
+        status: "failed",
+        error,
+        handoffId: entry.payload.handoffId,
+        handedOffAtMs: entry.payload.handedOffAtMs,
+        workerId: entry.payload.workerId || this.workerId
+      })
+      if (entry.resolve) entry.resolve(undefined)
+    }))
   }
 
   /**
@@ -820,7 +971,7 @@ export default class BackgroundJobsWorker {
    * @returns {ForkedJobTimeoutState} - Timeout state.
    */
   _armForkedJobTimeout({child}) {
-    const timeoutMs = this._resolveForkedJobTimeoutMs()
+    const timeoutMs = this._resolveJobTimeoutMs()
     /** @type {ForkedJobTimeoutState} */
     const state = {timedOut: false, timeoutMs, timer: null, sigkillTimer: null}
 
@@ -832,12 +983,12 @@ export default class BackgroundJobsWorker {
   }
 
   /**
-   * Resolves the effective forked-job timeout in ms, or null when disabled. The
+   * Resolves the effective wall-clock job timeout in ms (shared by forked and pooled jobs), or null when disabled. The
    * constructor override wins; otherwise the value comes from the background-jobs
    * configuration. A non-positive value disables the backstop.
    * @returns {number | null} - Timeout in ms, or null when disabled.
    */
-  _resolveForkedJobTimeoutMs() {
+  _resolveJobTimeoutMs() {
     const raw = typeof this.jobTimeoutMsOverride === "number"
       ? this.jobTimeoutMsOverride
       : (this.configuration ? this.configuration.getBackgroundJobsConfig().jobTimeoutMs : null)

package/build/configuration-types.js

@@ -165,6 +165,7 @@
  *   for workload-shaped limits use per-queue caps (`queues`) instead, which are
  *   enforced cluster-wide and are immune to duplicate worker processes.
  * @property {number} [pooledRunnerCount] - Number of warm, reusable child runners owned by each worker. Pooled capacity is separate from inline and forked/spawned capacity. Default: `4`.
+ * @property {number} [pooledRunnerConcurrency] - Number of jobs each pooled child runs concurrently on its own event loop. Total per-worker pooled capacity is `pooledRunnerCount × pooledRunnerConcurrency`. `1` (default) keeps each child serial; raise it for I/O-bound jobs so a bounded set of isolated processes handles high concurrency (like the inline lane) without one process per concurrent job. Default: `1`.
  * @property {number} [pooledRunnerMaxJobs] - Number of sequential jobs a pooled child runs before it is replaced, bounding process-level resource accumulation. Default: `100`.
  * @property {number} [pooledRunnerMaxRssBytes] - RSS bytes after an acknowledged job at which a pooled child is replaced. Default: `536870912` (512 MiB).
  * @property {number} [pooledRunnerMaxLifetimeMs] - Age after an acknowledged job at which a pooled child is replaced. Default: `3600000` (one hour).

package/build/configuration.js

@@ -192,6 +192,12 @@ export default class VelociousConfiguration {
     }
 
     this._isInitialized = false
+    /**
+     * In-progress `initialize()` promise, memoized so concurrent callers await
+     * the same bootstrap. Reset to undefined if initialization fails.
+     * @type {Promise<void> | undefined}
+     */
+    this._initializePromise = undefined
     this.httpServer = httpServer || {}
     /**
      * Stores the http server instance value.
@@ -1269,6 +1275,7 @@ export default class VelociousConfiguration {
     const envMaxConcurrentForkedRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_MAX_CONCURRENT_FORKED_JOBS
     const envMaxConcurrentRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_MAX_CONCURRENT_INLINE_JOBS
     const envPooledRunnerCountRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_COUNT
+    const envPooledRunnerConcurrencyRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_CONCURRENCY
     const envPooledRunnerMaxJobsRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_MAX_JOBS
     const envPooledRunnerMaxRssBytesRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_MAX_RSS_BYTES
     const envPooledRunnerMaxLifetimeMsRaw = process.env.VELOCIOUS_BACKGROUND_JOBS_POOLED_RUNNER_MAX_LIFETIME_MS
@@ -1279,6 +1286,7 @@ export default class VelociousConfiguration {
     const envMaxConcurrentForked = envMaxConcurrentForkedRaw ? Number(envMaxConcurrentForkedRaw) : undefined
     const envMaxConcurrent = envMaxConcurrentRaw ? Number(envMaxConcurrentRaw) : undefined
     const envPooledRunnerCount = envPooledRunnerCountRaw ? Number(envPooledRunnerCountRaw) : undefined
+    const envPooledRunnerConcurrency = envPooledRunnerConcurrencyRaw ? Number(envPooledRunnerConcurrencyRaw) : undefined
     const envPooledRunnerMaxJobs = envPooledRunnerMaxJobsRaw ? Number(envPooledRunnerMaxJobsRaw) : undefined
     const envPooledRunnerMaxRssBytes = envPooledRunnerMaxRssBytesRaw ? Number(envPooledRunnerMaxRssBytesRaw) : undefined
     const envPooledRunnerMaxLifetimeMs = envPooledRunnerMaxLifetimeMsRaw ? Number(envPooledRunnerMaxLifetimeMsRaw) : undefined
@@ -1299,6 +1307,9 @@ export default class VelociousConfiguration {
     const pooledRunnerCount = typeof configured.pooledRunnerCount === "number" && Number.isFinite(configured.pooledRunnerCount) && Number.isInteger(configured.pooledRunnerCount) && configured.pooledRunnerCount >= 1
       ? configured.pooledRunnerCount
       : (!("pooledRunnerCount" in configured) && typeof envPooledRunnerCount === "number" && Number.isFinite(envPooledRunnerCount) && Number.isInteger(envPooledRunnerCount) && envPooledRunnerCount >= 1 ? envPooledRunnerCount : 4)
+    const pooledRunnerConcurrency = typeof configured.pooledRunnerConcurrency === "number" && Number.isFinite(configured.pooledRunnerConcurrency) && Number.isInteger(configured.pooledRunnerConcurrency) && configured.pooledRunnerConcurrency >= 1
+      ? configured.pooledRunnerConcurrency
+      : (!("pooledRunnerConcurrency" in configured) && typeof envPooledRunnerConcurrency === "number" && Number.isFinite(envPooledRunnerConcurrency) && Number.isInteger(envPooledRunnerConcurrency) && envPooledRunnerConcurrency >= 1 ? envPooledRunnerConcurrency : 1)
     const pooledRunnerMaxJobs = typeof configured.pooledRunnerMaxJobs === "number" && Number.isFinite(configured.pooledRunnerMaxJobs) && Number.isInteger(configured.pooledRunnerMaxJobs) && configured.pooledRunnerMaxJobs >= 1
       ? configured.pooledRunnerMaxJobs
       : (!("pooledRunnerMaxJobs" in configured) && typeof envPooledRunnerMaxJobs === "number" && Number.isFinite(envPooledRunnerMaxJobs) && Number.isInteger(envPooledRunnerMaxJobs) && envPooledRunnerMaxJobs >= 1 ? envPooledRunnerMaxJobs : 100)
@@ -1336,7 +1347,7 @@ export default class VelociousConfiguration {
         : 60 * 60 * 1000
     }
 
-    return {host, port, databaseIdentifier, maxConcurrentForkedJobs, maxConcurrentInlineJobs, pooledRunnerCount, pooledRunnerMaxJobs, pooledRunnerMaxRssBytes, pooledRunnerMaxLifetimeMs, dispatchStrategy, pollIntervalMs, queues, jobTimeoutMs, retention}
+    return {host, port, databaseIdentifier, maxConcurrentForkedJobs, maxConcurrentInlineJobs, pooledRunnerCount, pooledRunnerConcurrency, pooledRunnerMaxJobs, pooledRunnerMaxRssBytes, pooledRunnerMaxLifetimeMs, dispatchStrategy, pollIntervalMs, queues, jobTimeoutMs, retention}
   }
 
   /**
@@ -1934,9 +1945,16 @@ export default class VelociousConfiguration {
    * @returns {Promise<void>} - Resolves when complete.
    */
   async initialize({type} = {type: "undefined"}) {
-    if (!this.isInitialized()) {
-      this._isInitialized = true
-
+    if (this._isInitialized) return
+    // Memoize the in-progress initialization so concurrent callers await the same
+    // bootstrap instead of racing. `_isInitialized` was previously set to `true`
+    // up front, so a second caller (e.g. a pooled runner with
+    // `pooledRunnerConcurrency > 1` starting several jobs on a cold child) could
+    // skip initialization and load models / perform a job while the first call
+    // was still awaiting model discovery and initializers. Mirrors connectBeacon.
+    if (this._initializePromise) return await this._initializePromise
+
+    this._initializePromise = (async () => {
       await this.initializeModels({type})
       await this.getEnvironmentHandler().autoDiscoverResources(this)
       this._mergeDiscoveredAbilityResources()
@@ -1957,6 +1975,17 @@ export default class VelociousConfiguration {
           }
         }
       }
+
+      this._isInitialized = true
+    })()
+
+    try {
+      await this._initializePromise
+    } catch (error) {
+      // Let a later call retry a failed initialization instead of every future
+      // caller awaiting the same cached rejection.
+      this._initializePromise = undefined
+      throw error
     }
   }