velocious 1.0.676 → 1.0.678

package/README.md

@@ -2502,7 +2502,7 @@ creation across generation processes.
 
 Jobs can opt into cross-worker durable concurrency limits by pairing a non-empty `concurrencyKey` with a positive-integer `maxConcurrency` in their background-job options, or by deriving the key in a hydrated job instance's non-static `concurrencyKey()` method. Explicit enqueue options win. The first cap registered for a key is stable; conflicting caps are rejected. See [durable concurrency limits](docs/background-jobs.md#durable-concurrency-limits).
 
-Production apps can listen for `background-job-failed` (or its `all-error` mirror) to report accepted failed attempts, including retry and terminal-state metadata. Process-level pooled-runner failures also carry one shared `context.runnerFailure` snapshot for every affected job, with active handoff identities, runner/worker lifecycle and PIDs, exit code/signal, termination reason, and an explicit nullable OOM verdict. Listen for `background-job-orphaned` to react to a specific job the main process reclaimed after its worker died mid-run — e.g. enqueue a targeted recovery for the work it left behind, instead of only polling for the aftermath. Orphan handlers run before the sweep waits for reclaimed jobs to be dispatched, so a stalled dispatcher does not delay application recovery. See [docs/background-jobs.md](docs/background-jobs.md#failure-events).
+Production apps can listen for `background-job-failed` (or its `all-error` mirror) to report accepted failed attempts, including retry and terminal-state metadata. Process-level pooled-runner failures also carry one shared `context.runnerFailure` snapshot for every affected job, with active handoff identities, child instance/PID, runner/worker lifecycle, exit code/signal, an exact typed shutdown reason and request/observation timestamps, bounded in-flight job IDs, and an explicit nullable OOM verdict. The deprecated `terminationReason` compatibility category remains present, while new consumers should use authoritative `shutdownReason`. Parent timeout/stop/retirement requests are recorded before signalling; normal drained retirement is permitted only with zero tracked jobs. Listen for `background-job-orphaned` to react to a specific job the main process reclaimed after its worker died mid-run — e.g. enqueue a targeted recovery for the work it left behind, instead of only polling for the aftermath. Orphan handlers run before the sweep waits for reclaimed jobs to be dispatched, so a stalled dispatcher does not delay application recovery. An orphan without a delivered runner-failure report has unknown shutdown cause; do not infer OOM, deploy, timeout, signal, or normal retirement from acceptance evidence alone. See [docs/background-jobs.md](docs/background-jobs.md#failure-events).
 
 ## Setup
 
@@ -2641,7 +2641,7 @@ actual state sources must still agree.
 
 `maxConcurrentInlineJobs` (default: `4`) caps how many `executionMode: "inline"` jobs a single `background-jobs-worker` process runs in parallel. Concurrency is at the JS event-loop level: every job in flight shares the worker's process and DB connection pool, so the cap should fit the pool, not the CPU count. Forking remains the right tool when you need memory isolation across long-running jobs or want to use more cores; select it with `executionMode: "forked"`.
 
-New jobs default to `executionMode: "pooled"`: a worker runs them in warm, reusable Node child runners. `pooledRunnerCount` (default: `4`) bounds this independent per-worker pool, and `pooledRunnerConcurrency` (default: `1`) sets how many jobs each child runs at once on its own event loop, so total pooled capacity is `pooledRunnerCount × pooledRunnerConcurrency` — raise concurrency for I/O-bound jobs to get high throughput from a bounded, isolated set of processes. Workers advertise that exact free-slot count and the main consumes one slot per durable handoff, allowing one readiness notification to fill the pool. If an initialized child exits unexpectedly, the worker immediately advertises the freed capacity while failure reports retry; the replacement is spawned lazily by the next dispatch, and a pre-startup crash does not trigger a respawn loop. `pooledRunnerCount`, `pooledRunnerConcurrency`, and `pooledRunnerMaxJobs` must be finite positive integers; the RSS and lifetime limits must be finite positive numbers. A child is recycled after an acknowledged job when it reaches `pooledRunnerMaxJobs` (default: `100`), `pooledRunnerMaxRssBytes` (default: `536870912`, or 512 MiB), or `pooledRunnerMaxLifetimeMs` (default: `3600000`, or one hour). `execution_mode` is the single source of truth for a job's runtime — pooled rows persist as `execution_mode = "pooled"` directly. Pooled jobs also persist acceptance evidence (`child_received_at_ms`, `child_started_at_ms`, `child_instance_id`, `child_pid`) so a handed-off job can be told apart from one whose runner never picked it up. See [execution modes and pooled runners](docs/background-jobs.md#execution-modes-and-pooled-runners).
+New jobs default to `executionMode: "pooled"`: a worker runs them in warm, reusable Node child runners. `pooledRunnerCount` (default: `4`) bounds this independent per-worker pool, and `pooledRunnerConcurrency` (default: `1`) sets how many jobs each child runs at once on its own event loop, so total pooled capacity is `pooledRunnerCount × pooledRunnerConcurrency` — raise concurrency for I/O-bound jobs to get high throughput from a bounded, isolated set of processes. Workers advertise that exact free-slot count and the main consumes one slot per durable handoff, allowing one readiness notification to fill the pool. If an initialized child exits unexpectedly, the worker immediately advertises the freed capacity while failure reports retry; the replacement is spawned lazily by the next dispatch, and a pre-startup crash does not trigger a respawn loop. `pooledRunnerCount`, `pooledRunnerConcurrency`, and `pooledRunnerMaxJobs` must be finite positive integers; the RSS and lifetime limits must be finite positive numbers. A child is recycled after an acknowledged job when it reaches `pooledRunnerMaxJobs` (default: `100`), `pooledRunnerMaxRssBytes` (default: `536870912`, or 512 MiB), or `pooledRunnerMaxLifetimeMs` (default: `3600000`, or one hour). Drained recycling records `parent_retire_drained` and refuses to signal while any job remains tracked; timeout and bounded worker-stop paths retain `job_timeout` and `worker_stop` across SIGKILL escalation. Before resource teardown the child attempts one bounded lifecycle observation over IPC. `execution_mode` is the single source of truth for a job's runtime — pooled rows persist as `execution_mode = "pooled"` directly. Pooled jobs also persist acceptance evidence (`child_received_at_ms`, `child_started_at_ms`, `child_instance_id`, `child_pid`) so a handed-off job can be told apart from one whose runner never picked it up. See [execution modes and pooled runners](docs/background-jobs.md#execution-modes-and-pooled-runners).
 
 Cold pooled jobs share one atomic model-bootstrap phase. If that phase fails, all
 waiting jobs receive the failure; a later job in the surviving child cannot run

package/build/background-jobs/pooled-runner-child.js

@@ -1,13 +1,17 @@
 // @ts-check
 
 import { randomUUID } from "node:crypto"
+import timeout from "awaitery/build/timeout.js"
 import runJobPayload, { BackgroundJobPerformedFailure } from "./job-runner.js"
+import { boundedPooledRunnerInflightJobIds, isPooledChildShutdownReason, isPooledChildShutdownSignal } from "./pooled-runner-shutdown.js"
 import { closeRunnerConnections, closeRunnerFrameworkConnections, currentConfigurationOrNull } from "./runner-graceful-shutdown.js"
 import setRunnerProcessTitle from "./runner-process-title.js"
 import PooledRunnerBrokerIdentity from "./pooled-runner-broker-identity.js"
 import { runWithSharedTransactionBrokerConfig } from "../testing/shared-transaction-proxy-driver.js"
 
 const BASE_PROCESS_TITLE = "velocious background-jobs-runner"
+/** A shutdown observation may delay resource teardown only for this bounded IPC send. */
+const SHUTDOWN_OBSERVATION_SEND_TIMEOUT_MS = 100
 /** Stable identity of this pooled child process for the life of the process. */
 const childInstanceId = randomUUID()
 
@@ -20,13 +24,18 @@ let shutdownPromise
  * Closes the runner's connections — releasing any advisory lock a killed-mid-pass
  * job still holds — before exiting, instead of leaving a half-open session that
  * keeps the lock until the DB server's `wait_timeout`.
- * @param {number} exitCode - Process exit code.
+ * @param {object} args - Shutdown observation.
+ * @param {number} args.exitCode - Process exit code.
+ * @param {import("./types.js").PooledChildShutdownReason} args.reason - Exact shutdown reason observed by the child.
+ * @param {number | null} [args.shutdownRequestedAtMs] - Parent request timestamp when supplied over IPC.
+ * @param {import("node:child_process").ChildProcess["signalCode"]} [args.signal] - Requested or observed signal.
  * @returns {Promise<void>}
  */
-function shutdownRunner(exitCode) {
+function shutdownRunner({exitCode, reason, shutdownRequestedAtMs = null, signal = null}) {
   if (shutdownPromise) return shutdownPromise
 
   shutdownPromise = (async () => {
+    await sendShutdownObservation({reason, shutdownRequestedAtMs, signal})
     await closeRunnerConnections(currentConfigurationOrNull())
     process.exit(exitCode)
   })()
@@ -60,6 +69,52 @@ function updateProcessTitle() {
   process.title = count > 0 ? `${BASE_PROCESS_TITLE}: ${count} ${count === 1 ? "job" : "jobs"}` : BASE_PROCESS_TITLE
 }
 
+/**
+ * Sends one bounded lifecycle observation before application/framework teardown.
+ * A closed IPC channel is expected for `ipc_disconnect`; other send failures are
+ * surfaced on stderr but never hold connection cleanup past the fixed bound.
+ * @param {object} args - Shutdown observation.
+ * @param {import("./types.js").PooledChildShutdownReason} args.reason - Shutdown reason.
+ * @param {number | null} args.shutdownRequestedAtMs - Parent request timestamp.
+ * @param {import("node:child_process").ChildProcess["signalCode"]} args.signal - Requested or observed signal.
+ * @returns {Promise<void>} - Resolves after IPC accepts the observation or the bound expires.
+ */
+async function sendShutdownObservation({reason, shutdownRequestedAtMs, signal}) {
+  if (!process.send || !process.connected) return
+
+  const boundedInflight = boundedPooledRunnerInflightJobIds(runningJobIds)
+  /** @type {import("./types.js").PooledChildShutdownObservation & {type: "shutdown-observation"}} */
+  const message = {
+    childInstanceId,
+    ...boundedInflight,
+    reason,
+    shutdownObservedAtMs: Date.now(),
+    shutdownRequestedAtMs,
+    signal,
+    type: "shutdown-observation"
+  }
+
+  try {
+    await timeout({timeout: SHUTDOWN_OBSERVATION_SEND_TIMEOUT_MS}, async () => {
+      await new Promise((resolve, reject) => {
+        try {
+          process.send?.(message, (error) => {
+            if (error) {
+              reject(error)
+            } else {
+              resolve(undefined)
+            }
+          })
+        } catch (error) {
+          reject(error)
+        }
+      })
+    })
+  } catch (error) {
+    console.error("Pooled background job runner could not send its shutdown observation:", error)
+  }
+}
+
 /**
  * Reports one acceptance observation (job received / perform started) to the
  * worker over IPC. The message carries the job's exact handoff lease so the
@@ -105,6 +160,22 @@ function isJobMessage(message) {
   return record.type === "job" && !!record.payload && typeof record.payload === "object" && typeof record.payload.id === "string"
 }
 
+/**
+ * Checks whether an IPC value requests a typed pooled-child shutdown.
+ * @param {ReturnType<typeof JSON.parse>} message - IPC message.
+ * @returns {message is {type: "shutdown-request", reason: import("./types.js").PooledChildShutdownReason, shutdownRequestedAtMs: number, signal: import("node:child_process").ChildProcess["signalCode"]}} - Whether this is a valid parent shutdown request.
+ */
+function isShutdownRequestMessage(message) {
+  if (!message || typeof message !== "object") return false
+  const record = /** @type {{type?: ReturnType<typeof JSON.parse>, reason?: ReturnType<typeof JSON.parse>, shutdownRequestedAtMs?: ReturnType<typeof JSON.parse>, signal?: ReturnType<typeof JSON.parse>}} */ (message)
+
+  return record.type === "shutdown-request"
+    && isPooledChildShutdownReason(record.reason)
+    && typeof record.shutdownRequestedAtMs === "number"
+    && Number.isFinite(record.shutdownRequestedAtMs)
+    && isPooledChildShutdownSignal(record.signal)
+}
+
 /**
  * Sends the terminal outcome after the main/DB report has been acknowledged or rejected.
  * @param {object} args - Outcome.
@@ -176,6 +247,16 @@ async function runJob(payload, sharedTransactionBroker) {
  * @returns {void}
  */
 function handleMessage(message) {
+  if (isShutdownRequestMessage(message)) {
+    void shutdownRunner({
+      exitCode: message.reason === "parent_retire_drained" ? 0 : 1,
+      reason: message.reason,
+      shutdownRequestedAtMs: message.shutdownRequestedAtMs,
+      signal: message.signal
+    })
+    return
+  }
+
   if (!isJobMessage(message) || runningJobIds.has(message.payload.id)) return
 
   runningJobIds.add(message.payload.id)
@@ -185,6 +266,7 @@ function handleMessage(message) {
 }
 
 process.on("message", (message) => handleMessage(message))
-process.once("disconnect", () => void shutdownRunner(0))
-for (const signal of ["SIGTERM", "SIGINT"]) process.once(signal, () => void shutdownRunner(1))
-if (process.send) process.send({type: "ready"})
+process.once("disconnect", () => void shutdownRunner({exitCode: 0, reason: "ipc_disconnect"}))
+process.once("SIGTERM", () => void shutdownRunner({exitCode: 1, reason: "signal_sigterm", signal: "SIGTERM"}))
+process.once("SIGINT", () => void shutdownRunner({exitCode: 1, reason: "signal_sigint", signal: "SIGINT"}))
+if (process.send) process.send({childInstanceId, childPid: process.pid, type: "ready"})

package/build/background-jobs/pooled-runner-shutdown.js

@@ -0,0 +1,55 @@
+// @ts-check
+
+import { constants } from "node:os"
+
+/** Maximum job ids copied into a pooled-child shutdown observation or failure snapshot. */
+export const POOLED_RUNNER_INFLIGHT_JOB_ID_LIMIT = 100
+
+/** @type {readonly import("./types.js").PooledChildShutdownReason[]} */
+export const POOLED_CHILD_SHUTDOWN_REASONS = Object.freeze([
+  "parent_retire_drained",
+  "job_timeout",
+  "worker_stop",
+  "signal_sigterm",
+  "signal_sigint",
+  "signal_sigkill",
+  "signal_other",
+  "ipc_disconnect",
+  "process_error",
+  "unexpected_exit"
+])
+
+/**
+ * Checks one dynamic IPC reason against the complete shutdown contract.
+ * @param {ReturnType<typeof JSON.parse>} value - Candidate shutdown reason.
+ * @returns {value is import("./types.js").PooledChildShutdownReason} - Whether the value is a known reason.
+ */
+export function isPooledChildShutdownReason(value) {
+  if (typeof value !== "string") return false
+
+  return POOLED_CHILD_SHUTDOWN_REASONS.some((reason) => reason === value)
+}
+
+/**
+ * Checks one dynamic IPC value against Node's named process signals.
+ * @param {ReturnType<typeof JSON.parse>} value - Candidate signal.
+ * @returns {value is import("node:child_process").ChildProcess["signalCode"]} - Whether the value is null or a named signal.
+ */
+export function isPooledChildShutdownSignal(value) {
+  return value === null || (typeof value === "string" && Object.hasOwn(constants.signals, value))
+}
+
+/**
+ * Sorts, deduplicates, and bounds job ids before they cross IPC/report boundaries.
+ * @param {Iterable<string>} jobIds - In-flight durable job ids.
+ * @returns {{inflightJobIds: string[], inflightJobIdsTruncatedCount: number}} - Bounded snapshot.
+ */
+export function boundedPooledRunnerInflightJobIds(jobIds) {
+  const sortedJobIds = [...new Set(jobIds)].sort((left, right) => left.localeCompare(right))
+  const inflightJobIds = sortedJobIds.slice(0, POOLED_RUNNER_INFLIGHT_JOB_ID_LIMIT)
+
+  return {
+    inflightJobIds,
+    inflightJobIdsTruncatedCount: sortedJobIds.length - inflightJobIds.length
+  }
+}

package/build/background-jobs/types.js

@@ -11,7 +11,11 @@
 /** @typedef {BackgroundJobActiveStatus | BackgroundJobTerminalStatus} BackgroundJobStatus */
 /** @typedef {"exit" | "process-error" | "ipc-send"} PooledRunnerFailureOrigin */
 /** @typedef {"starting" | "running" | "retiring"} PooledRunnerLifecycleState */
-/** @typedef {"unexpected" | "job-timeout" | "worker-shutdown-timeout"} PooledRunnerTerminationReason */
+/** @typedef {"parent_retire_drained" | "job_timeout" | "worker_stop" | "signal_sigterm" | "signal_sigint" | "signal_sigkill" | "signal_other" | "ipc_disconnect" | "process_error" | "unexpected_exit"} PooledChildShutdownReason */
+/**
+ * @deprecated Use PooledChildShutdownReason for exact shutdown provenance.
+ * @typedef {"unexpected" | "job-timeout" | "worker-shutdown-timeout"} PooledRunnerTerminationReason
+ */
 /** @typedef {"running" | "retiring" | "stopping"} BackgroundJobsWorkerLifecycleState */
 /**
  * Exact durable handoff ownership carried by an executing job when it produces
@@ -34,8 +38,11 @@
  * One process-failure snapshot shared by every job lost with a pooled child.
  * @typedef {object} PooledRunnerFailure
  * @property {PooledRunnerActiveJob[]} activeJobs - Jobs that were in flight when the child failed, ordered by job id.
+ * @property {string | null} childInstanceId - Stable pooled-child identity when its startup or shutdown observation arrived.
  * @property {number | null} exitCode - Child exit code, or null for signal/process errors.
  * @property {string | null} generationId - Release generation identity, or null in legacy mode.
+ * @property {string[]} inflightJobIds - Bounded job ids in flight when failure handling started.
+ * @property {number} inflightJobIdsTruncatedCount - In-flight ids omitted from the bounded snapshot.
  * @property {boolean | null} oomKilled - False when the observed exit rules OOM out; null when an unexpected SIGKILL cannot be distinguished from an OOM kill without supervisor/kernel evidence.
  * @property {PooledRunnerFailureOrigin} origin - Worker observation that initiated failure handling.
  * @property {number} runnerAgeMs - Child age when failure handling started.
@@ -45,12 +52,27 @@
  * @property {PooledRunnerLifecycleState} runnerLifecycle - Child lifecycle immediately before recovery.
  * @property {number | null} runnerPid - Child process id when available.
  * @property {import("node:child_process").ChildProcess["signalCode"]} signal - Child termination signal when available.
- * @property {PooledRunnerTerminationReason} terminationReason - Why the worker expected or did not expect termination.
+ * @property {number | null} shutdownObservedAtMs - Time the child or parent observed shutdown beginning.
+ * @property {number | null} shutdownRequestedAtMs - Exact parent request timestamp, or null for external/unrequested shutdown.
+ * @property {PooledChildShutdownReason} shutdownReason - Exact recorded parent request or observed child shutdown cause.
+ * @property {import("node:child_process").ChildProcess["signalCode"]} shutdownSignal - Parent-requested or child-observed shutdown signal when available.
+ * @property {PooledRunnerTerminationReason} terminationReason - Deprecated compatibility category; use shutdownReason for exact provenance.
  * @property {string | null} timeoutJobId - Job whose timeout initiated child termination, or null.
  * @property {string} workerId - Stable generation-qualified worker id.
  * @property {BackgroundJobsWorkerLifecycleState} workerLifecycle - Parent worker lifecycle immediately before recovery.
  * @property {number} workerPid - Parent worker process id.
  */
+/**
+ * Best-effort observation sent before a pooled child closes its resources.
+ * @typedef {object} PooledChildShutdownObservation
+ * @property {string} childInstanceId - Stable pooled-child identity.
+ * @property {string[]} inflightJobIds - Bounded in-flight durable job ids.
+ * @property {number} inflightJobIdsTruncatedCount - In-flight ids omitted from the bounded snapshot.
+ * @property {PooledChildShutdownReason} reason - Shutdown reason observed by the child.
+ * @property {number} shutdownObservedAtMs - Child observation timestamp.
+ * @property {number | null} shutdownRequestedAtMs - Parent request timestamp when supplied over IPC.
+ * @property {import("node:child_process").ChildProcess["signalCode"]} signal - Requested or observed signal when available.
+ */
 /**
  * @typedef {object} LocalBackgroundJobsClock
  * @property {() => number} now - Current epoch milliseconds.

package/build/background-jobs/worker.js

@@ -14,6 +14,7 @@ import performBackgroundJob from "./perform-job.js"
 import { runWithBackgroundJobPayload } from "./execution-context.js"
 import { createGenerationWorkerId } from "./generation-identity.js"
 import BackgroundJobsGenerationHandshakeTimeoutError, { DEFAULT_GENERATION_HANDSHAKE_TIMEOUT_MS, validateGenerationHandshakeTimeoutMs } from "./generation-handshake-timeout-error.js"
+import { POOLED_RUNNER_INFLIGHT_JOB_ID_LIMIT, boundedPooledRunnerInflightJobIds, isPooledChildShutdownReason, isPooledChildShutdownSignal } from "./pooled-runner-shutdown.js"
 
 /**
  * Per-forked-child timeout bookkeeping.
@@ -33,18 +34,26 @@ import BackgroundJobsGenerationHandshakeTimeoutError, { DEFAULT_GENERATION_HANDS
 /**
  * @typedef {object} PooledChildState
  * @property {number} createdAtMs - Child creation timestamp.
+ * @property {string} [childInstanceId] - Stable identity reported by the child.
  * @property {number} jobsRun - Acknowledged jobs completed by this child.
  * @property {Map<string, PooledJobEntry>} inflight - Jobs currently owned by this child.
  * @property {number} lastDispatchSeq - Round-robin dispatch sequence.
  * @property {boolean} retiring - Whether this child is draining before retirement.
  * @property {boolean} [started] - Whether the child completed its startup handshake.
  * @property {boolean} [settling] - Whether failure handling already owns this child.
+ * @property {number} [ipcDisconnectedAtMs] - Parent observation of IPC disconnect.
+ * @property {import("./types.js").PooledChildShutdownObservation} [shutdownObservation] - Child observation sent before teardown.
+ * @property {import("./types.js").PooledChildShutdownReason} [shutdownReason] - Exact parent-requested shutdown reason.
+ * @property {number} [shutdownRequestedAtMs] - Exact parent shutdown-request timestamp.
+ * @property {import("node:child_process").ChildProcess["signalCode"]} [shutdownSignal] - Signal selected by the parent request.
+ * @property {ReturnType<typeof setTimeout>} [shutdownSignalTimer] - Drained-retirement fallback signal timer.
  * @property {ReturnType<typeof setTimeout> | null} [timeoutSigkillTimer] - Pending timeout SIGKILL timer.
- * @property {import("./types.js").PooledRunnerTerminationReason} [terminationReason] - Expected termination reason.
  * @property {string} [timeoutJobId] - Job whose timeout initiated termination.
  */
 /** Grace period after SIGTERM before a lingering process runner is SIGKILLed. */
 const FORKED_CHILD_SIGKILL_GRACE_MS = 5000
+/** Time a drained retirement gives the child IPC request to begin teardown before SIGTERM fallback. */
+const POOLED_RUNNER_SHUTDOWN_REQUEST_GRACE_MS = 250
 /**
  * Largest delay Node's `setTimeout` accepts without overflowing to a 1ms delay
  * (a 32-bit signed int of ms, ~24.8 days). A `jobTimeoutMs` above this — or a
@@ -100,6 +109,29 @@ function isChildAcceptanceMessage(message) {
     && (record.childPid === undefined || Number.isInteger(record.childPid))
 }
 
+/**
+ * Checks whether an IPC value is the child's bounded pre-teardown observation.
+ * @param {ReturnType<typeof JSON.parse>} message - IPC message.
+ * @returns {message is import("./types.js").PooledChildShutdownObservation & {type: "shutdown-observation"}} - Whether this is a valid shutdown observation.
+ */
+function isChildShutdownObservationMessage(message) {
+  if (!message || typeof message !== "object") return false
+  const record = /** @type {{childInstanceId?: ReturnType<typeof JSON.parse>, inflightJobIds?: ReturnType<typeof JSON.parse>, inflightJobIdsTruncatedCount?: ReturnType<typeof JSON.parse>, reason?: ReturnType<typeof JSON.parse>, shutdownObservedAtMs?: ReturnType<typeof JSON.parse>, shutdownRequestedAtMs?: ReturnType<typeof JSON.parse>, signal?: ReturnType<typeof JSON.parse>, type?: ReturnType<typeof JSON.parse>}} */ (message)
+
+  return record.type === "shutdown-observation"
+    && typeof record.childInstanceId === "string"
+    && Array.isArray(record.inflightJobIds)
+    && record.inflightJobIds.length <= POOLED_RUNNER_INFLIGHT_JOB_ID_LIMIT
+    && record.inflightJobIds.every((jobId) => typeof jobId === "string")
+    && Number.isInteger(record.inflightJobIdsTruncatedCount)
+    && record.inflightJobIdsTruncatedCount >= 0
+    && isPooledChildShutdownReason(record.reason)
+    && typeof record.shutdownObservedAtMs === "number"
+    && Number.isFinite(record.shutdownObservedAtMs)
+    && (record.shutdownRequestedAtMs === null || (typeof record.shutdownRequestedAtMs === "number" && Number.isFinite(record.shutdownRequestedAtMs)))
+    && isPooledChildShutdownSignal(record.signal)
+}
+
 /**
  * Normalizes a candidate pooled-runner resource limit.
  * @param {number | undefined} value - Candidate positive number.
@@ -561,14 +593,14 @@ export default class BackgroundJobsWorker {
 
     for (const child of this.inflightProcessChildren) {
       const pooledState = this.pooledChildStates.get(child)
-      if (pooledState && pooledState.inflight.size > 0 && !pooledState.terminationReason) {
-        pooledState.terminationReason = "worker-shutdown-timeout"
-      }
-
-      try {
-        child.kill("SIGTERM")
-      } catch {
-        // Child already exited; nothing to do.
+      if (pooledState) {
+        this._requestPooledChildShutdown({child, reason: "worker_stop", signal: "SIGTERM"})
+      } else {
+        try {
+          child.kill("SIGTERM")
+        } catch {
+          // Child already exited; nothing to do.
+        }
       }
     }
 
@@ -1185,16 +1217,10 @@ export default class BackgroundJobsWorker {
     const state = this.pooledChildStates.get(child)
 
     // Already settling/gone, or the job finished in the race with this timer.
-    if (!state || state.settling || state.terminationReason || !state.inflight.has(jobId)) return
+    if (!state || state.settling || state.shutdownReason || !state.inflight.has(jobId)) return
 
-    state.terminationReason = "job-timeout"
     state.timeoutJobId = jobId
-
-    try {
-      child.kill("SIGTERM")
-    } catch {
-      // Child already exited; nothing to do.
-    }
+    this._requestPooledChildShutdown({child, reason: "job_timeout", signal: "SIGTERM"})
 
     state.timeoutSigkillTimer = setTimeout(() => {
       try {
@@ -1234,6 +1260,10 @@ export default class BackgroundJobsWorker {
       origin: "process-error",
       signal: child.signalCode
     }))
+    child.once("disconnect", () => {
+      const state = this.pooledChildStates.get(child)
+      if (state) state.ipcDisconnectedAtMs ??= Date.now()
+    })
     return child
   }
 
@@ -1247,10 +1277,24 @@ export default class BackgroundJobsWorker {
    */
   _handlePooledChildMessage({child, message}) {
     if (!message || typeof message !== "object") return
-    const record = /** @type {{type?: ReturnType<typeof JSON.parse>, jobId?: ReturnType<typeof JSON.parse>, acknowledged?: ReturnType<typeof JSON.parse>, rssBytes?: ReturnType<typeof JSON.parse>, error?: ReturnType<typeof JSON.parse>}} */ (message)
+    const record = /** @type {{type?: ReturnType<typeof JSON.parse>, childInstanceId?: ReturnType<typeof JSON.parse>, jobId?: ReturnType<typeof JSON.parse>, acknowledged?: ReturnType<typeof JSON.parse>, rssBytes?: ReturnType<typeof JSON.parse>, error?: ReturnType<typeof JSON.parse>}} */ (message)
     const state = this.pooledChildStates.get(child)
     if (record.type === "ready") {
-      if (state) state.started = true
+      if (state) {
+        state.started = true
+        if (typeof record.childInstanceId === "string") state.childInstanceId = record.childInstanceId
+      }
+      return
+    }
+    if (isChildShutdownObservationMessage(message)) {
+      if (state) {
+        state.childInstanceId = message.childInstanceId
+        state.shutdownObservation = message
+        if (state.shutdownSignalTimer) {
+          clearTimeout(state.shutdownSignalTimer)
+          state.shutdownSignalTimer = undefined
+        }
+      }
       return
     }
     if (isChildAcceptanceMessage(message)) {
@@ -1354,10 +1398,78 @@ export default class BackgroundJobsWorker {
    * @returns {void}
    */
   _retirePooledChild(child) {
+    const state = this.pooledChildStates.get(child)
+    if (!state) throw new Error("Cannot retire pooled child without tracked state")
+    if (state.inflight.size > 0) {
+      throw new Error(`Cannot retire pooled child while ${state.inflight.size} ${state.inflight.size === 1 ? "job remains" : "jobs remain"} in flight`)
+    }
+
     this.pooledChildren.delete(child)
-    this.pooledChildStates.delete(child)
-    this.inflightProcessChildren.delete(child)
-    child.kill("SIGTERM")
+    state.retiring = true
+    this._requestPooledChildShutdown({child, reason: "parent_retire_drained", signal: "SIGTERM", waitForObservation: true})
+  }
+
+  /**
+   * Records an exact parent request before IPC or signal delivery. Drained
+   * retirement gets a brief IPC-first grace so its zero-job observation is
+   * deterministic; timeout/worker-stop paths signal immediately.
+   * @param {object} args - Shutdown request.
+   * @param {import("node:child_process").ChildProcess} args.child - Pooled child.
+   * @param {import("./types.js").PooledChildShutdownReason} args.reason - Exact parent reason.
+   * @param {keyof typeof import("node:os").constants.signals} args.signal - Signal to deliver.
+   * @param {boolean} [args.waitForObservation] - Whether IPC observation may precede signal fallback.
+   * @returns {void}
+   */
+  _requestPooledChildShutdown({child, reason, signal, waitForObservation = false}) {
+    const state = this.pooledChildStates.get(child)
+    if (!state || state.settling || state.shutdownReason) return
+
+    const shutdownRequestedAtMs = Date.now()
+    state.shutdownReason = reason
+    state.shutdownRequestedAtMs = shutdownRequestedAtMs
+    state.shutdownSignal = signal
+
+    let requestSent = false
+    if (child.connected) {
+      try {
+        child.send({type: "shutdown-request", reason, shutdownRequestedAtMs, signal}, (error) => {
+          if (!error || !waitForObservation || state.settling) return
+
+          if (state.shutdownSignalTimer) {
+            clearTimeout(state.shutdownSignalTimer)
+            state.shutdownSignalTimer = undefined
+          }
+          this._signalPooledChild({child, signal})
+        })
+        requestSent = true
+      } catch {
+        // The signal below remains the bounded shutdown mechanism.
+      }
+    }
+
+    if (waitForObservation && requestSent) {
+      state.shutdownSignalTimer = setTimeout(() => {
+        state.shutdownSignalTimer = undefined
+        this._signalPooledChild({child, signal})
+      }, POOLED_RUNNER_SHUTDOWN_REQUEST_GRACE_MS)
+      state.shutdownSignalTimer.unref()
+      return
+    }
+
+    this._signalPooledChild({child, signal})
+  }
+
+  /**
+   * Delivers one parent-owned process signal without changing recorded provenance.
+   * @param {{child: import("node:child_process").ChildProcess, signal: keyof typeof import("node:os").constants.signals}} args - Signal request.
+   * @returns {void}
+   */
+  _signalPooledChild({child, signal}) {
+    try {
+      child.kill(signal)
+    } catch {
+      // Child already exited; its exit/error handler owns state settlement.
+    }
   }
 
   /**
@@ -1384,6 +1496,7 @@ export default class BackgroundJobsWorker {
       // Cancel this child's pending timers before its in-flight set is reported —
       // the SIGKILL grace from a timeout kill, and every armed per-job backstop.
       if (state.timeoutSigkillTimer) clearTimeout(state.timeoutSigkillTimer)
+      if (state.shutdownSignalTimer) clearTimeout(state.shutdownSignalTimer)
       for (const inflightEntry of state.inflight.values()) {
         if (inflightEntry.timeoutTimer) clearTimeout(inflightEntry.timeoutTimer)
       }
@@ -1415,19 +1528,24 @@ export default class BackgroundJobsWorker {
     // observe a replacement slot before the failed jobs' reports are in flight.
     // The report promises remain tracked below; a slow retry must not hold the
     // newly freed runner capacity hostage.
-    if (state && state.started !== false) {
-      this._sendReadyIfRunning()
-    } else if (state) {
-      for (const entry of entries) {
-        if (entry.pooledJob) this._pooledStartupFailureJobs.add(entry.pooledJob)
-        const queueTracker = this.pooledJobQueueTrackers.get(entry.payload.id)
-        if (queueTracker) this._pooledStartupFailureJobs.add(queueTracker)
+    // A drained retirement already advertised its replacement capacity when
+    // the final job completed. Re-advertising here can restore a credit main
+    // consumed before its handoff reached the replacement child.
+    if (state?.shutdownReason !== "parent_retire_drained") {
+      if (state && state.started !== false) {
+        this._sendReadyIfRunning()
+      } else if (state) {
+        for (const entry of entries) {
+          if (entry.pooledJob) this._pooledStartupFailureJobs.add(entry.pooledJob)
+          const queueTracker = this.pooledJobQueueTrackers.get(entry.payload.id)
+          if (queueTracker) this._pooledStartupFailureJobs.add(queueTracker)
+        }
+        // A previous ready message may still have unconsumed pooled credits at the
+        // main. Revoke them authoritatively without suppressing valid inline or
+        // process-runner readiness; otherwise queued jobs can trigger a startup
+        // crash loop using the stale credits.
+        this._sendReadyIfRunning({revokePooledAdmission: true})
       }
-      // A previous ready message may still have unconsumed pooled credits at the
-      // main. Revoke them authoritatively without suppressing valid inline or
-      // process-runner readiness; otherwise queued jobs can trigger a startup
-      // crash loop using the stale credits.
-      this._sendReadyIfRunning({revokePooledAdmission: true})
     }
 
     await Promise.allSettled(failureReports)
@@ -1444,9 +1562,31 @@ export default class BackgroundJobsWorker {
    * @returns {import("./types.js").PooledRunnerFailure} - Shared failure provenance.
    */
   _pooledRunnerFailure({child, exitCode, origin, signal, state}) {
-    const terminationReason = state.terminationReason ?? "unexpected"
+    const observation = state.shutdownObservation
+    let shutdownReason = state.shutdownReason ?? observation?.reason
+    if (!shutdownReason) {
+      if (origin === "process-error" || origin === "ipc-send") {
+        shutdownReason = "process_error"
+      } else if (signal === "SIGTERM") {
+        shutdownReason = "signal_sigterm"
+      } else if (signal === "SIGINT") {
+        shutdownReason = "signal_sigint"
+      } else if (signal === "SIGKILL") {
+        shutdownReason = "signal_sigkill"
+      } else if (signal) {
+        shutdownReason = "signal_other"
+      } else if (state.ipcDisconnectedAtMs && exitCode === 0) {
+        shutdownReason = "ipc_disconnect"
+      } else {
+        shutdownReason = "unexpected_exit"
+      }
+    }
+    const terminationReason = shutdownReason === "job_timeout"
+      ? "job-timeout"
+      : shutdownReason === "worker_stop" ? "worker-shutdown-timeout" : "unexpected"
     const workerLifecycle = this.shouldStop ? "stopping" : this.isRetiring ? "retiring" : "running"
     const runnerLifecycle = state.started === false ? "starting" : state.retiring ? "retiring" : "running"
+    const boundedInflight = boundedPooledRunnerInflightJobIds(state.inflight.keys())
     const activeJobs = [...state.inflight.values()]
       .map((entry) => ({
         handoffId: entry.payload.handoffId ?? null,
@@ -1459,9 +1599,11 @@ export default class BackgroundJobsWorker {
 
     return Object.freeze({
       activeJobs,
+      childInstanceId: state.childInstanceId ?? observation?.childInstanceId ?? null,
       exitCode,
       generationId: this.generationId ?? null,
-      oomKilled: signal === "SIGKILL" && terminationReason === "unexpected" ? null : false,
+      ...boundedInflight,
+      oomKilled: signal === "SIGKILL" && shutdownReason === "signal_sigkill" ? null : false,
       origin,
       runnerAgeMs: Math.max(0, Date.now() - state.createdAtMs),
       runnerCreatedAtMs: state.createdAtMs,
@@ -1470,6 +1612,10 @@ export default class BackgroundJobsWorker {
       runnerLifecycle,
       runnerPid: child.pid ?? null,
       signal,
+      shutdownObservedAtMs: observation?.shutdownObservedAtMs ?? state.ipcDisconnectedAtMs ?? null,
+      shutdownRequestedAtMs: state.shutdownRequestedAtMs ?? observation?.shutdownRequestedAtMs ?? null,
+      shutdownReason,
+      shutdownSignal: state.shutdownSignal ?? observation?.signal ?? signal,
       terminationReason,
       timeoutJobId: state.timeoutJobId ?? null,
       workerId: this.workerId,