@bastani/atomic 0.9.61-alpha.2 → 0.9.61-alpha.3

package/CHANGELOG.md

@@ -6,9 +6,9 @@
 
 Local worker pools that hold up under faults and load, a lighter terminal under heavy workflow load, and workflows that can work in any repository's worktree.
 
-- **One failing child no longer takes down a pool:** a child that fails to dispatch fails alone, a worker or tool process group that survives SIGKILL is a warning rather than a failure, and pool faults replay the child within its three attempts ([#4159](https://github.com/bastani-inc/atomic/pull/4159))
+- **One failing child no longer takes down a pool:** a child that fails to dispatch fails alone, a worker or tool process group that survives SIGKILL is a warning rather than a failure, and pool faults replay the child within its three attempts, and a replay blocked by those groups' quarantined permits fails after a bounded wait instead of waiting forever ([#4159](https://github.com/bastani-inc/atomic/pull/4159), [#4170](https://github.com/bastani-inc/atomic/pull/4170))
 - **Local worker pools keep their leases:** a slow journal or root no longer gets every healthy worker killed at once, and a race in lease recovery no longer fails every child of a pool with `stale_work_lease` ([#4165](https://github.com/bastani-inc/atomic/pull/4165))
-- **A lighter terminal under heavy load:** a workflow run no longer keeps every completed stage's session in memory until it ends, which cut peak heap from 2.8 GB to 0.87 GB in a 600-stage test and shortened GC pauses. Rebuilding the chat no longer flashes "Loading isolated renderer…" ([#4166](https://github.com/bastani-inc/atomic/pull/4166), [#4167](https://github.com/bastani-inc/atomic/pull/4167))
+- **A lighter terminal under heavy load:** a workflow run no longer keeps every completed stage's session in memory until it ends, which cut peak heap from 2.8 GB to 0.87 GB in a 600-stage test and shortened GC pauses. Rebuilding the chat no longer flashes "Loading isolated renderer…", and rows keep the state you left them in ([#4166](https://github.com/bastani-inc/atomic/pull/4166), [#4167](https://github.com/bastani-inc/atomic/pull/4167), [#4169](https://github.com/bastani-inc/atomic/pull/4169))
 - **Broadcasts reach pooled stages at once:** stages on local workers get queued and sticky Intercom messages before their first turn, without a 60 s stall or a warning per stage ([#4164](https://github.com/bastani-inc/atomic/pull/4164))
 - **Content-filter refusals fall back:** a provider content-filter refusal, such as OpenAI's 422 `cyber_policy`, moves to the next `fallbackModels` entry instead of retrying the model that refused; turn it off with `retry.fallbackOnContentFilter` ([#4163](https://github.com/bastani-inc/atomic/pull/4163))
 - **Workflow worktrees from anywhere:** `git_worktree_dir` can name a worktree of any repository, and a non-empty folder gets its own checkout inside it ([#4162](https://github.com/bastani-inc/atomic/pull/4162))
@@ -36,6 +36,20 @@ Local worker pools that hold up under faults and load, a lighter terminal under
 - A stage on a local worker (`workflows.localWorkers`) that starts after a sticky Intercom broadcast (`workflow:<rootRunId>/**`) or a name or pattern send to future stages gets those messages before its first turn, under "Messages received before you started" with the real sender, even when the root is too busy to answer. Before, each such stage on a busy root could wait 60 seconds and then print a long "starting without its queued Intercom messages" warning in the chat, listing messages it may already have received. The worker now delivers the stage's queued messages itself and never waits on the root. If the root later fails to record or deliver them, the run's status shows one `Intercom delivery:` line for the whole run, not a warning per stage ([#4164](https://github.com/bastani-inc/atomic/pull/4164)).
 - A workflow run no longer keeps the live session of every stage it completed in memory until the whole run ends. A stage run with `ctx.task`, `ctx.chain` or `ctx.parallel` gives its session back as soon as it completes, and one run with `ctx.stage` once the workflow no longer holds its stage context; its saved transcript still opens from the graph, `/workflow attach` or the stage chat. A session hosting long runs with hundreds of completed stages held gigabytes this way, and the garbage collection that memory caused made the interactive TUI, `/tasks` and queued messages slow to respond. ([#4166](https://github.com/bastani-inc/atomic/pull/4166))
 - In the interactive TUI, rows that the engine renders, such as tool results and extension messages, keep their last frame when the chat is rebuilt (after compaction, `/reload` or switching sessions), instead of showing `Loading isolated renderer…` until a busy engine renders each one again. A rebuilt chat also no longer leaves the renderers of the rows it replaced running in the engine for the rest of the session, and engine messages reach chat rows through one subscription instead of one per row. ([#4167](https://github.com/bastani-inc/atomic/pull/4167))
+- A rebuilt chat in the interactive TUI no longer briefly shows a collapsed tool row with its expanded output when the row was collapsed while its expanded frame was still on its way from the engine. Rows the chat drops while the engine connection is backed up also stay released, instead of being rendered again in the engine by a render request that arrived after their disposal. ([#4169](https://github.com/bastani-inc/atomic/pull/4169))
+- A child on a local worker pool (`workflows.localWorkers`) whose replay needs a tool process permit no longer waits forever when the only capacity left under `admission.limits.process` (or `admission.userLimits.process`) is held by quarantined permits of process groups that survived SIGKILL. Those permits stay quarantined, and capacity held by running work is still waited for as before. A process admission that only quarantined permits block now waits at most 15 seconds for those groups to exit, then fails with the retryable `worker_cleanup_unconfirmed`, naming the groups and the limit. That ends the child's attempt, so it replays within its three attempts and then fails with `worker_retry_limit`, letting the root finish. If the groups exit in time, the replay goes on normally. ([#4170](https://github.com/bastani-inc/atomic/pull/4170))
+
+## [0.9.61-alpha.3] - 2026-11-02
+
+Replays on a local worker that can no longer wait forever, and a steadier chat renderer.
+
+- **Replays can't wait forever on cleanup:** when process groups that survived SIGKILL fill `admission.limits.process`, a child's replay now fails after a bounded wait with a retryable error naming those groups and the limit, instead of waiting indefinitely; the groups' permits stay quarantined ([#4170](https://github.com/bastani-inc/atomic/pull/4170))
+- **Chat rows stay as you left them:** a row released under engine backpressure no longer comes back, and a row you collapse no longer shows its expanded output after the chat is rebuilt ([#4169](https://github.com/bastani-inc/atomic/pull/4169))
+
+### Fixed
+
+- A rebuilt chat in the interactive TUI no longer briefly shows a collapsed tool row with its expanded output when the row was collapsed while its expanded frame was still on its way from the engine. Rows the chat drops while the engine connection is backed up also stay released, instead of being rendered again in the engine by a render request that arrived after their disposal. ([#4169](https://github.com/bastani-inc/atomic/pull/4169))
+- A child on a local worker pool (`workflows.localWorkers`) whose replay needs a tool process permit no longer waits forever when the only capacity left under `admission.limits.process` (or `admission.userLimits.process`) is held by quarantined permits of process groups that survived SIGKILL. Those permits stay quarantined, and capacity held by running work is still waited for as before. A process admission that only quarantined permits block now waits at most 15 seconds for those groups to exit, then fails with the retryable `worker_cleanup_unconfirmed`, naming the groups and the limit. That ends the child's attempt, so it replays within its three attempts and then fails with `worker_retry_limit`, letting the root finish. If the groups exit in time, the replay goes on normally. ([#4170](https://github.com/bastani-inc/atomic/pull/4170))
 
 ## [0.9.61-alpha.2] - 2026-11-02
 

package/dist/builtin/intercom/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bastani/intercom",
-  "version": "0.9.61-alpha.2",
+  "version": "0.9.61-alpha.3",
   "private": true,
   "description": "Atomic extension providing a private coordination channel between parent and child agent sessions. Fork of: https://github.com/nicobailon/pi-intercom",
   "contributors": [

package/dist/builtin/mcp/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bastani/mcp",
-  "version": "0.9.61-alpha.2",
+  "version": "0.9.61-alpha.3",
   "private": true,
   "description": "Atomic's pi-compatible MCP integration for server connections, OAuth, tools, resources, and terminal management.",
   "contributors": [

package/dist/builtin/subagents/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bastani/subagents",
-  "version": "0.9.61-alpha.2",
+  "version": "0.9.61-alpha.3",
   "private": true,
   "description": "Atomic extension for delegating tasks to subagents with parallel execution. Fork of: https://github.com/nicobailon/pi-subagents",
   "contributors": [

package/dist/builtin/web-access/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bastani/web-access",
-  "version": "0.9.61-alpha.2",
+  "version": "0.9.61-alpha.3",
   "private": true,
   "description": "Atomic extension for web search, URL fetching, GitHub repo cloning, PDF/video extraction. Fork of: https://github.com/nicobailon/pi-web-access",
   "contributors": [

package/dist/builtin/workflows/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@bastani/workflows",
-  "version": "0.9.61-alpha.2",
+  "version": "0.9.61-alpha.3",
   "private": true,
   "description": "Atomic extension for multi-stage workflow authoring and execution.",
   "contributors": [

package/dist/builtin/workflows/src/extension/index.bundle.mjs

@@ -20723,6 +20723,7 @@ class SqlResourceAdmission {
         epoch: this.options.attempt.epoch
       }) : null);
     });
+    let quarantinedSince;
     try {
       for (;; ) {
         signal?.throwIfAborted();
@@ -20732,7 +20733,9 @@ class SqlResourceAdmission {
         }
         const hostReason = this.hostReason();
         const pressure = this.stageStartPressure(kind, request?.delegated);
+        let quarantined = undefined;
         const granted = await this.storage.transaction(async (tx) => {
+          quarantined = undefined;
           await tx.run(`UPDATE ${this.owners} SET expires_at = CAST(unixepoch('subsec') * 1000 AS INTEGER) + ? WHERE id = ?`, OWNER_LEASE_MS, this.owner);
           const { root } = this.options;
           if (!await this.assertCurrentAttempt(tx)) {
@@ -20750,6 +20753,8 @@ class SqlResourceAdmission {
             if (control && control.state !== "running")
               reason ??= `root ${control.state}`;
           }
+          const held = [];
+          let blockedOtherwise = reason !== undefined;
           for (const scope of [`root:${root}`, "user"]) {
             const row = await tx.get(`SELECT config FROM ${this.configuration} WHERE scope = ?`, scope);
             const limit = row ? JSON.parse(String(row.config))[kind] : undefined;
@@ -20759,9 +20764,21 @@ class SqlResourceAdmission {
             const args = scope === "user" ? [kind] : [kind, root];
             const count = await tx.get(`SELECT COUNT(*) AS count FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'granted'`, ...args);
             const older = await tx.get(`SELECT 1 AS waiting FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'waiting' AND reason IS NOT ? AND ticket < (SELECT ticket FROM ${this.permits} WHERE operation = ?) LIMIT 1`, ...args, STAGE_START_PRESSURE_REASON, operation);
-            if (Number(count?.count) >= limit || older)
-              reason ??= `${scope} ${kind}`;
+            const used = Number(count?.count);
+            if (used < limit && !older)
+              continue;
+            reason ??= `${scope} ${kind}`;
+            const quarantinedPermits = kind === "process" && used >= limit ? await tx.all(`SELECT process_identity FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'granted' AND reason = ?`, ...args, QUARANTINED_REASON) : [];
+            if (quarantinedPermits.length > 0 && used - quarantinedPermits.length < limit)
+              held.push({
+                limit: `${scope === "user" ? "admission.userLimits" : "admission.limits"}.${kind} (${limit})`,
+                groups: quarantinedPermits.flatMap(({ process_identity }) => process_identity === null ? [] : [JSON.parse(process_identity).id])
+              });
+            else
+              blockedOtherwise = true;
           }
+          if (!blockedOtherwise && held.length > 0)
+            quarantined = held;
           if (reason && request?.delegated)
             throw new Error(`Delegated session cannot be admitted without capacity headroom: ${reason}`);
           const stageStart = kind === "session" && request?.delegated !== true;
@@ -20774,6 +20791,14 @@ class SqlResourceAdmission {
           signal?.throwIfAborted();
           return release;
         }
+        if (quarantined === undefined)
+          quarantinedSince = undefined;
+        else {
+          const now = performance.now();
+          quarantinedSince ??= now;
+          if (now - quarantinedSince >= QUARANTINED_CAPACITY_WAIT_MS)
+            throw new QuarantinedCapacityError(quarantined, now - quarantinedSince);
+        }
         await delay(50 + Math.floor(Math.random() * 25), undefined, { signal });
       }
     } catch (error) {
@@ -20815,9 +20840,9 @@ class SqlResourceAdmission {
           await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'host reboot confirmed' WHERE owner = ? AND state = 'granted'`, owner.id);
         } else if (stopped) {
           await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'owner death confirmed' WHERE owner = ? AND kind = 'session' AND state = 'granted'`, owner.id);
-          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ?`, UNCONFIRMED_REASON, owner.id, UNCONFIRMED_REASON);
+          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ? AND reason IS NOT ?`, UNCONFIRMED_REASON, owner.id, UNCONFIRMED_REASON, QUARANTINED_REASON);
         } else {
-          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND state = 'granted' AND reason IS NOT ? AND EXISTS (SELECT 1 FROM ${this.owners} WHERE id = ? AND expires_at < ${NOW_MS})`, EXPIRED_REASON, owner.id, EXPIRED_REASON, owner.id);
+          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND state = 'granted' AND reason IS NOT ? AND reason IS NOT ? AND EXISTS (SELECT 1 FROM ${this.owners} WHERE id = ? AND expires_at < ${NOW_MS})`, EXPIRED_REASON, owner.id, EXPIRED_REASON, QUARANTINED_REASON, owner.id);
         }
         if (owner.host === admissionHost && process.platform !== "win32") {
           const processes = await tx.all(`SELECT * FROM ${this.permits} WHERE owner = ? AND state = 'granted' AND kind = 'process' AND process_identity IS NOT NULL`, owner.id);
@@ -20845,9 +20870,10 @@ class SqlResourceAdmission {
       return true;
     if (permit.state === "waiting")
       return stopped || !!permit.expired;
-    if (stopped && (permit.kind === "session" || permit.reason !== UNCONFIRMED_REASON))
+    const quarantined = permit.reason === QUARANTINED_REASON;
+    if (stopped && (permit.kind === "session" || permit.reason !== UNCONFIRMED_REASON && !quarantined))
       return true;
-    if (!stopped && permit.expired && permit.reason !== EXPIRED_REASON)
+    if (!stopped && permit.expired && permit.reason !== EXPIRED_REASON && !quarantined)
       return true;
     return permit.kind === "process" && permit.host === admissionHost && process.platform !== "win32" && permit.process_identity !== null && processGroupStopped(permit.process_identity);
   }
@@ -20858,7 +20884,9 @@ class SqlResourceAdmission {
     await this.reconcile();
     return this.storage.transaction(async (tx) => {
       await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'worker exit confirmed' WHERE root = ? AND holder = ? AND state IN ('waiting', 'granted') AND (kind = 'session' OR state = 'waiting')`, this.options.root, holder);
-      return (await tx.run(`UPDATE ${this.permits} SET reason = 'cleanup unconfirmed' WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted'`, this.options.root, holder)).changes;
+      await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ?`, UNCONFIRMED_REASON, this.options.root, holder, QUARANTINED_REASON);
+      const held = await tx.get(`SELECT COUNT(*) AS count FROM ${this.permits} WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted'`, this.options.root, holder);
+      return Number(held?.count);
     });
   }
   async reapStoppedProcesses(holder) {
@@ -20905,10 +20933,13 @@ async function settleProcessGroups(storage, root, scope, reap) {
     survivors = await settle(REAP_KILL_GRACE_MS);
   }
   const confirmed = [...groups].flatMap(([group, operations]) => survivors.includes(group) ? [] : operations);
-  if (confirmed.length > 0)
+  const quarantined = reap ? survivors.flatMap((group) => groups.get(group) ?? []) : [];
+  if (confirmed.length > 0 || quarantined.length > 0)
     await storage.transaction(async (tx) => {
       for (const operation of confirmed)
         await tx.run(`UPDATE ${permits} SET state = 'released', reason = 'process group cleanup confirmed' WHERE operation = ? AND state = 'granted'`, operation);
+      for (const operation of quarantined)
+        await tx.run(`UPDATE ${permits} SET reason = ? WHERE operation = ? AND state = 'granted'`, QUARANTINED_REASON, operation);
     });
   return { survivors, unprobeable };
 }
@@ -20925,11 +20956,20 @@ async function reapProcessGroups(journal, root, scope) {
     return { survivors: [], unprobeable: [] };
   return settleProcessGroups(storage, root, scope, true);
 }
-var OWNER_LEASE_MS = 30000, REAP_TERM_GRACE_MS = 5000, REAP_KILL_GRACE_MS = 1e4, REAP_POLL_MS = 50, RELEASED_PERMIT_RETENTION = 256, RELEASED_PERMIT_PRUNE_BATCH = 1000, EXPIRED_REASON = "owner expired; cleanup unconfirmed", UNCONFIRMED_REASON = "cleanup unconfirmed", ANONYMOUS_REASON = "owner identity unavailable; cleanup unconfirmed", NOW_MS = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", LATE_ATTEMPT_POLL_MS = 100, LATE_ATTEMPT_REASON = "waiting for its attempt lease", STAGE_START_SPACING_MS = 5000, STAGE_START_PRESSURE_REASON = "paced for host load";
+var OWNER_LEASE_MS = 30000, REAP_TERM_GRACE_MS = 5000, REAP_KILL_GRACE_MS = 1e4, REAP_POLL_MS = 50, RELEASED_PERMIT_RETENTION = 256, RELEASED_PERMIT_PRUNE_BATCH = 1000, EXPIRED_REASON = "owner expired; cleanup unconfirmed", UNCONFIRMED_REASON = "cleanup unconfirmed", QUARANTINED_REASON = "cleanup unconfirmed; permit quarantined", ANONYMOUS_REASON = "owner identity unavailable; cleanup unconfirmed", QUARANTINED_CAPACITY_WAIT_MS, QuarantinedCapacityError, NOW_MS = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", LATE_ATTEMPT_POLL_MS = 100, LATE_ATTEMPT_REASON = "waiting for its attempt lease", STAGE_START_SPACING_MS = 5000, STAGE_START_PRESSURE_REASON = "paced for host load";
 var init_resource_admission = __esm(() => {
   init_internal();
   init_sql_run_journal();
+  init_workflow_infrastructure_error();
   init_admission_host();
+  QUARANTINED_CAPACITY_WAIT_MS = REAP_TERM_GRACE_MS + REAP_KILL_GRACE_MS;
+  QuarantinedCapacityError = class QuarantinedCapacityError extends WorkflowInfrastructureError {
+    constructor(capacity, waitedMs) {
+      const limits = capacity.map(({ limit }) => limit).join(" and ");
+      const groups = [...new Set(capacity.flatMap(({ groups }) => groups))].join(", ");
+      super("worker_cleanup_unconfirmed", `Local worker process cleanup unconfirmed; process permit refused after ${Math.round(waitedMs)} ms: ${limits} ${capacity.length === 1 ? "is" : "are"} held by quarantined permits of process groups ${groups} that survived SIGKILL`);
+    }
+  };
 });
 
 // dist/builtin/workflows/src/workers/admission-binding.ts

package/dist/builtin/workflows/src/extension/local-worker-entry.mjs

@@ -17462,6 +17462,7 @@ class SqlResourceAdmission {
         epoch: this.options.attempt.epoch
       }) : null);
     });
+    let quarantinedSince;
     try {
       for (;; ) {
         signal?.throwIfAborted();
@@ -17471,7 +17472,9 @@ class SqlResourceAdmission {
         }
         const hostReason = this.hostReason();
         const pressure = this.stageStartPressure(kind, request?.delegated);
+        let quarantined = undefined;
         const granted = await this.storage.transaction(async (tx) => {
+          quarantined = undefined;
           await tx.run(`UPDATE ${this.owners} SET expires_at = CAST(unixepoch('subsec') * 1000 AS INTEGER) + ? WHERE id = ?`, OWNER_LEASE_MS, this.owner);
           const { root } = this.options;
           if (!await this.assertCurrentAttempt(tx)) {
@@ -17489,6 +17492,8 @@ class SqlResourceAdmission {
             if (control && control.state !== "running")
               reason ??= `root ${control.state}`;
           }
+          const held = [];
+          let blockedOtherwise = reason !== undefined;
           for (const scope of [`root:${root}`, "user"]) {
             const row = await tx.get(`SELECT config FROM ${this.configuration} WHERE scope = ?`, scope);
             const limit = row ? JSON.parse(String(row.config))[kind] : undefined;
@@ -17498,9 +17503,21 @@ class SqlResourceAdmission {
             const args = scope === "user" ? [kind] : [kind, root];
             const count = await tx.get(`SELECT COUNT(*) AS count FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'granted'`, ...args);
             const older = await tx.get(`SELECT 1 AS waiting FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'waiting' AND reason IS NOT ? AND ticket < (SELECT ticket FROM ${this.permits} WHERE operation = ?) LIMIT 1`, ...args, STAGE_START_PRESSURE_REASON, operation);
-            if (Number(count?.count) >= limit || older)
-              reason ??= `${scope} ${kind}`;
+            const used = Number(count?.count);
+            if (used < limit && !older)
+              continue;
+            reason ??= `${scope} ${kind}`;
+            const quarantinedPermits = kind === "process" && used >= limit ? await tx.all(`SELECT process_identity FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'granted' AND reason = ?`, ...args, QUARANTINED_REASON) : [];
+            if (quarantinedPermits.length > 0 && used - quarantinedPermits.length < limit)
+              held.push({
+                limit: `${scope === "user" ? "admission.userLimits" : "admission.limits"}.${kind} (${limit})`,
+                groups: quarantinedPermits.flatMap(({ process_identity }) => process_identity === null ? [] : [JSON.parse(process_identity).id])
+              });
+            else
+              blockedOtherwise = true;
           }
+          if (!blockedOtherwise && held.length > 0)
+            quarantined = held;
           if (reason && request?.delegated)
             throw new Error(`Delegated session cannot be admitted without capacity headroom: ${reason}`);
           const stageStart = kind === "session" && request?.delegated !== true;
@@ -17513,6 +17530,14 @@ class SqlResourceAdmission {
           signal?.throwIfAborted();
           return release;
         }
+        if (quarantined === undefined)
+          quarantinedSince = undefined;
+        else {
+          const now = performance.now();
+          quarantinedSince ??= now;
+          if (now - quarantinedSince >= QUARANTINED_CAPACITY_WAIT_MS)
+            throw new QuarantinedCapacityError(quarantined, now - quarantinedSince);
+        }
         await delay(50 + Math.floor(Math.random() * 25), undefined, { signal });
       }
     } catch (error) {
@@ -17554,9 +17579,9 @@ class SqlResourceAdmission {
           await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'host reboot confirmed' WHERE owner = ? AND state = 'granted'`, owner.id);
         } else if (stopped) {
           await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'owner death confirmed' WHERE owner = ? AND kind = 'session' AND state = 'granted'`, owner.id);
-          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ?`, UNCONFIRMED_REASON, owner.id, UNCONFIRMED_REASON);
+          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ? AND reason IS NOT ?`, UNCONFIRMED_REASON, owner.id, UNCONFIRMED_REASON, QUARANTINED_REASON);
         } else {
-          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND state = 'granted' AND reason IS NOT ? AND EXISTS (SELECT 1 FROM ${this.owners} WHERE id = ? AND expires_at < ${NOW_MS})`, EXPIRED_REASON, owner.id, EXPIRED_REASON, owner.id);
+          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND state = 'granted' AND reason IS NOT ? AND reason IS NOT ? AND EXISTS (SELECT 1 FROM ${this.owners} WHERE id = ? AND expires_at < ${NOW_MS})`, EXPIRED_REASON, owner.id, EXPIRED_REASON, QUARANTINED_REASON, owner.id);
         }
         if (owner.host === admissionHost && process.platform !== "win32") {
           const processes = await tx.all(`SELECT * FROM ${this.permits} WHERE owner = ? AND state = 'granted' AND kind = 'process' AND process_identity IS NOT NULL`, owner.id);
@@ -17584,9 +17609,10 @@ class SqlResourceAdmission {
       return true;
     if (permit.state === "waiting")
       return stopped || !!permit.expired;
-    if (stopped && (permit.kind === "session" || permit.reason !== UNCONFIRMED_REASON))
+    const quarantined = permit.reason === QUARANTINED_REASON;
+    if (stopped && (permit.kind === "session" || permit.reason !== UNCONFIRMED_REASON && !quarantined))
       return true;
-    if (!stopped && permit.expired && permit.reason !== EXPIRED_REASON)
+    if (!stopped && permit.expired && permit.reason !== EXPIRED_REASON && !quarantined)
       return true;
     return permit.kind === "process" && permit.host === admissionHost && process.platform !== "win32" && permit.process_identity !== null && processGroupStopped(permit.process_identity);
   }
@@ -17597,7 +17623,9 @@ class SqlResourceAdmission {
     await this.reconcile();
     return this.storage.transaction(async (tx) => {
       await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'worker exit confirmed' WHERE root = ? AND holder = ? AND state IN ('waiting', 'granted') AND (kind = 'session' OR state = 'waiting')`, this.options.root, holder);
-      return (await tx.run(`UPDATE ${this.permits} SET reason = 'cleanup unconfirmed' WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted'`, this.options.root, holder)).changes;
+      await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ?`, UNCONFIRMED_REASON, this.options.root, holder, QUARANTINED_REASON);
+      const held = await tx.get(`SELECT COUNT(*) AS count FROM ${this.permits} WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted'`, this.options.root, holder);
+      return Number(held?.count);
     });
   }
   async reapStoppedProcesses(holder) {
@@ -17644,10 +17672,13 @@ async function settleProcessGroups(storage, root, scope, reap) {
     survivors = await settle(REAP_KILL_GRACE_MS);
   }
   const confirmed = [...groups].flatMap(([group, operations]) => survivors.includes(group) ? [] : operations);
-  if (confirmed.length > 0)
+  const quarantined = reap ? survivors.flatMap((group) => groups.get(group) ?? []) : [];
+  if (confirmed.length > 0 || quarantined.length > 0)
     await storage.transaction(async (tx) => {
       for (const operation of confirmed)
         await tx.run(`UPDATE ${permits} SET state = 'released', reason = 'process group cleanup confirmed' WHERE operation = ? AND state = 'granted'`, operation);
+      for (const operation of quarantined)
+        await tx.run(`UPDATE ${permits} SET reason = ? WHERE operation = ? AND state = 'granted'`, QUARANTINED_REASON, operation);
     });
   return { survivors, unprobeable };
 }
@@ -17664,11 +17695,20 @@ async function reapProcessGroups(journal, root, scope) {
     return { survivors: [], unprobeable: [] };
   return settleProcessGroups(storage, root, scope, true);
 }
-var OWNER_LEASE_MS = 30000, REAP_TERM_GRACE_MS = 5000, REAP_KILL_GRACE_MS = 1e4, REAP_POLL_MS = 50, RELEASED_PERMIT_RETENTION = 256, RELEASED_PERMIT_PRUNE_BATCH = 1000, EXPIRED_REASON = "owner expired; cleanup unconfirmed", UNCONFIRMED_REASON = "cleanup unconfirmed", ANONYMOUS_REASON = "owner identity unavailable; cleanup unconfirmed", NOW_MS = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", LATE_ATTEMPT_POLL_MS = 100, LATE_ATTEMPT_REASON = "waiting for its attempt lease", STAGE_START_SPACING_MS = 5000, STAGE_START_PRESSURE_REASON = "paced for host load";
+var OWNER_LEASE_MS = 30000, REAP_TERM_GRACE_MS = 5000, REAP_KILL_GRACE_MS = 1e4, REAP_POLL_MS = 50, RELEASED_PERMIT_RETENTION = 256, RELEASED_PERMIT_PRUNE_BATCH = 1000, EXPIRED_REASON = "owner expired; cleanup unconfirmed", UNCONFIRMED_REASON = "cleanup unconfirmed", QUARANTINED_REASON = "cleanup unconfirmed; permit quarantined", ANONYMOUS_REASON = "owner identity unavailable; cleanup unconfirmed", QUARANTINED_CAPACITY_WAIT_MS, QuarantinedCapacityError, NOW_MS = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", LATE_ATTEMPT_POLL_MS = 100, LATE_ATTEMPT_REASON = "waiting for its attempt lease", STAGE_START_SPACING_MS = 5000, STAGE_START_PRESSURE_REASON = "paced for host load";
 var init_resource_admission = __esm(() => {
   init_internal();
   init_sql_run_journal();
+  init_workflow_infrastructure_error();
   init_admission_host();
+  QUARANTINED_CAPACITY_WAIT_MS = REAP_TERM_GRACE_MS + REAP_KILL_GRACE_MS;
+  QuarantinedCapacityError = class QuarantinedCapacityError extends WorkflowInfrastructureError {
+    constructor(capacity, waitedMs) {
+      const limits = capacity.map(({ limit }) => limit).join(" and ");
+      const groups = [...new Set(capacity.flatMap(({ groups }) => groups))].join(", ");
+      super("worker_cleanup_unconfirmed", `Local worker process cleanup unconfirmed; process permit refused after ${Math.round(waitedMs)} ms: ${limits} ${capacity.length === 1 ? "is" : "are"} held by quarantined permits of process groups ${groups} that survived SIGKILL`);
+    }
+  };
 });
 
 // dist/builtin/workflows/src/workers/admission-binding.ts
@@ -81929,7 +81969,18 @@ async function runWorker(start) {
       const resourceAdmission = {
         async acquire(kind, signal, requestOptions) {
           await awaitAdmission(signal);
-          return admission ? admission.acquire(kind, signal, requestOptions) : async () => {};
+          if (!admission)
+            return async () => {};
+          try {
+            return await admission.acquire(kind, signal, requestOptions);
+          } catch (error) {
+            if (error instanceof QuarantinedCapacityError && !attemptFenced) {
+              poolFault = error.message;
+              attemptFenced = true;
+              controller.abort(new WorkAttemptFencedError(error.message));
+            }
+            throw error;
+          }
         }
       };
       const admitted = admittedSessionAdapter(adapter, resourceAdmission, async (signal) => {

package/dist/builtin/workflows/src/index.js

@@ -17573,6 +17573,7 @@ class SqlResourceAdmission {
         epoch: this.options.attempt.epoch
       }) : null);
     });
+    let quarantinedSince;
     try {
       for (;; ) {
         signal?.throwIfAborted();
@@ -17582,7 +17583,9 @@ class SqlResourceAdmission {
         }
         const hostReason = this.hostReason();
         const pressure = this.stageStartPressure(kind, request?.delegated);
+        let quarantined = undefined;
         const granted = await this.storage.transaction(async (tx) => {
+          quarantined = undefined;
           await tx.run(`UPDATE ${this.owners} SET expires_at = CAST(unixepoch('subsec') * 1000 AS INTEGER) + ? WHERE id = ?`, OWNER_LEASE_MS, this.owner);
           const { root } = this.options;
           if (!await this.assertCurrentAttempt(tx)) {
@@ -17600,6 +17603,8 @@ class SqlResourceAdmission {
             if (control && control.state !== "running")
               reason ??= `root ${control.state}`;
           }
+          const held = [];
+          let blockedOtherwise = reason !== undefined;
           for (const scope of [`root:${root}`, "user"]) {
             const row = await tx.get(`SELECT config FROM ${this.configuration} WHERE scope = ?`, scope);
             const limit = row ? JSON.parse(String(row.config))[kind] : undefined;
@@ -17609,9 +17614,21 @@ class SqlResourceAdmission {
             const args = scope === "user" ? [kind] : [kind, root];
             const count = await tx.get(`SELECT COUNT(*) AS count FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'granted'`, ...args);
             const older = await tx.get(`SELECT 1 AS waiting FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'waiting' AND reason IS NOT ? AND ticket < (SELECT ticket FROM ${this.permits} WHERE operation = ?) LIMIT 1`, ...args, STAGE_START_PRESSURE_REASON, operation);
-            if (Number(count?.count) >= limit || older)
-              reason ??= `${scope} ${kind}`;
+            const used = Number(count?.count);
+            if (used < limit && !older)
+              continue;
+            reason ??= `${scope} ${kind}`;
+            const quarantinedPermits = kind === "process" && used >= limit ? await tx.all(`SELECT process_identity FROM ${this.permits} WHERE kind = ? ${rootClause} AND state = 'granted' AND reason = ?`, ...args, QUARANTINED_REASON) : [];
+            if (quarantinedPermits.length > 0 && used - quarantinedPermits.length < limit)
+              held.push({
+                limit: `${scope === "user" ? "admission.userLimits" : "admission.limits"}.${kind} (${limit})`,
+                groups: quarantinedPermits.flatMap(({ process_identity }) => process_identity === null ? [] : [JSON.parse(process_identity).id])
+              });
+            else
+              blockedOtherwise = true;
           }
+          if (!blockedOtherwise && held.length > 0)
+            quarantined = held;
           if (reason && request?.delegated)
             throw new Error(`Delegated session cannot be admitted without capacity headroom: ${reason}`);
           const stageStart = kind === "session" && request?.delegated !== true;
@@ -17624,6 +17641,14 @@ class SqlResourceAdmission {
           signal?.throwIfAborted();
           return release;
         }
+        if (quarantined === undefined)
+          quarantinedSince = undefined;
+        else {
+          const now = performance.now();
+          quarantinedSince ??= now;
+          if (now - quarantinedSince >= QUARANTINED_CAPACITY_WAIT_MS)
+            throw new QuarantinedCapacityError(quarantined, now - quarantinedSince);
+        }
         await delay(50 + Math.floor(Math.random() * 25), undefined, { signal });
       }
     } catch (error) {
@@ -17665,9 +17690,9 @@ class SqlResourceAdmission {
           await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'host reboot confirmed' WHERE owner = ? AND state = 'granted'`, owner.id);
         } else if (stopped) {
           await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'owner death confirmed' WHERE owner = ? AND kind = 'session' AND state = 'granted'`, owner.id);
-          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ?`, UNCONFIRMED_REASON, owner.id, UNCONFIRMED_REASON);
+          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ? AND reason IS NOT ?`, UNCONFIRMED_REASON, owner.id, UNCONFIRMED_REASON, QUARANTINED_REASON);
         } else {
-          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND state = 'granted' AND reason IS NOT ? AND EXISTS (SELECT 1 FROM ${this.owners} WHERE id = ? AND expires_at < ${NOW_MS})`, EXPIRED_REASON, owner.id, EXPIRED_REASON, owner.id);
+          await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE owner = ? AND state = 'granted' AND reason IS NOT ? AND reason IS NOT ? AND EXISTS (SELECT 1 FROM ${this.owners} WHERE id = ? AND expires_at < ${NOW_MS})`, EXPIRED_REASON, owner.id, EXPIRED_REASON, QUARANTINED_REASON, owner.id);
         }
         if (owner.host === admissionHost && process.platform !== "win32") {
           const processes = await tx.all(`SELECT * FROM ${this.permits} WHERE owner = ? AND state = 'granted' AND kind = 'process' AND process_identity IS NOT NULL`, owner.id);
@@ -17695,9 +17720,10 @@ class SqlResourceAdmission {
       return true;
     if (permit.state === "waiting")
       return stopped || !!permit.expired;
-    if (stopped && (permit.kind === "session" || permit.reason !== UNCONFIRMED_REASON))
+    const quarantined = permit.reason === QUARANTINED_REASON;
+    if (stopped && (permit.kind === "session" || permit.reason !== UNCONFIRMED_REASON && !quarantined))
       return true;
-    if (!stopped && permit.expired && permit.reason !== EXPIRED_REASON)
+    if (!stopped && permit.expired && permit.reason !== EXPIRED_REASON && !quarantined)
       return true;
     return permit.kind === "process" && permit.host === admissionHost && process.platform !== "win32" && permit.process_identity !== null && processGroupStopped(permit.process_identity);
   }
@@ -17708,7 +17734,9 @@ class SqlResourceAdmission {
     await this.reconcile();
     return this.storage.transaction(async (tx) => {
       await tx.run(`UPDATE ${this.permits} SET state = 'released', reason = 'worker exit confirmed' WHERE root = ? AND holder = ? AND state IN ('waiting', 'granted') AND (kind = 'session' OR state = 'waiting')`, this.options.root, holder);
-      return (await tx.run(`UPDATE ${this.permits} SET reason = 'cleanup unconfirmed' WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted'`, this.options.root, holder)).changes;
+      await tx.run(`UPDATE ${this.permits} SET reason = ? WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted' AND reason IS NOT ?`, UNCONFIRMED_REASON, this.options.root, holder, QUARANTINED_REASON);
+      const held = await tx.get(`SELECT COUNT(*) AS count FROM ${this.permits} WHERE root = ? AND holder = ? AND kind = 'process' AND state = 'granted'`, this.options.root, holder);
+      return Number(held?.count);
     });
   }
   async reapStoppedProcesses(holder) {
@@ -17755,10 +17783,13 @@ async function settleProcessGroups(storage, root, scope, reap) {
     survivors = await settle(REAP_KILL_GRACE_MS);
   }
   const confirmed = [...groups].flatMap(([group, operations]) => survivors.includes(group) ? [] : operations);
-  if (confirmed.length > 0)
+  const quarantined = reap ? survivors.flatMap((group) => groups.get(group) ?? []) : [];
+  if (confirmed.length > 0 || quarantined.length > 0)
     await storage.transaction(async (tx) => {
       for (const operation of confirmed)
         await tx.run(`UPDATE ${permits} SET state = 'released', reason = 'process group cleanup confirmed' WHERE operation = ? AND state = 'granted'`, operation);
+      for (const operation of quarantined)
+        await tx.run(`UPDATE ${permits} SET reason = ? WHERE operation = ? AND state = 'granted'`, QUARANTINED_REASON, operation);
     });
   return { survivors, unprobeable };
 }
@@ -17775,11 +17806,20 @@ async function reapProcessGroups(journal, root, scope) {
     return { survivors: [], unprobeable: [] };
   return settleProcessGroups(storage, root, scope, true);
 }
-var OWNER_LEASE_MS = 30000, REAP_TERM_GRACE_MS = 5000, REAP_KILL_GRACE_MS = 1e4, REAP_POLL_MS = 50, RELEASED_PERMIT_RETENTION = 256, RELEASED_PERMIT_PRUNE_BATCH = 1000, EXPIRED_REASON = "owner expired; cleanup unconfirmed", UNCONFIRMED_REASON = "cleanup unconfirmed", ANONYMOUS_REASON = "owner identity unavailable; cleanup unconfirmed", NOW_MS = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", LATE_ATTEMPT_POLL_MS = 100, LATE_ATTEMPT_REASON = "waiting for its attempt lease", STAGE_START_SPACING_MS = 5000, STAGE_START_PRESSURE_REASON = "paced for host load";
+var OWNER_LEASE_MS = 30000, REAP_TERM_GRACE_MS = 5000, REAP_KILL_GRACE_MS = 1e4, REAP_POLL_MS = 50, RELEASED_PERMIT_RETENTION = 256, RELEASED_PERMIT_PRUNE_BATCH = 1000, EXPIRED_REASON = "owner expired; cleanup unconfirmed", UNCONFIRMED_REASON = "cleanup unconfirmed", QUARANTINED_REASON = "cleanup unconfirmed; permit quarantined", ANONYMOUS_REASON = "owner identity unavailable; cleanup unconfirmed", QUARANTINED_CAPACITY_WAIT_MS, QuarantinedCapacityError, NOW_MS = "CAST(unixepoch('subsec') * 1000 AS INTEGER)", LATE_ATTEMPT_POLL_MS = 100, LATE_ATTEMPT_REASON = "waiting for its attempt lease", STAGE_START_SPACING_MS = 5000, STAGE_START_PRESSURE_REASON = "paced for host load";
 var init_resource_admission = __esm(() => {
   init_internal();
   init_sql_run_journal();
+  init_workflow_infrastructure_error();
   init_admission_host();
+  QUARANTINED_CAPACITY_WAIT_MS = REAP_TERM_GRACE_MS + REAP_KILL_GRACE_MS;
+  QuarantinedCapacityError = class QuarantinedCapacityError extends WorkflowInfrastructureError {
+    constructor(capacity, waitedMs) {
+      const limits = capacity.map(({ limit }) => limit).join(" and ");
+      const groups = [...new Set(capacity.flatMap(({ groups }) => groups))].join(", ");
+      super("worker_cleanup_unconfirmed", `Local worker process cleanup unconfirmed; process permit refused after ${Math.round(waitedMs)} ms: ${limits} ${capacity.length === 1 ? "is" : "are"} held by quarantined permits of process groups ${groups} that survived SIGKILL`);
+    }
+  };
 });
 
 // dist/builtin/workflows/src/workers/admission-binding.ts