@lincoln504/pi-research 1.0.12 → 1.1.0

package/README.md

@@ -3,7 +3,7 @@
 </p>
 
 <p align="center">
-  <strong>Limitless research and knowledge store — for pi, agent skill, or SDK.</strong>
+  <strong>Local web research and knowledge store</strong>
 </p>
 
 <p align="center">
@@ -24,9 +24,20 @@
 
 <a href="https://github.com/Lincoln504/pi-research/actions/workflows/ci.yml"><img alt="Build status" src="https://img.shields.io/github/actions/workflow/status/Lincoln504/pi-research/ci.yml?style=flat-square&branch=main" /></a> <a href="https://www.npmjs.com/package/@lincoln504/pi-research"><img alt="npm version" src="https://img.shields.io/npm/v/@lincoln504/pi-research.svg?style=flat-square" /></a>
 
-Research is broken into subtasks automatically, with each sub-researcher given a high volume of sources to investigate. Then, evaluator decides whether the answer is complete or an additional round of research is needed. The final result is a synthesized cited report.
+**Free, unlimited web search &amp; scraping — no monthly quota, no subscription.**
+You pay only for the LLM tokens you use, and it builds your own private, searchable
+knowledge store on your machine.
 
-Ask in natural language — the tool understands the depth needed:
+### How it works
+
+A research run is a loop of agent teams: a coordinator plans the work and runs an
+opening search, parallel researcher agents scrape and read sources through a fixed,
+sandboxed toolkit (no filesystem or shell access), and an evaluator decides whether
+to run another round or synthesize — returning one cited Markdown report while its
+sources are saved to the knowledge store in the background.
+
+Three depth levels — normal, deep, ultra — set the team size and number of rounds;
+ask in natural language and the tool picks the right one:
 
 ![Prompt-driven multi-round research in the pi TUI](docs/media/02-prompt-research.gif)
 
@@ -39,12 +50,13 @@ Ask in natural language — the tool understands the depth needed:
 - Using the pi-research agent skill as OpenClaw's web access.
 - Building agent systems that identify and examine web sources.
 
-### Advantages
+### Why pi-research
 
-- Unlimited search and scrape, for free — you only pay for LLM tokens.
-- Context-efficient — it returns a synthesized, cited report to the chat instead of dumping raw web content into the conversation.
-- Safe by design — web access runs inside a specialized, limited research agent with no filesystem or shell access.
-- Search a little or a lot — three depth levels in the pi tool (four via the SDK and standalone CLI, which add the quick depth-0 path). Levels 1 and 2 are recommended for everyday workflow use; level 3 is for larger-scale investigations.
+- **Free, unlimited search &amp; scrape — no quota, no monthly fee.** Most AI-search providers cap free searches per month and then charge; results run on *their* index on *their* servers. pi-research's search and scraping layer (DuckDuckGo via a stealth browser) is free and unlimited — you pay only for the LLM tokens you use, never per search or per month.
+- **You own your index.** Every finding is written into a local [LanceDB](https://lancedb.com) knowledge store on *your* machine — a persistent, searchable index you control and keep, which also seeds future runs so repeat questions get faster, cheaper answers. No borrowed, server-side index you can't keep or query. Findings can also be kept **globally**, scoped to a **single project**, or **not kept at all** — set per directory, changed whenever you like. The optionality is the point: the store is yours to build, partition, or switch off.
+- **Cited reports, not raw dumps.** It returns one synthesized, sourced Markdown report to the chat instead of dumping raw web content into the conversation — context-efficient and easy to verify.
+- **Sandboxed by design.** Web access runs inside a specialized, limited research agent with no filesystem or shell access — safe to point at anything.
+- **Search a little or a lot** — three depth levels in the pi tool (four via the SDK and standalone CLI, which add the quick depth-0 path). Levels 1 and 2 are recommended for everyday workflow use; level 3 is for larger-scale investigations.
 
 ### Requirements / limitations
 

package/dist/cli.mjs

@@ -14228,14 +14228,14 @@ init_file_lock_service();
 init_service_registry();
 init_service_names();
 import * as pathmod2 from "node:path";
-var DEFAULT_MAX_CONCURRENT_RUNS = 2;
-var DEFAULT_ACQUIRE_TIMEOUT_MS = 0;
+var DEFAULT_MAX_CONCURRENT_RUNS = 3;
+var DEFAULT_ACQUIRE_TIMEOUT_MS = 6e5;
 var POLL_DELAY_MS = 250;
 var ResearchRunCapacityError = class extends Error {
   slots;
   constructor(slots, maxWaitMs) {
     super(
-      `Maximum concurrent research runs (${slots}) reached${maxWaitMs > 0 ? ` after waiting ${maxWaitMs}ms` : ""}; wait for an in-progress run to finish and retry. (Override with PI_RESEARCH_MAX_CONCURRENT_RUNS.)`
+      `Maximum concurrent research runs (${slots}) reached${maxWaitMs > 0 ? ` and still busy after queueing for ${Math.round(maxWaitMs / 1e3)}s` : ""}; another run is still in progress. This is temporary \u2014 retry shortly. (Raise the cap with PI_RESEARCH_MAX_CONCURRENT_RUNS, or the queue wait with PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS.)`
     );
     this.name = "ResearchRunCapacityError";
     this.slots = slots;
@@ -14335,16 +14335,20 @@ var ResearchRunSemaphore = class {
    * @param signal aborts the wait. Without it a run started under a long
    *   `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS` would ignore Ctrl-C until the deadline
    *   passed, because the wait happens before any orchestrator exists to cancel.
+   * @param onQueued invoked at most once, when the first full sweep finds every
+   *   slot busy and the call is about to wait. Lets a front-end tell the user the
+   *   run is queued rather than hung. Never allowed to break the acquire.
    * @throws {ResearchRunCapacityError} if no slot frees within `maxWaitMs`.
    * @throws {Error} named `AbortError` if `signal` aborts while waiting.
    */
-  async acquire(maxWaitMs = this.defaultMaxWaitMs, signal) {
+  async acquire(maxWaitMs = this.defaultMaxWaitMs, signal, onQueued) {
     if (!this._initialized) await this.initialize();
     if (signal?.aborted) {
       metrics.increment("research_run_acquire_total", 1, { status: "aborted" });
       throwIfAborted(signal);
     }
     const deadline = Date.now() + Math.max(0, maxWaitMs);
+    let announcedQueued = false;
     for (; ; ) {
       const start = this.nextStartIndex;
       this.nextStartIndex = (this.nextStartIndex + 1) % this.maxSlots;
@@ -14361,6 +14365,15 @@ var ResearchRunSemaphore = class {
         metrics.increment("research_run_acquire_total", 1, { status: "exhausted" });
         throw new ResearchRunCapacityError(this.maxSlots, maxWaitMs);
       }
+      if (!announcedQueued) {
+        announcedQueued = true;
+        logger.info(`[ResearchRunSemaphore] All ${this.maxSlots} run slot(s) busy \u2014 queueing for up to ${maxWaitMs}ms.`);
+        try {
+          onQueued?.(this.maxSlots, maxWaitMs);
+        } catch (err) {
+          logger.debug(`[ResearchRunSemaphore] onQueued callback threw: ${err instanceof Error ? err.message : String(err)}`);
+        }
+      }
       try {
         await sleep(POLL_DELAY_MS, signal);
       } catch (err) {
@@ -14886,6 +14899,17 @@ init_service_registry();
 init_service_interfaces();
 var lastRestartTime = 0;
 var RESTART_COOLDOWN_MS = 1e4;
+var LEADER_HANDOVER_RETRIES = 3;
+var HANDOVER_BACKOFF_MS = 1200;
+async function recoverFromLeaderHandover(container) {
+  logger.warn("[BrowserManager] Leader handover detected (pool draining) \u2014 dropping stale scheduler handle and re-resolving...");
+  await forceSchedulerRestart2(false, container).catch(
+    (err) => logger.debug("[BrowserManager] Scheduler re-resolve during handover failed (non-fatal):", err)
+  );
+  await waitForBrowserPoolIdle(15e3).catch(
+    (err) => logger.debug("Wait for browser idle timed out or failed:", err)
+  );
+}
 async function getScheduler2(config, container = getServiceContainer()) {
   try {
     const factory = await getService(ServiceNames.SCHEDULER_FACTORY, void 0, container);
@@ -14902,7 +14926,7 @@ async function forceSchedulerRestart2(forceClearRemoteState = false, container =
     return await forceSchedulerRestart(forceClearRemoteState, container);
   }
 }
-async function runBrowserTask(taskOrUrl, type = "scrape", config, signal, retries = 1, container = getServiceContainer()) {
+async function runBrowserTask(taskOrUrl, type = "scrape", config, signal, retries = 1, container = getServiceContainer(), handoverRetries = LEADER_HANDOVER_RETRIES) {
   if (signal?.aborted) throw new Error("Aborted");
   const sessionId = typeof taskOrUrl === "object" ? taskOrUrl.sessionId : void 0;
   const breaker2 = sessionId ? getBrowserCircuitBreaker(sessionId) : browserCircuitBreaker;
@@ -14923,6 +14947,19 @@ async function runBrowserTask(taskOrUrl, type = "scrape", config, signal, retrie
     });
   } catch (error) {
     if (signal?.aborted || error instanceof Error && error.message === "Aborted") throw new Error("Aborted", { cause: error });
+    const isHandover = isPoolShutdownError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error);
+    if (isHandover && handoverRetries > 0) {
+      errorTracker.trackError(error, {
+        component: "browser-manager",
+        operation: type,
+        taskType: type,
+        errorType: "leader_handover"
+      });
+      logger.warn(`[BrowserManager] Leader handover during ${type} task (handover attempts left: ${handoverRetries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
+      await recoverFromLeaderHandover(container);
+      await new Promise((resolve8) => setTimeout(resolve8, HANDOVER_BACKOFF_MS + Math.floor(Math.random() * 400)));
+      return runBrowserTask(taskOrUrl, type, config, signal, retries, container, handoverRetries - 1);
+    }
     if (retries > 0 && isTransientSocketError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error)) {
       errorTracker.trackError(error, {
         component: "browser-manager",
@@ -14948,7 +14985,7 @@ async function runBrowserTask(taskOrUrl, type = "scrape", config, signal, retrie
       }
       const jitter = 100 + Math.floor(Math.random() * 400);
       await new Promise((resolve8) => setTimeout(resolve8, jitter));
-      return runBrowserTask(taskOrUrl, type, config, signal, retries - 1, container);
+      return runBrowserTask(taskOrUrl, type, config, signal, retries - 1, container, handoverRetries);
     }
     throw error;
   }
@@ -14968,7 +15005,7 @@ async function runBrowserHealthCheck(config, retries = 1, signal, container = ge
       });
       logger.warn(`[BrowserManager] Transient socket error during healthcheck (retries left: ${retries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
       if (isPoolShutdownError(error)) {
-        await waitForBrowserPoolIdle(15e3).catch((err) => logger.debug("Wait for browser idle timed out or failed:", err));
+        await recoverFromLeaderHandover(container);
       } else {
         const now = Date.now();
         if (now - lastRestartTime > RESTART_COOLDOWN_MS) {
@@ -14987,7 +15024,7 @@ async function runBrowserHealthCheck(config, retries = 1, signal, container = ge
     throw error;
   }
 }
-async function runWorkerSearch(query, config, signal, retries = 1, sessionId, container = getServiceContainer()) {
+async function runWorkerSearch(query, config, signal, retries = 1, sessionId, container = getServiceContainer(), handoverRetries = LEADER_HANDOVER_RETRIES) {
   if (signal?.aborted) throw new Error("Aborted");
   const breaker2 = sessionId ? getBrowserCircuitBreaker(sessionId) : browserCircuitBreaker;
   try {
@@ -14998,6 +15035,19 @@ async function runWorkerSearch(query, config, signal, retries = 1, sessionId, co
     });
   } catch (error) {
     if (signal?.aborted || error instanceof Error && error.message === "Aborted") throw new Error("Aborted", { cause: error });
+    const isHandover = isPoolShutdownError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error);
+    if (isHandover && handoverRetries > 0) {
+      errorTracker.trackError(error, {
+        component: "browser-manager",
+        operation: "search",
+        query,
+        errorType: "leader_handover"
+      });
+      logger.warn(`[BrowserManager] Leader handover during search (handover attempts left: ${handoverRetries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
+      await recoverFromLeaderHandover(container);
+      await new Promise((resolve8) => setTimeout(resolve8, HANDOVER_BACKOFF_MS + Math.floor(Math.random() * 400)));
+      return runWorkerSearch(query, config, signal, retries, sessionId, container, handoverRetries - 1);
+    }
     if (retries > 0 && isTransientSocketError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error)) {
       errorTracker.trackError(error, {
         component: "browser-manager",
@@ -15021,7 +15071,7 @@ async function runWorkerSearch(query, config, signal, retries = 1, sessionId, co
       }
       const jitter = 100 + Math.floor(Math.random() * 400);
       await new Promise((resolve8) => setTimeout(resolve8, jitter));
-      return runWorkerSearch(query, config, signal, retries - 1, sessionId, container);
+      return runWorkerSearch(query, config, signal, retries - 1, sessionId, container, handoverRetries);
     }
     throw error;
   }
@@ -20813,6 +20863,9 @@ var HeadlessObserver = class {
   onStart(query, complexity) {
     this.emit("start", { query, complexity });
   }
+  onRunQueued(slots, maxWaitMs) {
+    this.emit("run_queued", { slots, maxWaitMs });
+  }
   onPlanningStart(attempt) {
     this.emit("planning_start", { attempt });
   }
@@ -21611,7 +21664,9 @@ var ResearchOrchestrationService = class {
     try {
       const container = tryGetServiceContainerFromCtx(ctx);
       const semaphore = await getService(ServiceNames.RESEARCH_RUN_SEMAPHORE, ctx, container);
-      runSlot = await semaphore.acquire(void 0, signal);
+      runSlot = await semaphore.acquire(void 0, signal, (slots, maxWaitMs) => {
+        notifyRunQueued(observer, slots, maxWaitMs);
+      });
       logger.info(`[ResearchOrchestrationService] Acquired research run slot ${runSlot.slotIndex} (cap ${semaphore.getMaxSlots()}).`);
     } catch (err) {
       if (err instanceof ResearchRunCapacityError) throw err;
@@ -21965,6 +22020,15 @@ var ResearchOrchestrationService = class {
     }
   }
 };
+function notifyRunQueued(observer, slots, maxWaitMs) {
+  try {
+    const obs = observer;
+    if (typeof obs?.onRunQueued === "function") obs.onRunQueued(slots, maxWaitMs);
+    else if (typeof obs?.onProgress === "function") obs.onProgress("run_queued", { slots, maxWaitMs });
+  } catch (err) {
+    logger.debug(`[ResearchOrchestrationService] run_queued notification threw: ${err instanceof Error ? err.message : String(err)}`);
+  }
+}
 
 // src/orchestration/research-session-service.ts
 init_logger();
@@ -23221,7 +23285,15 @@ var EXIT = {
   /** Missing setup: package, model, or API key not configured. */
   CONFIG: 78,
   /** Something went wrong at runtime (network, provider, internal). */
-  SOFTWARE: 70
+  SOFTWARE: 70,
+  /**
+   * Temporary failure — the request was well-formed and the install is healthy,
+   * but the machine is at capacity right now (all research run slots held by
+   * live runs). Distinct from SOFTWARE so a calling agent can tell "busy, try
+   * again shortly" apart from "this run crashed" and back off instead of
+   * re-running a broken command.
+   */
+  TEMPFAIL: 75
 };
 function toStderr(s) {
   process.stderr.write(s);
@@ -23241,6 +23313,12 @@ function makeProgressObserver() {
           toStderr(`  \u2022 plan: ${data?.plan?.researchers?.length ?? "?"} researcher(s)
 `);
           break;
+        case "run_queued":
+          toStderr(
+            `  \u2022 queued: ${data?.slots ?? "?"} concurrent run(s) already active \u2014 waiting up to ${Math.round((data?.maxWaitMs ?? 0) / 1e3)}s for a slot\u2026
+`
+          );
+          break;
         case "round_start":
           toStderr(`  \u2022 round ${data?.round}
 `);
@@ -23785,17 +23863,25 @@ async function reportError(err, what, json) {
   const msg2 = err instanceof Error ? err.message : String(err);
   const lower = msg2.toLowerCase();
   const isTaggedResearchStop = err?.code === RESEARCH_STOPPED_ERROR_CODE;
-  const isConfigError = !isTaggedResearchStop && (lower.includes("no llm model available") || lower.includes("not found in pi") || lower.includes("no api key") || lower.includes("provider must be specified") || lower.includes("invalid model string") || lower.includes("authentication") || lower.includes("unauthorized") || lower.includes("api key"));
+  const isCapacity = err instanceof Error && err.name === "ResearchRunCapacityError";
+  const isConfigError = !isTaggedResearchStop && !isCapacity && (lower.includes("no llm model available") || lower.includes("not found in pi") || lower.includes("no api key") || lower.includes("provider must be specified") || lower.includes("invalid model string") || lower.includes("authentication") || lower.includes("unauthorized") || lower.includes("api key"));
   const isRateLimit = lower.includes("429") || lower.includes("rate limit") || lower.includes("too many requests");
-  const exitCode = isConfigError && !isRateLimit ? EXIT.CONFIG : EXIT.SOFTWARE;
+  const exitCode = isCapacity ? EXIT.TEMPFAIL : isConfigError && !isRateLimit ? EXIT.CONFIG : EXIT.SOFTWARE;
   if (json) {
     const payload = { ok: false, error: msg2, exitCode };
+    if (isCapacity || isRateLimit) payload["retryable"] = true;
     if (process.env["PI_RESEARCH_DEBUG"] === "true" && err instanceof Error && err.stack) {
       payload["stack"] = err.stack;
     }
     toStdout(pretty(payload));
     return exitCode;
   }
+  if (isCapacity) {
+    toStderr(`
+Error: pi-research ${what} did not start: ${msg2}
+`);
+    return exitCode;
+  }
   if (isRateLimit) {
     toStderr(`
 Error: pi-research ${what} halted: provider rate limit reached. Wait a moment and retry.

package/docs/ARCHITECTURE.md

@@ -203,6 +203,30 @@ Cross-session, cross-process state (active sessions, browser status, metrics) li
 `StateManagerService` (`src/infrastructure/state/`), which serializes concurrent writes
 with file-based locking (`FileLockService`).
 
+### Concurrent runs (the run cap)
+
+Every pi-research process on a machine — CLI, agent skill, pi extension, SDK — shares
+one leader-elected browser pool and one embedding model. Letting an unbounded number of
+research runs onto that shared pool does not slow runs down gracefully; it saturates the
+priority queue and degrades *all* of them at once.
+
+`ResearchRunSemaphore` (`src/infrastructure/research-run-semaphore.ts`) therefore gates
+every `runResearch()` entry on one of N slots, realized as N well-known lock files in the
+state directory and coordinated by the same `FileLockService`. Because slot ownership is
+recorded as PID + process start time, a slot held by a crashed run is reclaimed
+immediately by the next acquire, while a *live* holder is never stolen — a legitimate run
+holds its slot for minutes, and stealing it would admit the (N+1)th run the cap exists to
+prevent.
+
+Runs beyond the cap **queue** rather than fail: the acquire polls until a slot frees,
+announcing itself once through the observer (`onRunQueued`, surfaced by the CLI as
+`• queued: …`) so a waiting run is never mistaken for a hung one. Only if nothing frees
+within the whole queue window does it raise `ResearchRunCapacityError` — a temporary
+condition the CLI reports as exit code `75`, distinct from a crash. The cap fails *open*
+on any internal or IO error, so a fault in the semaphore itself can never prevent research
+from running. Both the cap and the queue window are configurable
+(`PI_RESEARCH_MAX_CONCURRENT_RUNS`, `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS`).
+
 ### TUI
 
 The live progress panel uses `@earendil-works/pi-tui`, which handles terminal state

package/docs/CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.1.0] - 2026-08-04
+
+### Changed
+- **Concurrent runs over the cap now QUEUE instead of failing — the run-cap default is a queue, not a fail-fast.** v1.0.11 added the machine-wide run cap that stopped parallel runs saturating the shared browser/embedding pool, but it shipped with `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS` defaulting to `0`. That made contention *fatal*: the cap was enforced by rejecting the run outright, in about two seconds, rather than by serializing it. Verified end-to-end before the change — four simultaneous skill runs against the default cap produced two clean reports and two hard failures, with the two rejected queries simply lost. That is the wrong trade for a semaphore guarding a shared resource: an extra run is a legitimate request to do work, not an error. The default acquire wait is now **600000 ms (10 minutes)**, so over-cap runs wait for a slot and then execute normally; a depth-1 run measured at ~2–3 minutes, so this absorbs a backlog of roughly three further runs per slot. `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS=0` restores the old strict fail-fast. The cap itself is raised from **2 to 3** concurrent runs. The agent skill's recommended command timeouts are raised to match (depth 1 `1500000` ms, depth 2 `1800000`, depth 3 `2400000`), since a queued run legitimately takes longer than an unqueued one and the old budgets no longer bounded the worst case.
+- **A queued run now says so** (`• queued: N concurrent run(s) already active — waiting up to Ns for a slot…`) instead of sitting silent for the whole acquire wait. Silence for up to ten minutes is indistinguishable from a hang, which would have made the queueing change above *look* like a regression. The notice is emitted exactly once per run, via a new optional `ResearchObserver.onRunQueued(slots, maxWaitMs)`; because the queue happens before any orchestrator exists, the notification is dispatched to whichever observer shape the caller supplied — a `ResearchObserver` instance (SDK / pi extension) or the `HeadlessObserverOptions` bag carrying only `onProgress` (CLI / agent skill). A throwing front-end callback can never fail the run it is reporting on.
+
+### Added
+- **Distinct exit code `75` (`EX_TEMPFAIL`) for run-cap exhaustion.** Capacity exhaustion previously exited `70` — the same code as a genuine crash — so a calling agent could not tell "this machine is busy with other healthy runs, try again shortly" from "this run is broken", and the agent skill's own exit-code table told it to treat `70` as a transient error worth one immediate retry (which, under a 0 ms acquire, failed again instantly). It now exits `75`, documented in `SKILL.md` with explicit guidance not to reconfigure anything and not to retry in a tight loop, and `--json` output carries `retryable: true`. The classification is matched on the error name so it survives bundling into `dist/cli.mjs`, and is checked ahead of the credentials heuristic — the capacity message names `PI_RESEARCH_MAX_CONCURRENT_RUNS`, and the substring heuristic would otherwise have been at risk of reading a busy machine as a misconfigured one. Capacity exhaustion is also reported without a stack trace, since a routine "busy" state printed as a stack reads as a crash.
+
+### Fixed
+- **A browser-pool leader exiting could destroy a concurrent run outright — the retry reconnected to the dead leader instead of re-electing.** The browser pool is hosted by one elected leader process shared by every pi-research run on the machine, so when that leader finishes its own run and tears down its workers, every other run with work in flight takes `Worker pool is shutting down` on *all* outstanding tasks at once. The recovery path handled this incorrectly: on a pool-shutdown error it only waited for the pool to go idle and **never dropped the cached scheduler handle**, so the single retry called `getScheduler()`, got the same cached client still pointing at the dead leader's port (visible in logs as a retry "Connecting to existing scheduler" with an *identical* version id), and failed the same way. The generic transient-socket branch did the right thing and forced a re-resolve; the pool-shutdown branch — the case where the leader is provably going away — did not. Reproduced in a real six-way concurrent run: a run acquired its slot in the same second the leader shut down, all 20 queries of its search burst failed, and the run aborted with `Search completely failed: all 20 queries encountered worker errors` and the tell-tale `60 tracked error(s) … "Worker pool is shutting down" ×60` — roughly two seconds *before* a fresh leader finished being elected and served everyone else normally. A handover now drops the stale handle and forces re-resolution (joining the new leader, or winning the election itself), and gets its own retry budget of 3 attempts with a ~1.2 s backoff, sized to a real election rather than to the 100–500 ms jitter meant for a socket blip. The budget is tracked separately from the generic retry so a handover cannot be starved by, or consume, the single transient-socket attempt; genuinely unrecoverable pool loss still gives up rather than retrying forever. This failure mode became far more likely with over-cap queueing, since a queued run is admitted at exactly the moment another run finishes — which is exactly when the leader may be tearing down.
+- **The run-cap's two environment variables were entirely undocumented.** `PI_RESEARCH_MAX_CONCURRENT_RUNS` and `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS` shipped in v1.0.11 and appeared in neither `docs/CONFIGURATION.md` nor `.env.example` — the latter described in the docs as "the canonical, exhaustive list" — even though the capacity error message explicitly instructed the user to override the first of them. Both are now documented in both places, and the run cap has an `ARCHITECTURE.md` section covering slot ownership, crash reclamation, queueing and fail-open behaviour.
+- **`docs/SDK.md` recommended a workaround that the run cap had since made unreliable** — it told callers to "use a separate process per concurrent run" with no mention that separate processes are themselves bounded machine-wide, so following the documented advice past the cap hit a hard failure. It now documents the cap, the queue window, `ResearchRunCapacityError`, exit code `75`, and `onRunQueued`.
+
 ## [1.0.12] - 2026-08-03
 
 ### Fixed

package/docs/CONFIGURATION.md

@@ -76,6 +76,8 @@ Research
 | `PI_RESEARCH_MAX_FAILED_RESEARCHERS` | `2` | 1–10 | Unique researcher failures that abort the whole run. Raise to let slower, still-in-flight researchers finish before giving up. |
 | `PI_RESEARCH_WORKER_THREADS` | `4` | 1–10 | Browser worker processes. Higher = more throughput, more CPU/RAM. |
 | `PI_RESEARCH_WORKER_CONCURRENCY` | `2` | 1–10 | Tasks per worker process. |
+| `PI_RESEARCH_MAX_CONCURRENT_RUNS` | `3` | ≥1 | Machine-wide cap on research runs executing at the same time, across **every** process (CLI, agent skill, pi extension, SDK). Runs beyond the cap queue rather than fail. All concurrent runs share one leader-elected browser/embedding pool, so oversubscribing it degrades every run at once. |
+| `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS` | `600000` | ≥0 | How long a run queues for a free slot before failing with "maximum concurrent research runs reached" (CLI exit `75`). `0` = fail immediately instead of queueing. |
 | `PI_RESEARCH_MODEL` | _(pi: session model; CLI/skill: required)_ | — | The model research runs on. **Required for the standalone CLI / agent skill** — they use only this configured model (never the model selected inside the pi extension) and refuse to start without one (the CLI's per-run `--model` flag also satisfies this). On the SDK it selects the session model when no `model` option is given. In the pi extension it overrides researcher sub-agents and knowledge synthesis, while the coordinator and evaluator keep using the session model. Accepts `provider/id` or a bare model id. |
 | `PI_RESEARCH_DISABLED_TOOLS` | _(none)_ | — | Comma-separated research tools to disable for a run (`search`, `scrape`, `security_search`, `stackexchange`, `youtube_transcript`, `grep`). Removed from every researcher's toolset and named in the coordinator/evaluator prompt. |
 | `PI_RESEARCH_REPORT_EXPORT_ENABLED` (TUI) | `false` | — | Front-ends write a Markdown report to disk and surface its path. |

package/docs/SDK.md

@@ -85,6 +85,15 @@ report caused by a sparse topic apart from one where most researchers failed). B
 > Concurrency: a single initialized SDK instance runs one research call at a time.
 > Overlapping `runDeepResearch`/`runQuickResearch` calls on the same instance throw
 > — run them sequentially, or use a separate process per concurrent run.
+>
+> Separate processes are additionally bounded by a **machine-wide run cap**
+> (`PI_RESEARCH_MAX_CONCURRENT_RUNS`, default 3) covering every pi-research process
+> on the host, because they all share one leader-elected browser/embedding pool. A
+> run over the cap queues for up to `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS` (default
+> 10 min) and only then rejects with `ResearchRunCapacityError` — a temporary
+> "try again shortly" condition, surfaced by the CLI as exit code `75`. Supply an
+> observer with `onRunQueued(slots, maxWaitMs)` to tell a waiting user the run is
+> queued rather than hung.
 
 The SDK does not write report files. Report export is a front-end concern — the pi
 extension and the CLI / agent skill do it when `PI_RESEARCH_REPORT_EXPORT_ENABLED=true`.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@lincoln504/pi-research",
-  "version": "1.0.12",
+  "version": "1.1.0",
   "description": "A set of web research tools (search, scrape, and others) and research agent system that uses them safely and efficiently.",
   "type": "module",
   "private": false,

package/skills/pi-research/SKILL.md

@@ -2,7 +2,7 @@
 name: pi-research
 description: "Multi-agent web research for any coding agent. Use for anything requiring current information from the internet — news, trends, product/service/company facts, statistics, people, places, events, APIs, library behavior, or 'what is X / how does X work' questions. Also use to search a local knowledge store of previously researched findings before doing live research. Powered by the pi-research engine (search, stealth scraping, security databases, Stack Exchange). Not for reading local files, running project commands, or analyzing this repository's own source."
 license: MIT
-metadata: { "author": "Lincoln Deen", "version": "1.0.12", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
+metadata: { "author": "Lincoln Deen", "version": "1.1.0", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
 allowed-tools: Bash(node:*)
 ---
 
@@ -22,8 +22,16 @@ node "<SKILL_DIR>/scripts/run.mjs" status    [--json]
 - **stderr** = progress + errors — relay errors, suppress routine progress.
 - **Always set a generous command timeout** so a stalled run can't block forever, but
   leave plenty of headroom — a real run should finish well inside it:
-  `knowledge` 600000ms · `research --depth 1` 900000ms · `--depth 2` 1200000ms · `--depth 3` 1800000ms.
+  `knowledge` 600000ms · `research --depth 1` 1500000ms · `--depth 2` 1800000ms · `--depth 3` 2400000ms.
+  These already include headroom for the run to sit **queued** behind other runs on a
+  busy machine (see below); don't shorten them on the assumption a slot is free.
 - **Background**: append `&` and keep the PID to work while a run completes; otherwise it runs foreground (blocks). Timeouts apply either way.
+- **Several at once is fine.** Up to 3 runs execute concurrently machine-wide (shared
+  with any other tool or agent on this machine); further runs **queue** for a slot and
+  report `• queued: …` on stderr — that is normal, not a stall. Only if nothing frees
+  up within the queue window does a run exit `75` (see *Exit codes*). Prefer one
+  `research` call with a well-scoped query over splitting one topic into many parallel
+  calls — depth 1 already parallelises internally.
 
 ## Workflow
 
@@ -57,6 +65,7 @@ node "<SKILL_DIR>/scripts/run.mjs" status    [--json]
 | `0` | Success | Read stdout; cite sources; relay any "saved to: <path>" line. |
 | `78` on `knowledge` | Store disabled/unconfigured | Silent; proceed to `research`. |
 | `78` on `research` | Engine missing or no model/key | **Stop. Relay full stderr** (it shows where to configure). Don't retry until fixed. |
+| `75` | Machine at capacity — other research runs are already using every slot | **Nothing is broken.** Wait for an in-flight run to finish, then retry this query once. Don't reconfigure anything and don't retry in a tight loop. |
 | `70` | Runtime error | Relay stderr; suggest one retry for transient failures (network, rate limit). |
 | `64` | Bad arguments | Fix args; don't surface unless the query itself is malformed. |
 

package/src/cli.ts

@@ -76,6 +76,14 @@ export const EXIT = {
   CONFIG: 78,
   /** Something went wrong at runtime (network, provider, internal). */
   SOFTWARE: 70,
+  /**
+   * Temporary failure — the request was well-formed and the install is healthy,
+   * but the machine is at capacity right now (all research run slots held by
+   * live runs). Distinct from SOFTWARE so a calling agent can tell "busy, try
+   * again shortly" apart from "this run crashed" and back off instead of
+   * re-running a broken command.
+   */
+  TEMPFAIL: 75,
 } as const;
 
 // ---------------------------------------------------------------------------
@@ -108,6 +116,12 @@ function makeProgressObserver(): HeadlessObserverOptions {
         case 'planning_success':
           toStderr(`  • plan: ${data?.plan?.researchers?.length ?? '?'} researcher(s)\n`);
           break;
+        case 'run_queued':
+          toStderr(
+            `  • queued: ${data?.slots ?? '?'} concurrent run(s) already active — waiting up to ` +
+              `${Math.round((data?.maxWaitMs ?? 0) / 1000)}s for a slot…\n`,
+          );
+          break;
         case 'round_start':
           toStderr(`  • round ${data?.round}\n`);
           break;
@@ -829,9 +843,19 @@ export async function reportError(err: unknown, what: string, json?: boolean): P
   // research-stop — regardless of real cause — as a config error (exit 78).
   const isTaggedResearchStop = (err as { code?: unknown } | undefined)?.code === RESEARCH_STOPPED_ERROR_CODE;
 
+  // Run-cap exhaustion is an operational state, not a fault: every slot is held by
+  // another *healthy* live run. Matched on the error name rather than `instanceof`
+  // so it survives bundling (dist/cli.mjs) and any duplicate module instance.
+  // Checked ahead of the substring heuristics below rather than relying on the
+  // current wording of the message: capacity text is config-shaped (it names two
+  // PI_RESEARCH_* env vars), so an exact classification must not depend on that
+  // message never drifting into one of the credential keywords. Misreading a busy
+  // machine as a setup problem would send the user to reconfigure a working install.
+  const isCapacity = err instanceof Error && err.name === 'ResearchRunCapacityError';
+
   // Auth / model / config-shaped errors → CONFIG exit + config block.
   const isConfigError =
-    !isTaggedResearchStop && (
+    !isTaggedResearchStop && !isCapacity && (
     lower.includes('no llm model available') ||
     lower.includes('not found in pi') ||
     lower.includes('no api key') ||
@@ -846,12 +870,19 @@ export async function reportError(err: unknown, what: string, json?: boolean): P
   // classifying that as a setup problem sends the user to reconfigure a
   // perfectly working key instead of waiting out the limit.
   const isRateLimit = lower.includes('429') || lower.includes('rate limit') || lower.includes('too many requests');
-  const exitCode = isConfigError && !isRateLimit ? EXIT.CONFIG : EXIT.SOFTWARE;
+  const exitCode = isCapacity
+    ? EXIT.TEMPFAIL
+    : isConfigError && !isRateLimit
+      ? EXIT.CONFIG
+      : EXIT.SOFTWARE;
 
   // --json: emit a structured error on stdout so a machine consumer never has to
   // special-case plain-text stderr. Mirrors the { ok: true, ... } success shape.
   if (json) {
     const payload: Record<string, unknown> = { ok: false, error: msg, exitCode };
+    // Machine-readable "this will succeed later" flag, so a consumer can back off
+    // and retry without string-matching the human message.
+    if (isCapacity || isRateLimit) payload['retryable'] = true;
     if (process.env['PI_RESEARCH_DEBUG'] === 'true' && err instanceof Error && err.stack) {
       payload['stack'] = err.stack;
     }
@@ -859,6 +890,13 @@ export async function reportError(err: unknown, what: string, json?: boolean): P
     return exitCode;
   }
 
+  // Capacity exhaustion is a healthy-machine "busy" state — report it as such and
+  // never with a stack trace, which would read as a crash for a routine condition.
+  if (isCapacity) {
+    toStderr(`\nError: pi-research ${what} did not start: ${msg}\n`);
+    return exitCode;
+  }
+
   // Rate limits are operational, not fatal setup problems — checked FIRST.
   if (isRateLimit) {
     toStderr(`\nError: pi-research ${what} halted: provider rate limit reached. Wait a moment and retry.\n  ${msg}\n`);

package/src/core/interfaces/observer-interfaces.ts

@@ -18,6 +18,15 @@ export interface HeadlessObserverOptions {
 export interface ResearchObserver {
   onStart?(query: string, complexity: number): void;
 
+  /**
+   * Fired once when the run cannot start immediately because every concurrent-run
+   * slot is held, and it is about to queue. Without this a queued run is silent
+   * for as long as the acquire wait allows, which is indistinguishable from a hang.
+   * @param slots  the machine-wide concurrent-run cap
+   * @param maxWaitMs how long the run will queue before giving up
+   */
+  onRunQueued?(slots: number, maxWaitMs: number): void;
+
   // Coordinator/Planning phase
   onPlanningStart?(attempt: number): void;
   onPlanningProgress?(status: string): void;

package/src/infrastructure/browser/browser-error-utils.ts

@@ -90,9 +90,18 @@ export function isServerDrainingError(error: unknown): boolean {
 
 /**
  * Check if an error is specifically a pool-shutdown / pool-drain error.
- * For these errors the scheduler itself is fine — only the WorkerPool is temporarily
- * draining. The correct recovery is to wait for the drain to finish and retry,
- * NOT to call forceSchedulerRestart (which would start another drain cycle).
+ *
+ * These are never counted against the circuit breaker: the pool recovers on its own.
+ *
+ * NOTE: this was once documented as "wait for the drain to finish and retry, NOT
+ * forceSchedulerRestart". That is right only when the draining pool is *local*. In the
+ * common case the pool belongs to another process — the elected leader — which is
+ * exiting, so waiting alone leaves the cached client pointed at a dead port and the
+ * retry fails identically (this cost whole search bursts; see the v1.1.0 changelog).
+ * Recovery must also drop the cached scheduler handle so the next call re-resolves the
+ * new leader; `recoverFromLeaderHandover` in task-execution-service.ts does exactly
+ * that, using `forceSchedulerRestart(false)` whose liveness probe and in-progress guard
+ * prevent the restart churn the old note was warning about.
  */
 export function isPoolShutdownError(error: unknown): boolean {
     if (!error || typeof error !== 'object') return false;

package/src/infrastructure/browser/task-execution-service.ts

@@ -22,6 +22,46 @@ import type { ISchedulerFactory, IScheduler } from '../../core/scheduler-factory
 let lastRestartTime = 0;
 const RESTART_COOLDOWN_MS = 10000;
 
+/**
+ * Extra attempts reserved for a *leader handover* — the elected browser-pool leader
+ * exiting while this process still has work in flight ("Worker pool is shutting
+ * down" / "draining").
+ *
+ * This has its own budget because it is not a random socket blip: it is a known,
+ * self-healing window that lasts as long as a fresh election takes (measured at
+ * ~4 s on a loaded machine), and it hits *every* in-flight task of the affected
+ * run at once. With only the single generic retry, a whole 20-query search burst
+ * could be wiped out by one leader exiting — the run then failed outright with
+ * "all N queries encountered worker errors", seconds before the new leader came up.
+ * Spending these attempts is strictly cheaper than losing the run.
+ */
+const LEADER_HANDOVER_RETRIES = 3;
+/** Backoff between handover attempts — long enough for an election to complete. */
+const HANDOVER_BACKOFF_MS = 1200;
+
+/**
+ * Recover from a leader handover before retrying.
+ *
+ * Critically this DROPS the cached scheduler handle. A pool-shutdown error means
+ * the leader we are connected to is going away, so the cached client points at a
+ * corpse; simply waiting and retrying reconnects to the same dead port (observed
+ * as a retry logging "Connecting to existing scheduler" with the *same* version id
+ * and failing identically). Clearing it forces the next `getScheduler()` to
+ * re-resolve — joining the new leader, or winning the election itself.
+ *
+ * `forceSchedulerRestart(false)` keeps its own liveness probe and in-progress
+ * guard, so concurrent callers coalesce instead of stampeding.
+ */
+async function recoverFromLeaderHandover(container: ServiceContainer): Promise<void> {
+    logger.warn('[BrowserManager] Leader handover detected (pool draining) — dropping stale scheduler handle and re-resolving...');
+    await forceSchedulerRestart(false, container).catch((err) =>
+        logger.debug('[BrowserManager] Scheduler re-resolve during handover failed (non-fatal):', err),
+    );
+    await waitForBrowserPoolIdle(15000).catch((err) =>
+        logger.debug('Wait for browser idle timed out or failed:', err),
+    );
+}
+
 /**
  * Get the scheduler from the factory, optionally using a specific container.
  */
@@ -56,7 +96,8 @@ export async function runBrowserTask<T>(
     config?: Config,
     signal?: AbortSignal,
     retries = 1,
-    container: ServiceContainer = getServiceContainer()
+    container: ServiceContainer = getServiceContainer(),
+    handoverRetries = LEADER_HANDOVER_RETRIES,
 ): Promise<T> {
     if (signal?.aborted) throw new Error('Aborted');
 
@@ -84,6 +125,23 @@ export async function runBrowserTask<T>(
     } catch (error: unknown) {
         if (signal?.aborted || (error instanceof Error && error.message === 'Aborted')) throw new Error('Aborted', { cause: error });
 
+        // A leader handover gets its own budget: it is a recoverable, seconds-long
+        // window that strikes every in-flight task at once, so it must not be
+        // charged against (or limited by) the single generic transient retry.
+        const isHandover = isPoolShutdownError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error);
+        if (isHandover && handoverRetries > 0) {
+            errorTracker.trackError(error, {
+                component: 'browser-manager',
+                operation: type,
+                taskType: type,
+                errorType: 'leader_handover',
+            });
+            logger.warn(`[BrowserManager] Leader handover during ${type} task (handover attempts left: ${handoverRetries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
+            await recoverFromLeaderHandover(container);
+            await new Promise((resolve) => setTimeout(resolve, HANDOVER_BACKOFF_MS + Math.floor(Math.random() * 400)));
+            return runBrowserTask<T>(taskOrUrl, type, config, signal, retries, container, handoverRetries - 1);
+        }
+
         if (retries > 0 && isTransientSocketError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error)) {
             errorTracker.trackError(error, {
                 component: 'browser-manager',
@@ -92,9 +150,9 @@ export async function runBrowserTask<T>(
                 errorType: 'transient_socket_error',
             });
             logger.warn(`[BrowserManager] Transient socket error during ${type} task (retries left: ${retries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
-            
+
             if (isPoolShutdownError(error)) {
-                // Pool is temporarily draining — wait for the drain to finish.
+                // Handover budget already spent — fall back to a plain drain wait.
                 logger.warn(`[BrowserManager] Pool is draining — waiting for pool idle before retry...`);
                 await waitForBrowserPoolIdle(15000).catch((err) => logger.debug('Wait for browser idle timed out or failed:', err));
             } else {
@@ -120,7 +178,7 @@ export async function runBrowserTask<T>(
             // Retry with jitter (100-500ms) to prevent thundering herd
             const jitter = 100 + Math.floor(Math.random() * 400);
             await new Promise(resolve => setTimeout(resolve, jitter));
-            return runBrowserTask<T>(taskOrUrl, type, config, signal, retries - 1, container);
+            return runBrowserTask<T>(taskOrUrl, type, config, signal, retries - 1, container, handoverRetries);
         }
         throw error;
     }
@@ -143,9 +201,12 @@ export async function runBrowserHealthCheck(config?: Config, retries = 1, signal
                 errorType: 'transient_socket_error',
             });
             logger.warn(`[BrowserManager] Transient socket error during healthcheck (retries left: ${retries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
-            
+
             if (isPoolShutdownError(error)) {
-                await waitForBrowserPoolIdle(15000).catch((err) => logger.debug('Wait for browser idle timed out or failed:', err));
+                // Same stale-handle trap as the task paths: waiting alone leaves the
+                // cached client aimed at the departing leader, so the retry re-probes
+                // a corpse and reports the pool unhealthy during a routine handover.
+                await recoverFromLeaderHandover(container);
             } else {
                 // See runBrowserTask: don't force-clear remote leader state on a bare
                 // transient socket error — let the liveness probe inside
@@ -172,7 +233,7 @@ export async function runBrowserHealthCheck(config?: Config, retries = 1, signal
 /**
  * Run a worker search query with retry logic.
  */
-export async function runWorkerSearch(query: string, config?: Config, signal?: AbortSignal, retries = 1, sessionId?: string, container: ServiceContainer = getServiceContainer()): Promise<SearchResult[]> {
+export async function runWorkerSearch(query: string, config?: Config, signal?: AbortSignal, retries = 1, sessionId?: string, container: ServiceContainer = getServiceContainer(), handoverRetries = LEADER_HANDOVER_RETRIES): Promise<SearchResult[]> {
     if (signal?.aborted) throw new Error('Aborted');
     
     const breaker = sessionId ? getBrowserCircuitBreaker(sessionId) : browserCircuitBreaker;
@@ -186,6 +247,22 @@ export async function runWorkerSearch(query: string, config?: Config, signal?: A
     } catch (error: unknown) {
         if (signal?.aborted || (error instanceof Error && error.message === 'Aborted')) throw new Error('Aborted', { cause: error });
 
+        // See runBrowserTask: a leader handover wipes out an entire search burst at
+        // once, so it gets its own attempts rather than the single generic retry.
+        const isHandover = isPoolShutdownError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error);
+        if (isHandover && handoverRetries > 0) {
+            errorTracker.trackError(error, {
+                component: 'browser-manager',
+                operation: 'search',
+                query,
+                errorType: 'leader_handover',
+            });
+            logger.warn(`[BrowserManager] Leader handover during search (handover attempts left: ${handoverRetries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
+            await recoverFromLeaderHandover(container);
+            await new Promise((resolve) => setTimeout(resolve, HANDOVER_BACKOFF_MS + Math.floor(Math.random() * 400)));
+            return runWorkerSearch(query, config, signal, retries, sessionId, container, handoverRetries - 1);
+        }
+
         if (retries > 0 && isTransientSocketError(error) && !isTaskTimeoutError(error) && !isCloudflareBlockError(error)) {
             errorTracker.trackError(error, {
                 component: 'browser-manager',
@@ -194,7 +271,7 @@ export async function runWorkerSearch(query: string, config?: Config, signal?: A
                 errorType: 'transient_socket_error',
             });
             logger.warn(`[BrowserManager] Transient socket error during search (retries left: ${retries}): ${(error instanceof Error ? error.message : String(error)).substring(0, 100)}...`);
-            
+
             if (isPoolShutdownError(error)) {
                 await waitForBrowserPoolIdle(15000).catch((err) => logger.debug('Wait for browser idle timed out or failed:', err));
             } else {
@@ -214,7 +291,7 @@ export async function runWorkerSearch(query: string, config?: Config, signal?: A
 
             const jitter = 100 + Math.floor(Math.random() * 400);
             await new Promise(resolve => setTimeout(resolve, jitter));
-            return runWorkerSearch(query, config, signal, retries - 1, sessionId, container);
+            return runWorkerSearch(query, config, signal, retries - 1, sessionId, container, handoverRetries);
         }
         throw error;
     }

package/src/infrastructure/research-run-semaphore.ts

@@ -21,9 +21,10 @@
  *   configured as a *try-take* (`lockTimeout: 0`): it either wins the slot (free,
  *   or reclaimed from a *dead* owner) immediately or throws as soon as it sees a
  *   *live* owner. If every slot is held by a live process, the loop waits
- *   `pollDelayMs` and retries until `maxWaitMs` elapses, then rejects with
- *   {@link ResearchRunCapacityError} (fail-fast with a clear message rather than
- *   a 60-error degraded run).
+ *   `pollDelayMs` and retries until `maxWaitMs` elapses (by default
+ *   {@link DEFAULT_ACQUIRE_TIMEOUT_MS} — extra runs **queue** behind the in-flight
+ *   ones instead of failing), then rejects with {@link ResearchRunCapacityError}
+ *   rather than degrading into a 60-error run.
  * - **Release** unlinks the slot file via the captured slot service.
  * - **Crash recovery** is automatic and PID-reuse-safe: a slot held by a process
  *   that has died (killed, crashed, SIGKILLed by earlyoom) is reclaimed
@@ -69,9 +70,21 @@ import type { IService } from '../core/service-registry.ts';
 import type { IProcessLifecycle } from '../core/interfaces/process-interfaces.ts';
 
 /** Default maximum number of concurrent research runs across all processes. */
-export const DEFAULT_MAX_CONCURRENT_RUNS = 2;
-/** Default per-acquire wait before failing fast when all slots are busy (0 = fail-fast). */
-export const DEFAULT_ACQUIRE_TIMEOUT_MS = 0;
+export const DEFAULT_MAX_CONCURRENT_RUNS = 3;
+/**
+ * Default per-acquire wait before giving up when all slots are held by live runs.
+ *
+ * This is deliberately a *queue*, not a fail-fast. The cap exists to stop N runs
+ * saturating the shared browser/embedding pool — but an extra run is a legitimate
+ * request to do work, not an error, so the correct response to contention is to
+ * serialize it behind the in-flight runs rather than to fail it. With a depth-1 run
+ * measured at ~2-3 minutes, 10 minutes absorbs a backlog of roughly three further
+ * runs per slot while still resolving well inside the agent-skill's documented
+ * command timeouts (so a queued run never looks like a hang to the caller).
+ *
+ * Set `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS=0` to restore strict fail-fast.
+ */
+export const DEFAULT_ACQUIRE_TIMEOUT_MS = 600_000;
 /** Poll interval between full slot sweeps while waiting for a slot. */
 const POLL_DELAY_MS = 250;
 
@@ -84,9 +97,10 @@ export class ResearchRunCapacityError extends Error {
   readonly slots: number;
   constructor(slots: number, maxWaitMs: number) {
     super(
-      `Maximum concurrent research runs (${slots}) reached${maxWaitMs > 0 ? ` after waiting ${maxWaitMs}ms` : ''}; ` +
-        `wait for an in-progress run to finish and retry. ` +
-        `(Override with PI_RESEARCH_MAX_CONCURRENT_RUNS.)`,
+      `Maximum concurrent research runs (${slots}) reached${maxWaitMs > 0 ? ` and still busy after queueing for ${Math.round(maxWaitMs / 1000)}s` : ''}; ` +
+        `another run is still in progress. This is temporary — retry shortly. ` +
+        `(Raise the cap with PI_RESEARCH_MAX_CONCURRENT_RUNS, or the queue wait with ` +
+        `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS.)`,
     );
     this.name = 'ResearchRunCapacityError';
     this.slots = slots;
@@ -209,10 +223,17 @@ export class ResearchRunSemaphore implements IService {
    * @param signal aborts the wait. Without it a run started under a long
    *   `PI_RESEARCH_RUN_ACQUIRE_TIMEOUT_MS` would ignore Ctrl-C until the deadline
    *   passed, because the wait happens before any orchestrator exists to cancel.
+   * @param onQueued invoked at most once, when the first full sweep finds every
+   *   slot busy and the call is about to wait. Lets a front-end tell the user the
+   *   run is queued rather than hung. Never allowed to break the acquire.
    * @throws {ResearchRunCapacityError} if no slot frees within `maxWaitMs`.
    * @throws {Error} named `AbortError` if `signal` aborts while waiting.
    */
-  async acquire(maxWaitMs: number = this.defaultMaxWaitMs, signal?: AbortSignal): Promise<RunSlotAcquisition> {
+  async acquire(
+    maxWaitMs: number = this.defaultMaxWaitMs,
+    signal?: AbortSignal,
+    onQueued?: (slots: number, maxWaitMs: number) => void,
+  ): Promise<RunSlotAcquisition> {
     if (!this._initialized) await this.initialize();
     if (signal?.aborted) {
       metrics.increment('research_run_acquire_total', 1, { status: 'aborted' });
@@ -220,6 +241,7 @@ export class ResearchRunSemaphore implements IService {
     }
 
     const deadline = Date.now() + Math.max(0, maxWaitMs);
+    let announcedQueued = false;
     for (;;) {
       // Round-robin from a rotating start so contending processes don't synchronize on slot 0.
       const start = this.nextStartIndex;
@@ -238,6 +260,17 @@ export class ResearchRunSemaphore implements IService {
         metrics.increment('research_run_acquire_total', 1, { status: 'exhausted' });
         throw new ResearchRunCapacityError(this.maxSlots, maxWaitMs);
       }
+      // Announce the queue exactly once, on the first sweep that found nothing free.
+      // A throwing front-end callback must never fail the acquire.
+      if (!announcedQueued) {
+        announcedQueued = true;
+        logger.info(`[ResearchRunSemaphore] All ${this.maxSlots} run slot(s) busy — queueing for up to ${maxWaitMs}ms.`);
+        try {
+          onQueued?.(this.maxSlots, maxWaitMs);
+        } catch (err) {
+          logger.debug(`[ResearchRunSemaphore] onQueued callback threw: ${err instanceof Error ? err.message : String(err)}`);
+        }
+      }
       // Count the aborted outcome too, so acquired + exhausted + aborted accounts for
       // every acquire attempt. A cancel that vanished from the counter would make the
       // capacity metrics quietly under-report demand.

package/src/orchestration/headless-observer.ts

@@ -95,6 +95,10 @@ export class HeadlessObserver implements ResearchObserver {
     this.emit('start', { query, complexity });
   }
 
+  onRunQueued(slots: number, maxWaitMs: number): void {
+    this.emit('run_queued', { slots, maxWaitMs });
+  }
+
   onPlanningStart(attempt: number): void {
     this.emit('planning_start', { attempt });
   }

package/src/orchestration/research-orchestration-service.ts

@@ -108,17 +108,27 @@ export class ResearchOrchestrationService implements IResearchOrchestration {
 
     const researchStart = Date.now();
 
-    // Cross-process run-cap (Phase 1): acquire one of N slots so concurrent
-    // research runs can't saturate the shared leader-elected browser/embedding
-    // pool and degrade into ~60-error failures. Fail-OPEN on internal/IO errors
-    // (a semaphore bug must never break research); only a genuine
-    // ResearchRunCapacityError (all slots held by live processes) propagates,
-    // failing the run fast with a clear message. The slot is released in finally.
+    // Cross-process run-cap: acquire one of N slots so concurrent research runs
+    // can't saturate the shared leader-elected browser/embedding pool and degrade
+    // into ~60-error failures. Over-cap runs QUEUE for a slot (announced once via
+    // the observer) rather than failing, because an extra run is a legitimate
+    // request to do work, not an error. Fail-OPEN on internal/IO errors (a
+    // semaphore bug must never break research); only a genuine
+    // ResearchRunCapacityError — nothing freed within the whole queue window —
+    // propagates. The slot is released in finally.
     let runSlot: { slotIndex: number; release(): Promise<void> } | null = null;
     try {
       const container = tryGetServiceContainerFromCtx(ctx);
       const semaphore = await getService<ResearchRunSemaphore>(ServiceNames.RESEARCH_RUN_SEMAPHORE, ctx, container);
-      runSlot = await semaphore.acquire(undefined, signal);
+      runSlot = await semaphore.acquire(undefined, signal, (slots, maxWaitMs) => {
+        // The queue notice fires BEFORE any orchestrator exists, so `observer` is
+        // still exactly what the caller passed: either a ResearchObserver, or the
+        // HeadlessObserverOptions bag (only `onProgress`) that the CLI supplies and
+        // the orchestrator would normally wrap later. Dispatch to whichever shape
+        // is present — handling only the former silently drops the notice on the
+        // CLI, which is the very front-end that most needs it.
+        notifyRunQueued(observer, slots, maxWaitMs);
+      });
       logger.info(`[ResearchOrchestrationService] Acquired research run slot ${runSlot.slotIndex} (cap ${semaphore.getMaxSlots()}).`);
     } catch (err) {
       if (err instanceof ResearchRunCapacityError) throw err; // capacity exhausted → fail fast
@@ -583,3 +593,28 @@ export class ResearchOrchestrationService implements IResearchOrchestration {
     }
   }
 }
+
+/**
+ * Deliver the "your run is queued behind other runs" notice to whichever observer
+ * shape the caller supplied.
+ *
+ * This fires before the orchestrator normalizes the observer, so both forms are
+ * live here: a {@link ResearchObserver} instance (SDK / pi extension) or the
+ * {@link HeadlessObserverOptions} bag carrying only `onProgress` (the CLI and
+ * agent skill). A queued run is otherwise completely silent for the whole acquire
+ * wait, which is indistinguishable from a hang.
+ *
+ * Never throws: a front-end callback must not be able to fail the run it is
+ * merely reporting on.
+ */
+function notifyRunQueued(observer: unknown, slots: number, maxWaitMs: number): void {
+  try {
+    const obs = observer as
+      | { onRunQueued?: (s: number, w: number) => void; onProgress?: (e: string, d?: unknown) => void }
+      | undefined;
+    if (typeof obs?.onRunQueued === 'function') obs.onRunQueued(slots, maxWaitMs);
+    else if (typeof obs?.onProgress === 'function') obs.onProgress('run_queued', { slots, maxWaitMs });
+  } catch (err) {
+    logger.debug(`[ResearchOrchestrationService] run_queued notification threw: ${err instanceof Error ? err.message : String(err)}`);
+  }
+}