@lincoln504/pi-research 1.3.8 → 1.3.10

package/README.md

@@ -1,9 +1,9 @@
 <p align="center">
-  <img src="https://raw.githubusercontent.com/Lincoln504/pi-research/main/docs/media/banner.jpg" alt="pi-research: free unlimited web search and knowledge store for coding agents" width="100%" />
+  <img src="https://raw.githubusercontent.com/Lincoln504/pi-research/main/docs/media/banner.jpg" alt="pi-research: free unlimited web search and knowledge store for agents" width="100%" />
 </p>
 
 <p align="center">
-  <strong>Free unlimited web search &amp; knowledge store for coding agents</strong>
+  <strong>Free unlimited web search &amp; knowledge store for agents</strong>
 </p>
 
 <p align="center">
@@ -31,7 +31,7 @@ Search and scraping run locally through a stealth browser with no search provide
 
 ### Install
 
-As a [pi](https://github.com/earendil-works/pi) extension:
+As a [Pi](https://github.com/earendil-works/pi) Coding Agent extension:
 
 ```bash
 pi install npm:@lincoln504/pi-research
@@ -44,7 +44,7 @@ npm install -g @lincoln504/pi-research
 pi-research skill install
 ```
 
-In pi it works out of the box on your session's model and pi's configuration. Standalone use (agent skill, CLI, or [SDK](docs/SDK.md)) needs a model configured. See [Configuration](docs/CONFIGURATION.md). The first install pulls the stealth browser engine, which takes a few minutes.
+In pi it works out of the box on your session's model and pi's configuration. Standalone use ([agent skill](docs/AGENT-SKILL.md) or [SDK](docs/SDK.md)) needs a model configured. See [Configuration](docs/CONFIGURATION.md). The first install pulls the stealth browser engine, which takes a few minutes.
 
 ### How it works
 
@@ -56,14 +56,14 @@ Three depth levels (normal, deep, ultra) set the team size and number of rounds.
 
 ### Use cases
 
-- Researching for your [Pi](https://github.com/earendil-works/pi) Coding Agent. Plug and play with no API key needed.
+- Researching for your pi. Plug and play with no API key needed.
 - Researching from Claude Code, Codex, OpenClaw, or another coding agent while a cheaper or local model drives the run so it doesn't spend your main agent's budget.
 - Keeping a persistent and searchable knowledge store of findings, scoped globally or per project.
 - Building agent systems that find and read web content or populating a dataset of web sources.
 
 ### Why pi-research
 
-- **No quota and no monthly fee.** Most AI search providers cap free searches and then charge, with results served from *their* index on *their* servers. pi-research searches and scrapes locally through DuckDuckGo in a stealth browser.
+- **No quota and no monthly fee.** Most AI search providers cap free searches and then charge, with results served from their index on their servers. pi-research searches and scrapes locally through DuckDuckGo in a stealth browser.
 - **You own your index.** Every finding is written to a local [LanceDB](https://lancedb.com) knowledge store that seeds future runs, so repeat questions get faster and cheaper answers.
 - **Cited reports rather than raw dumps.** You get one sourced Markdown report instead of raw web content pasted into the conversation. Context efficient and easy to verify.
 - **Read-only by design.** The research agent cannot run shell commands or write, edit, or delete anything. It can read and search files, so treat it as you would any read-only agent with access to your machine.
@@ -74,7 +74,7 @@ Three depth levels (normal, deep, ultra) set the team size and number of rounds.
 - Node.js >= 22.19.0
 - An LLM with a 100k+ context window (bring your own key or use a local model)
 - Internet access on a residential IP, since search, scraping, and YouTube transcripts get bot-blocked from datacenter/VPS/cloud IPs
-- The [pi runtime](https://github.com/earendil-works/pi). The pi extension uses the host's copy while the standalone CLI and agent skill install it as a dependency.
+- The pi runtime. The pi extension uses the host's copy while the standalone CLI and agent skill install it as a dependency.
 - Cloudflare and similar systems block scraping on some sites. A run flags pages it can't reach and compensates with a wide pool of search results to cite.
 
 ### Channels

package/dist/cli.mjs

@@ -247,22 +247,36 @@ function redactSecrets(message) {
 function stripTerminalEscapes(message) {
   return message.replace(TERMINAL_ESCAPE_PATTERN, "").replace(CONTROL_CHARS_PATTERN, " ");
 }
-var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TOKEN_CHAR_PATTERN, REDACT_SCAN_EXTENSION_MAX, TERMINAL_ESCAPE_PATTERN;
+var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, B_LEFT, B_RIGHT, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TOKEN_CHAR_PATTERN, REDACT_SCAN_EXTENSION_MAX, TERMINAL_ESCAPE_PATTERN;
 var init_log_utils = __esm({
   "src/utils/log-utils.ts"() {
     "use strict";
     logContextStorage = new AsyncLocalStorage2();
     MAX_LOG_MESSAGE_LENGTH = 1e4;
     ANSI_PATTERN = /\x1b\[[0-9;]*[A-Za-z]/g;
-    URL_CREDENTIALS_PATTERN = /([a-z][a-z0-9+.-]*:\/\/)[^/\s:@]+:[^/\s:@]+@/gi;
-    COOKIE_HEADER_PATTERN = /\b(set[_-]?cookie|cookie)\b(["']?\s*[:=]\s*["']?)([^\r\n]+)/gi;
-    SENSITIVE_KV_PATTERN = /\b(api[_-]?key|apikey|access[_-]?token|auth[_-]?token|refresh[_-]?token|secret|client[_-]?secret|password|passwd|pwd|authorization|bearer|set[_-]?cookie|cookie|session[_-]?id|session|csrf[_-]?token|xsrf[_-]?token|private[_-]?key|key)\b(["']?\s*[:=]\s*["']?)([^\s"',&)]+)/gi;
-    KNOWN_TOKEN_PATTERN = /\b(sk-[A-Za-z0-9_-]{16,}|gh[posru]_[A-Za-z0-9]{16,}|github_pat_[A-Za-z0-9_]{20,}|AKIA[0-9A-Z]{16}|xox[baprs]-[A-Za-z0-9-]{10,})\b/g;
-    JWT_PATTERN = /\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g;
-    BASIC_AUTH_PATTERN = /\bBasic\s+[A-Za-z0-9+/]{16,}={0,2}/gi;
-    BEARER_AUTH_PATTERN = /\bBearer\s+[A-Za-z0-9._~+/-]+=*/gi;
-    PROVIDER_TOKEN_PATTERN = /\b(AIza[0-9A-Za-z_-]{35}|ya29\.[0-9A-Za-z_-]{20,}|[sp]k_(?:live|test)_[0-9A-Za-z]{16,})\b/g;
-    LONG_HEX_SECRET_PATTERN = /\b[0-9a-fA-F]{32,}\b/g;
+    URL_CREDENTIALS_PATTERN = /([a-z][a-z0-9+.-]{0,31}:\/\/)[^/\s:@]+:[^/\s:@]+@/gi;
+    B_LEFT = "(?<![A-Za-z0-9])";
+    B_RIGHT = "(?![A-Za-z0-9])";
+    COOKIE_HEADER_PATTERN = new RegExp(
+      `${B_LEFT}(set[_-]?cookie|cookie)${B_RIGHT}(["']?\\s*[:=]\\s*["']?)([^\\r\\n]+)`,
+      "gi"
+    );
+    SENSITIVE_KV_PATTERN = new RegExp(
+      `${B_LEFT}(api[_-]?key|apikey|access[_-]?token|auth[_-]?token|refresh[_-]?token|secret|client[_-]?secret|password|passwd|pwd|authorization|bearer|set[_-]?cookie|cookie|session[_-]?id|session|csrf[_-]?token|xsrf[_-]?token|private[_-]?key|key)${B_RIGHT}(["']?\\s*[:=]\\s*["']?)([^\\s"',&)]+)`,
+      "gi"
+    );
+    KNOWN_TOKEN_PATTERN = new RegExp(
+      `${B_LEFT}(sk-[A-Za-z0-9_-]{16,}|gh[posru]_[A-Za-z0-9]{16,}|github_pat_[A-Za-z0-9_]{20,}|AKIA[0-9A-Z]{16}|xox[baprs]-[A-Za-z0-9-]{10,})${B_RIGHT}`,
+      "g"
+    );
+    JWT_PATTERN = new RegExp(`${B_LEFT}eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+`, "g");
+    BASIC_AUTH_PATTERN = new RegExp(`${B_LEFT}Basic\\s+[A-Za-z0-9+/]{16,}={0,2}`, "gi");
+    BEARER_AUTH_PATTERN = new RegExp(`${B_LEFT}Bearer\\s+[A-Za-z0-9._~+/-]+=*`, "gi");
+    PROVIDER_TOKEN_PATTERN = new RegExp(
+      `${B_LEFT}(AIza[0-9A-Za-z_-]{35}|ya29\\.[0-9A-Za-z_-]{20,}|[sp]k_(?:live|test)_[0-9A-Za-z]{16,})${B_RIGHT}`,
+      "g"
+    );
+    LONG_HEX_SECRET_PATTERN = new RegExp(`${B_LEFT}[0-9a-fA-F]{32,}${B_RIGHT}`, "g");
     CONTROL_CHARS_PATTERN = /[\x00-\x1f\x7f]/g;
     REDACT_SCAN_LENGTH = MAX_LOG_MESSAGE_LENGTH * 4;
     TOKEN_CHAR_PATTERN = /[A-Za-z0-9_\-./+=]/;
@@ -2648,6 +2662,7 @@ function saveConfig(config, scope = "local", cwd = process.cwd(), changedKeys) {
     PI_RESEARCH_LLM_THINKING_LEVEL: config.LLM_THINKING_LEVEL,
     PI_RESEARCH_PLANNING_MAX_TOKENS: String(config.PLANNING_MAX_TOKENS),
     PI_RESEARCH_SYNTHESIS_MAX_TOKENS: String(config.SYNTHESIS_MAX_TOKENS),
+    PI_RESEARCH_RESEARCHER_MAX_TOKENS: String(config.RESEARCHER_MAX_TOKENS),
     PI_RESEARCH_MIGRATION_STRATEGY: config.MIGRATION_STRATEGY,
     PI_RESEARCH_CONSOLE_LOG: String(config.CONSOLE_LOG),
     ...config.RESEARCH_MODEL ? { PI_RESEARCH_MODEL: config.RESEARCH_MODEL } : {},
@@ -2812,6 +2827,7 @@ function createConfig(env, processEnv) {
     LLM_THINKING_LEVEL: parseEnvEnum(e, "PI_RESEARCH_LLM_THINKING_LEVEL", ["off", "minimal", "low", "medium", "high"], DEFAULTS.LLM_THINKING_LEVEL),
     PLANNING_MAX_TOKENS: parseEnvNumber(e, "PI_RESEARCH_PLANNING_MAX_TOKENS", DEFAULTS.PLANNING_MAX_TOKENS, 1024, 131072),
     SYNTHESIS_MAX_TOKENS: parseEnvNumber(e, "PI_RESEARCH_SYNTHESIS_MAX_TOKENS", DEFAULTS.SYNTHESIS_MAX_TOKENS, 1024, 131072),
+    RESEARCHER_MAX_TOKENS: parseEnvNumber(e, "PI_RESEARCH_RESEARCHER_MAX_TOKENS", DEFAULTS.RESEARCHER_MAX_TOKENS, 1024, 131072),
     MIGRATION_STRATEGY: parseEnvEnum(e, "PI_RESEARCH_MIGRATION_STRATEGY", ["drop", "re-embed", "backup"], DEFAULTS.MIGRATION_STRATEGY),
     CONSOLE_LOG: parseEnvBool(e, "PI_RESEARCH_CONSOLE_LOG", DEFAULTS.CONSOLE_LOG),
     RESEARCH_MODEL: parseEnvString(e, "PI_RESEARCH_MODEL", DEFAULTS.RESEARCH_MODEL),
@@ -2999,6 +3015,15 @@ var init_config = __esm({
        *  the model's real maxTokens at call time.
        *  Not exposed in TUI — controlled via PI_RESEARCH_SYNTHESIS_MAX_TOKENS env var. */
       SYNTHESIS_MAX_TOKENS: Type.Number({ minimum: 1024, maximum: 131072, default: 32768 }),
+      /** Max OUTPUT tokens per researcher LLM call (deep and quick sub-agent sessions).
+       *  Without a cap the session inherits the model's full catalog ceiling as the
+       *  per-request max_tokens — ~230k for large-context models — which providers that
+       *  budget-check max_tokens up front (OpenRouter 402s against remaining credits)
+       *  reject outright on low balances, even though the actual response would cost
+       *  cents. Default 16384: researcher turns are tool calls plus a cited report, far
+       *  below this. Clamped to the model's real ceiling at session build time.
+       *  Not exposed in TUI — controlled via PI_RESEARCH_RESEARCHER_MAX_TOKENS env var. */
+      RESEARCHER_MAX_TOKENS: Type.Number({ minimum: 1024, maximum: 131072, default: 16384 }),
       /** LLM Model override for researcher sub-agents and knowledge synthesis.
        *  Format: provider/model-id (e.g. google/gemini-2.0-flash-001) or just model-id.
        *  When set, this overrides ctx.model for researcher sub-agents (both deep and quick)
@@ -3076,6 +3101,7 @@ var init_config = __esm({
       "PI_RESEARCH_LLM_THINKING_LEVEL",
       "PI_RESEARCH_PLANNING_MAX_TOKENS",
       "PI_RESEARCH_SYNTHESIS_MAX_TOKENS",
+      "PI_RESEARCH_RESEARCHER_MAX_TOKENS",
       "PI_RESEARCH_MIGRATION_STRATEGY",
       "PI_RESEARCH_CONSOLE_LOG",
       "PI_RESEARCH_MODEL",
@@ -3118,8 +3144,7 @@ function getMaxScrapeBatches(config) {
   }
 }
 function resolveExcludedTools(callerExclusions, configDisabled = []) {
-  const base = callerExclusions && callerExclusions.length > 0 ? callerExclusions : DEFAULT_EXCLUDED_TOOLS;
-  return [.../* @__PURE__ */ new Set([...base, ...configDisabled])];
+  return [.../* @__PURE__ */ new Set([...DEFAULT_EXCLUDED_TOOLS, ...callerExclusions ?? [], ...configDisabled])];
 }
 var MAX_GATHERING_CALLS, MAX_GREP_CALLS, MAX_SCRAPE_URLS, MAX_TEAM_SIZE_LEVEL_1, MAX_TEAM_SIZE_LEVEL_2, MAX_TEAM_SIZE_LEVEL_3, MAX_ROUNDS_LEVEL_1, MAX_ROUNDS_LEVEL_2, MAX_ROUNDS_LEVEL_3, MAX_EXTRA_ROUNDS_WITH_STEERING, OSV_TIMEOUT_MS, WEBGPU_PROBE_TIMEOUT_MS, MAX_QUERY_LENGTH, MAX_FILENAME_QUERY_LENGTH, MIN_QUERY_LENGTH, MAX_EXPORT_RETRIES, DEFAULT_MAX_RETRIES, DEFAULT_INITIAL_DELAY_MS, DEFAULT_MAX_DELAY_MS, DEFAULT_MODEL_CONTEXT_WINDOW, BATCH_2_DEFAULT_CONCURRENCY, MAX_SCRAPE_CONTENT_CHARS_PER_DOC, RESEARCHER_LAUNCH_DELAY_MS, MAX_QUERIES_PER_RESEARCHER_LEVEL_1, MAX_QUERIES_PER_RESEARCHER_LEVEL_2, MAX_QUERIES_PER_RESEARCHER_LEVEL_3, DEFAULT_EXCLUDED_TOOLS;
 var init_constants = __esm({
@@ -5131,14 +5156,16 @@ function unregisterGlobalEmbedder() {
 async function initializeDawnWebGPU() {
   if (dawnInitialized) return true;
   try {
-    const { create, globals } = await import("webgpu");
-    Object.assign(globalThis, globals);
-    Object.defineProperty(globalThis, "navigator", {
-      value: { gpu: create([]) },
-      writable: true,
-      configurable: true
+    const adapter = await logger.runCapturingStderr(async () => {
+      const { create, globals } = await import("webgpu");
+      Object.assign(globalThis, globals);
+      Object.defineProperty(globalThis, "navigator", {
+        value: { gpu: create([]) },
+        writable: true,
+        configurable: true
+      });
+      return await globalThis.navigator.gpu.requestAdapter({ powerPreference: "high-performance" });
     });
-    const adapter = await globalThis.navigator.gpu.requestAdapter({ powerPreference: "high-performance" });
     if (!adapter) {
       logger.warn("[embedder] No WebGPU adapter available \u2014 using CPU.");
       return false;
@@ -16808,21 +16835,21 @@ init_error_tracker();
 
 // src/utils/user-agent.ts
 var USER_AGENTS = [
-  // Chrome on Windows (updated 2025)
-  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
-  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
+  // Chrome on Windows
+  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
+  "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
   // Chrome on macOS
-  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
-  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
+  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
+  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
   // Chrome on Linux
-  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
-  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-  // Firefox on Windows (updated 2025)
-  "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0",
-  "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0",
+  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
+  "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
+  // Firefox on Windows
+  "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:153.0) Gecko/20100101 Firefox/153.0",
+  "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:152.0) Gecko/20100101 Firefox/152.0",
   // Firefox on macOS
-  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0",
-  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:138.0) Gecko/20100101 Firefox/138.0"
+  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:153.0) Gecko/20100101 Firefox/153.0",
+  "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0"
 ];
 
 // src/web-research/scraper-types.ts
@@ -17008,17 +17035,30 @@ function ssrfSafeLookup(hostname, options, callback, resolveImpl = dnsCb.lookup)
     return callback(null, safe[0].address, safe[0].family);
   });
 }
-var cachedSsrfDispatcher = void 0;
-async function getSsrfSafeDispatcher() {
-  if (cachedSsrfDispatcher !== void 0) return cachedSsrfDispatcher;
+function isSsrfBlockError(error) {
+  let e = error;
+  for (let depth = 0; depth < 5 && e instanceof Error; depth++) {
+    if (e.code === "ESSRFBLOCKED" || e.message.includes("not allowed")) {
+      return true;
+    }
+    e = e.cause;
+  }
+  return false;
+}
+var cachedSsrfFetcher = void 0;
+async function getSsrfSafeFetcher() {
+  if (cachedSsrfFetcher !== void 0) return cachedSsrfFetcher;
   try {
-    const { Agent: Agent2 } = await import("undici");
-    cachedSsrfDispatcher = new Agent2({ connect: { lookup: ssrfSafeLookup } });
+    const { Agent: Agent2, fetch: undiciFetch } = await import("undici");
+    cachedSsrfFetcher = {
+      fetch: undiciFetch,
+      dispatcher: new Agent2({ connect: { lookup: ssrfSafeLookup } })
+    };
   } catch (e) {
     logger.warn(`[Scrapers] undici unavailable; connect-time SSRF pinning disabled (request-time validation still active): ${e instanceof Error ? e.message : String(e)}`);
-    cachedSsrfDispatcher = null;
+    cachedSsrfFetcher = null;
   }
-  return cachedSsrfDispatcher;
+  return cachedSsrfFetcher;
 }
 function formatErrorWithCause(err) {
   try {
@@ -17068,6 +17108,7 @@ function isPrivateIpv6(ip) {
   if (normalized.startsWith("2002:")) return true;
   if (/^2001:0{0,4}:/.test(normalized) || normalized.startsWith("2001::")) return true;
   if (normalized.startsWith("64:ff9b:")) return true;
+  if (/^::(?:\d{1,3}(?:\.\d{1,3}){3}|[0-9a-f]{1,4}(?::[0-9a-f]{1,4})?)$/.test(normalized)) return true;
   if (normalized === "::" || normalized === "0:0:0:0:0:0:0:0") return true;
   return false;
 }
@@ -17267,7 +17308,7 @@ function decodeHtmlBody(bytes, contentTypeHeader) {
   }
 }
 function trackFetchLayerFailure(url, error) {
-  const isSsrf = error instanceof Error && error.message.includes("not allowed");
+  const isSsrf = isSsrfBlockError(error);
   errorTracker.trackError(error instanceof Error ? error : String(error), {
     component: "scrapers",
     operation: "fetch",
@@ -17279,7 +17320,8 @@ function trackFetchLayerFailure(url, error) {
 }
 async function scrapeWithFetch(url, signal) {
   await validateUrlForSSRF(url);
-  const dispatcher = await getSsrfSafeDispatcher();
+  const fetcher = await getSsrfSafeFetcher();
+  const doFetch = fetcher?.fetch ?? ((u, init) => fetch(u, init));
   const controller = new AbortController();
   const timeoutId = setTimeout(() => controller.abort(), FETCH_LAYER_TIMEOUT);
   safeUnref(timeoutId);
@@ -17296,16 +17338,17 @@ async function scrapeWithFetch(url, signal) {
     const MAX_REDIRECTS = 10;
     let currentUrl = url;
     let response;
+    const userAgent = getRandomUserAgent();
     for (let hop = 0; hop <= MAX_REDIRECTS; hop++) {
-      response = await fetch(currentUrl, {
+      response = await doFetch(currentUrl, {
         signal: controller.signal,
         redirect: "manual",
         headers: {
-          "User-Agent": getRandomUserAgent(),
+          "User-Agent": userAgent,
           "Accept": "text/html,application/xhtml+xml,application/pdf,*/*;q=0.8"
         },
-        // RequestInit's standard type omits undici's `dispatcher`; cast to attach it.
-        ...dispatcher ? { dispatcher } : {}
+        // Only ever attached alongside the fetch that honours it (see above).
+        ...fetcher ? { dispatcher: fetcher.dispatcher } : {}
       });
       if (response.status >= 300 && response.status < 400) {
         const location = response.headers.get("location");
@@ -17406,7 +17449,7 @@ async function scrapeWithFetch(url, signal) {
     metrics.observe("scrape_latency_ms", fetchDuration, { layer: "fetch", content_type: "html", status: "success" });
     return { source: "fetch", layer: "fetch", markdown };
   } catch (error) {
-    if (error instanceof Error && error.message.includes("not allowed")) {
+    if (isSsrfBlockError(error)) {
       metrics.increment("scrape_operations_total", 1, { layer: "fetch", status: "ssrf_blocked" });
       metrics.observe("scrape_latency_ms", Date.now() - fetchStart, { layer: "fetch", status: "ssrf_blocked" });
       throw error;
@@ -21506,6 +21549,8 @@ async function createResearcherSession(options) {
     cwd
   });
   logger.info(`[Researcher] Using model for researcher ${researcherId}: ${modelToUse.provider}/${modelToUse.id}`);
+  const outputCap = config?.RESEARCHER_MAX_TOKENS ?? 16384;
+  const sessionModel = modelToUse.maxTokens > outputCap ? { ...modelToUse, maxTokens: outputCap } : modelToUse;
   try {
     const tracker = new ToolUsageTracker(createDefaultToolLimits(config));
     const globalLinks = updateGlobalLinks || (() => {
@@ -21546,7 +21591,7 @@ async function createResearcherSession(options) {
       sessionManager: SessionManager.inMemory(),
       // Each researcher gets its own isolated session
       settingsManager: researcherSettings,
-      model: modelToUse,
+      model: sessionModel,
       // pi 0.80.8 replaced the SDK's `modelRegistry`/`authStorage` session options
       // with the async `modelRuntime`. Pass the runtime backing OUR registry when
       // this run built one (standalone CLI/SDK): it carries any explicit API key
@@ -21577,7 +21622,7 @@ async function createResearcherSession(options) {
         throw new Error("Session creation returned invalid result");
       }
       sessionRef.session = result.session;
-      return { session: result.session, resolvedModel: modelToUse };
+      return { session: result.session, resolvedModel: sessionModel };
     } catch (tailError) {
       const session = result?.session;
       if (session && typeof session.abort === "function") {
@@ -21600,7 +21645,7 @@ async function createResearcherSession(options) {
 
 // src/orchestration/researcher-executor.ts
 init_text_utils();
-init_llm();
+init_llm_usage();
 init_logger();
 init_metrics();
 init_service_interfaces();
@@ -22036,12 +22081,13 @@ ${prompt}`);
         if (msg2?.["role"] !== "assistant") return;
         const rawUsage = msg2["usage"];
         if (rawUsage) {
-          const { tokens, cost } = extractUsage(resolvedModel, rawUsage);
+          const { tokens, cost } = recordLlmUsage(resolvedModel, rawUsage, {
+            component: "researcher",
+            complexity,
+            observer: { onTokensConsumed: (t, c) => safeObserve(() => observer?.onTokensConsumed?.(t, c)) }
+          });
           if (tokens > 0 || cost > 0) {
-            metrics.increment("llm_tokens_total", tokens, { component: "researcher", complexity: String(complexity) });
-            metrics.increment("llm_cost_total", cost, { component: "researcher", complexity: String(complexity) });
             safeObserve(() => observer?.onResearcherProgress?.(id, void 0, tokens, cost));
-            safeObserve(() => observer?.onTokensConsumed?.(tokens, cost));
           }
         }
       } else if (event.type === "tool_execution_start") {
@@ -22168,7 +22214,7 @@ ${responseText}`);
 
 // src/orchestration/quick-research-orchestrator.ts
 init_prompts();
-init_llm();
+init_llm_usage();
 init_logger();
 init_config();
 init_text_utils();
@@ -22449,12 +22495,13 @@ ${prompt}`);
           if (msg2?.["role"] !== "assistant") return;
           const rawUsage = msg2["usage"];
           if (rawUsage) {
-            const { tokens, cost } = extractUsage(resolvedModel, rawUsage);
+            const { tokens, cost } = recordLlmUsage(resolvedModel, rawUsage, {
+              component: "quick_researcher",
+              complexity: 0,
+              observer
+            });
             if (tokens > 0 || cost > 0) {
-              metrics.increment("llm_tokens_total", tokens, { component: "quick_researcher", complexity: "0" });
-              metrics.increment("llm_cost_total", cost, { component: "quick_researcher", complexity: "0" });
               observer?.onResearcherProgress?.("quick", void 0, tokens, cost);
-              observer?.onTokensConsumed?.(tokens, cost);
             }
           }
         } else if (event.type === "tool_execution_start") {
@@ -25674,7 +25721,7 @@ function parseArgs(argv) {
       }
       if (a === "--depth") {
         const v = rest[++i];
-        const n = v === void 0 ? NaN : parseInt(v, 10);
+        const n = v !== void 0 && /^\d+$/.test(v.trim()) ? Number(v.trim()) : NaN;
         if (isNaN(n) || n < 0 || n > 3) throw new UsageError("--depth must be an integer 0\u20133.");
         depth = n;
       } else if (a === "--model") {

package/dist/thread-worker.mjs

@@ -77,22 +77,36 @@ function redactSecrets(message) {
 function stripTerminalEscapes(message) {
   return message.replace(TERMINAL_ESCAPE_PATTERN, "").replace(CONTROL_CHARS_PATTERN, " ");
 }
-var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TOKEN_CHAR_PATTERN, REDACT_SCAN_EXTENSION_MAX, TERMINAL_ESCAPE_PATTERN;
+var logContextStorage, MAX_LOG_MESSAGE_LENGTH, ANSI_PATTERN, URL_CREDENTIALS_PATTERN, B_LEFT, B_RIGHT, COOKIE_HEADER_PATTERN, SENSITIVE_KV_PATTERN, KNOWN_TOKEN_PATTERN, JWT_PATTERN, BASIC_AUTH_PATTERN, BEARER_AUTH_PATTERN, PROVIDER_TOKEN_PATTERN, LONG_HEX_SECRET_PATTERN, CONTROL_CHARS_PATTERN, REDACT_SCAN_LENGTH, TOKEN_CHAR_PATTERN, REDACT_SCAN_EXTENSION_MAX, TERMINAL_ESCAPE_PATTERN;
 var init_log_utils = __esm({
   "src/utils/log-utils.ts"() {
     "use strict";
     logContextStorage = new AsyncLocalStorage();
     MAX_LOG_MESSAGE_LENGTH = 1e4;
     ANSI_PATTERN = /\x1b\[[0-9;]*[A-Za-z]/g;
-    URL_CREDENTIALS_PATTERN = /([a-z][a-z0-9+.-]*:\/\/)[^/\s:@]+:[^/\s:@]+@/gi;
-    COOKIE_HEADER_PATTERN = /\b(set[_-]?cookie|cookie)\b(["']?\s*[:=]\s*["']?)([^\r\n]+)/gi;
-    SENSITIVE_KV_PATTERN = /\b(api[_-]?key|apikey|access[_-]?token|auth[_-]?token|refresh[_-]?token|secret|client[_-]?secret|password|passwd|pwd|authorization|bearer|set[_-]?cookie|cookie|session[_-]?id|session|csrf[_-]?token|xsrf[_-]?token|private[_-]?key|key)\b(["']?\s*[:=]\s*["']?)([^\s"',&)]+)/gi;
-    KNOWN_TOKEN_PATTERN = /\b(sk-[A-Za-z0-9_-]{16,}|gh[posru]_[A-Za-z0-9]{16,}|github_pat_[A-Za-z0-9_]{20,}|AKIA[0-9A-Z]{16}|xox[baprs]-[A-Za-z0-9-]{10,})\b/g;
-    JWT_PATTERN = /\beyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+/g;
-    BASIC_AUTH_PATTERN = /\bBasic\s+[A-Za-z0-9+/]{16,}={0,2}/gi;
-    BEARER_AUTH_PATTERN = /\bBearer\s+[A-Za-z0-9._~+/-]+=*/gi;
-    PROVIDER_TOKEN_PATTERN = /\b(AIza[0-9A-Za-z_-]{35}|ya29\.[0-9A-Za-z_-]{20,}|[sp]k_(?:live|test)_[0-9A-Za-z]{16,})\b/g;
-    LONG_HEX_SECRET_PATTERN = /\b[0-9a-fA-F]{32,}\b/g;
+    URL_CREDENTIALS_PATTERN = /([a-z][a-z0-9+.-]{0,31}:\/\/)[^/\s:@]+:[^/\s:@]+@/gi;
+    B_LEFT = "(?<![A-Za-z0-9])";
+    B_RIGHT = "(?![A-Za-z0-9])";
+    COOKIE_HEADER_PATTERN = new RegExp(
+      `${B_LEFT}(set[_-]?cookie|cookie)${B_RIGHT}(["']?\\s*[:=]\\s*["']?)([^\\r\\n]+)`,
+      "gi"
+    );
+    SENSITIVE_KV_PATTERN = new RegExp(
+      `${B_LEFT}(api[_-]?key|apikey|access[_-]?token|auth[_-]?token|refresh[_-]?token|secret|client[_-]?secret|password|passwd|pwd|authorization|bearer|set[_-]?cookie|cookie|session[_-]?id|session|csrf[_-]?token|xsrf[_-]?token|private[_-]?key|key)${B_RIGHT}(["']?\\s*[:=]\\s*["']?)([^\\s"',&)]+)`,
+      "gi"
+    );
+    KNOWN_TOKEN_PATTERN = new RegExp(
+      `${B_LEFT}(sk-[A-Za-z0-9_-]{16,}|gh[posru]_[A-Za-z0-9]{16,}|github_pat_[A-Za-z0-9_]{20,}|AKIA[0-9A-Z]{16}|xox[baprs]-[A-Za-z0-9-]{10,})${B_RIGHT}`,
+      "g"
+    );
+    JWT_PATTERN = new RegExp(`${B_LEFT}eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+`, "g");
+    BASIC_AUTH_PATTERN = new RegExp(`${B_LEFT}Basic\\s+[A-Za-z0-9+/]{16,}={0,2}`, "gi");
+    BEARER_AUTH_PATTERN = new RegExp(`${B_LEFT}Bearer\\s+[A-Za-z0-9._~+/-]+=*`, "gi");
+    PROVIDER_TOKEN_PATTERN = new RegExp(
+      `${B_LEFT}(AIza[0-9A-Za-z_-]{35}|ya29\\.[0-9A-Za-z_-]{20,}|[sp]k_(?:live|test)_[0-9A-Za-z]{16,})${B_RIGHT}`,
+      "g"
+    );
+    LONG_HEX_SECRET_PATTERN = new RegExp(`${B_LEFT}[0-9a-fA-F]{32,}${B_RIGHT}`, "g");
     CONTROL_CHARS_PATTERN = /[\x00-\x1f\x7f]/g;
     REDACT_SCAN_LENGTH = MAX_LOG_MESSAGE_LENGTH * 4;
     TOKEN_CHAR_PATTERN = /[A-Za-z0-9_\-./+=]/;
@@ -110,21 +124,21 @@ var init_user_agent = __esm({
   "src/utils/user-agent.ts"() {
     "use strict";
     USER_AGENTS = [
-      // Chrome on Windows (updated 2025)
-      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
-      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
+      // Chrome on Windows
+      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
+      "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
       // Chrome on macOS
-      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
-      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
+      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
+      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
       // Chrome on Linux
-      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/137.0.0.0 Safari/537.36",
-      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/136.0.0.0 Safari/537.36",
-      // Firefox on Windows (updated 2025)
-      "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:139.0) Gecko/20100101 Firefox/139.0",
-      "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:138.0) Gecko/20100101 Firefox/138.0",
+      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/151.0.0.0 Safari/537.36",
+      "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/150.0.0.0 Safari/537.36",
+      // Firefox on Windows
+      "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:153.0) Gecko/20100101 Firefox/153.0",
+      "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:152.0) Gecko/20100101 Firefox/152.0",
       // Firefox on macOS
-      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:139.0) Gecko/20100101 Firefox/139.0",
-      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:138.0) Gecko/20100101 Firefox/138.0"
+      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:153.0) Gecko/20100101 Firefox/153.0",
+      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:152.0) Gecko/20100101 Firefox/152.0"
     ];
   }
 });
@@ -1498,8 +1512,9 @@ __export(scraper_utils_exports, {
   extractDomain: () => extractDomain,
   formatErrorWithCause: () => formatErrorWithCause,
   getRandomUserAgent: () => getRandomUserAgent,
-  getSsrfSafeDispatcher: () => getSsrfSafeDispatcher,
+  getSsrfSafeFetcher: () => getSsrfSafeFetcher,
   isBenignScrapeFailure: () => isBenignScrapeFailure,
+  isSsrfBlockError: () => isSsrfBlockError,
   ssrfSafeLookup: () => ssrfSafeLookup,
   stripImageLinks: () => stripImageLinks,
   validateContent: () => validateContent,
@@ -1619,16 +1634,29 @@ function ssrfSafeLookup(hostname, options, callback, resolveImpl = dnsCb.lookup)
     return callback(null, safe[0].address, safe[0].family);
   });
 }
-async function getSsrfSafeDispatcher() {
-  if (cachedSsrfDispatcher !== void 0) return cachedSsrfDispatcher;
+function isSsrfBlockError(error) {
+  let e = error;
+  for (let depth = 0; depth < 5 && e instanceof Error; depth++) {
+    if (e.code === "ESSRFBLOCKED" || e.message.includes("not allowed")) {
+      return true;
+    }
+    e = e.cause;
+  }
+  return false;
+}
+async function getSsrfSafeFetcher() {
+  if (cachedSsrfFetcher !== void 0) return cachedSsrfFetcher;
   try {
-    const { Agent } = await import("undici");
-    cachedSsrfDispatcher = new Agent({ connect: { lookup: ssrfSafeLookup } });
+    const { Agent, fetch: undiciFetch } = await import("undici");
+    cachedSsrfFetcher = {
+      fetch: undiciFetch,
+      dispatcher: new Agent({ connect: { lookup: ssrfSafeLookup } })
+    };
   } catch (e) {
     logger.warn(`[Scrapers] undici unavailable; connect-time SSRF pinning disabled (request-time validation still active): ${e instanceof Error ? e.message : String(e)}`);
-    cachedSsrfDispatcher = null;
+    cachedSsrfFetcher = null;
   }
-  return cachedSsrfDispatcher;
+  return cachedSsrfFetcher;
 }
 function formatErrorWithCause(err) {
   try {
@@ -1678,6 +1706,7 @@ function isPrivateIpv6(ip) {
   if (normalized.startsWith("2002:")) return true;
   if (/^2001:0{0,4}:/.test(normalized) || normalized.startsWith("2001::")) return true;
   if (normalized.startsWith("64:ff9b:")) return true;
+  if (/^::(?:\d{1,3}(?:\.\d{1,3}){3}|[0-9a-f]{1,4}(?::[0-9a-f]{1,4})?)$/.test(normalized)) return true;
   if (normalized === "::" || normalized === "0:0:0:0:0:0:0:0") return true;
   return false;
 }
@@ -1761,7 +1790,7 @@ function validateContent(html, markdown, url) {
     throw error;
   }
 }
-var cachedSsrfDispatcher;
+var cachedSsrfFetcher;
 var init_scraper_utils = __esm({
   "src/web-research/scraper-utils.ts"() {
     "use strict";
@@ -1770,7 +1799,7 @@ var init_scraper_utils = __esm({
     init_logger();
     init_scraper_types();
     init_browser_error_utils();
-    cachedSsrfDispatcher = void 0;
+    cachedSsrfFetcher = void 0;
   }
 });
 
@@ -2515,6 +2544,15 @@ var ConfigSchema = Type.Object({
    *  the model's real maxTokens at call time.
    *  Not exposed in TUI — controlled via PI_RESEARCH_SYNTHESIS_MAX_TOKENS env var. */
   SYNTHESIS_MAX_TOKENS: Type.Number({ minimum: 1024, maximum: 131072, default: 32768 }),
+  /** Max OUTPUT tokens per researcher LLM call (deep and quick sub-agent sessions).
+   *  Without a cap the session inherits the model's full catalog ceiling as the
+   *  per-request max_tokens — ~230k for large-context models — which providers that
+   *  budget-check max_tokens up front (OpenRouter 402s against remaining credits)
+   *  reject outright on low balances, even though the actual response would cost
+   *  cents. Default 16384: researcher turns are tool calls plus a cited report, far
+   *  below this. Clamped to the model's real ceiling at session build time.
+   *  Not exposed in TUI — controlled via PI_RESEARCH_RESEARCHER_MAX_TOKENS env var. */
+  RESEARCHER_MAX_TOKENS: Type.Number({ minimum: 1024, maximum: 131072, default: 16384 }),
   /** LLM Model override for researcher sub-agents and knowledge synthesis.
    *  Format: provider/model-id (e.g. google/gemini-2.0-flash-001) or just model-id.
    *  When set, this overrides ctx.model for researcher sub-agents (both deep and quick)

package/docs/ARCHITECTURE.md

@@ -115,12 +115,14 @@ agent to invoke:
 In deep research `search` is excluded — the coordinator runs the search burst and hands
 out URLs directly.
 
-`grep` is excluded by default at **every** depth and on every front-end (CLI, SDK, agent
-skill, pi extension): this is web research, and a capable model otherwise spends turns
-searching the local filesystem. Passing an explicit `excludeTools` list — `--exclude-tools`
-on the CLI, the `excludeTools` tool parameter in the extension — REPLACES that default, so
-naming other tools re-enables `grep`. `PI_RESEARCH_DISABLED_TOOLS` is separate and strictly
-additive: it can only ever remove capabilities (see [CONFIGURATION.md](CONFIGURATION.md)).
+`grep` is excluded at **every** depth and on every front-end (CLI, SDK, agent skill,
+pi extension): this is web research, and a capable model otherwise spends turns
+searching the local filesystem. Both exclusion surfaces — the `excludeTools` list
+(`--exclude-tools` on the CLI, the `excludeTools` tool parameter in the extension) and
+`PI_RESEARCH_DISABLED_TOOLS` — are strictly additive on top of that default: they can
+only ever remove capabilities (see [CONFIGURATION.md](CONFIGURATION.md)). Before 1.3.10
+a non-empty `excludeTools` list replaced the default, so naming any other tool silently
+re-enabled `grep`.
 
 Researchers cannot write files, run shell commands, or reach the network outside these
 tools.
@@ -318,7 +320,7 @@ other). This sits above `@lancedb/lancedb` 0.29's declared Arrow peer ceiling
 whenever `@lancedb/lancedb` is upgraded.
 
 Pinned validation library — `typebox` is pinned to the exact version the pi host packages
-depend on (`@earendil-works/pi-ai` / `@earendil-works/pi-coding-agent` 0.84.1 pin `1.3.7`). Every
+depend on (`@earendil-works/pi-ai` / `@earendil-works/pi-coding-agent` 0.84.2 pin `1.3.7`). Every
 tool's parameter schema is built with TypeBox here and handed across the boundary to pi's
 tool system, so the two must agree on `Value.Check`/`Convert` semantics. A floating `^1.1.38`
 range let a fresh consumer install resolve pi-research to a newer TypeBox than pi's, shipping