@lincoln504/pi-research 1.5.3 → 1.6.0

package/README.md

@@ -27,7 +27,7 @@
 <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-97ca00?style=flat-square" /></a><br />
 <a href="https://github.com/Lincoln504/pi-research"><img alt="GitHub repository" src="https://img.shields.io/badge/GitHub-Lincoln504%2Fpi--research-181717?style=flat-square&logo=github&logoColor=white" /></a>
 
-Search and scraping run locally through a stealth browser with no search provider and no monthly cap. You pay only for the LLM tokens you use.
+Search and scraping run locally through a stealth browser with no search provider and no monthly cap. The only cost is LLM tokens.
 
 ### Install
 
@@ -44,38 +44,38 @@ npm install -g @lincoln504/pi-research
 pi-research skill install
 ```
 
-In pi it works out of the box on your session's model and pi's configuration. Standalone use ([agent skill](docs/AGENT-SKILL.md) or [SDK](docs/SDK.md)) needs a model configured. See [Configuration](docs/CONFIGURATION.md). The first install pulls the stealth browser engine, which takes a few minutes.
+In pi it works out of the box on the session's model and pi's configuration. Standalone use ([agent skill](docs/AGENT-SKILL.md) or [SDK](docs/SDK.md)) needs a model configured. See [Configuration](docs/CONFIGURATION.md). The first install pulls the stealth browser engine, which takes a few minutes.
 
 ### How it works
 
 A research run loops through agent teams: a coordinator plans and opens with a search, researcher agents scrape and read in parallel, and a research lead decides whether to go another round — then writes the report from every report collected. The result is one cited Markdown report with findings optionally saved to the knowledge store.
 
-Three depth levels (normal, deep, ultra) set the team size and number of rounds. Ask in natural language and the tool picks the right one:
+Three depth levels (normal, deep, ultra) set the team size and number of rounds. A natural-language request is enough — the tool picks the right one:
 
 ![Prompt-driven multi-round research in the pi TUI](https://raw.githubusercontent.com/Lincoln504/pi-research/main/docs/media/02-prompt-research.gif)
 
 ### Use cases
 
-- Researching for your pi. Plug and play with no API key needed.
-- Researching from Claude Code, Codex, or another coding agent while a cheaper or local model drives the run so it doesn't spend your main agent's budget.
+- Researching inside pi. Plug and play with no API key needed.
+- Researching from Claude Code, Codex, or another coding agent while a cheaper or local model drives the run, so it doesn't spend the main agent's budget.
 - Keeping a persistent and searchable knowledge store of findings, scoped globally or per project.
-- Building agent systems that find and read web content or populating a dataset of web sources.
+- Building agent systems that find and read web content, or populating a dataset of web sources.
 
 ### Why pi-research
 
 - **No quota and no monthly fee.** Most AI search providers cap free searches and then charge, with results served from their index on their servers. pi-research searches and scrapes locally through DuckDuckGo in a stealth browser.
-- **You own your index.** Every finding can be saved to a local [LanceDB](https://lancedb.com) knowledge store that seeds future runs, so repeat questions get faster and cheaper answers.
-- **Cited reports rather than raw dumps.** You get one sourced Markdown report instead of raw web content pasted into the conversation. Context efficient and easy to verify.
-- **Read-only by design.** The research agent cannot run shell commands or write, edit, or delete anything. It can read files, so treat it as you would any read-only agent with access to your machine.
+- **The index stays local.** Every finding can be saved to a local [LanceDB](https://lancedb.com) knowledge store that seeds future runs, so repeat questions get faster and cheaper answers.
+- **Cited reports rather than raw dumps.** Each run returns one sourced Markdown report instead of raw web content pasted into the conversation. Context efficient and easy to verify.
+- **Read-only by design.** The research agent cannot run shell commands or write, edit, or delete anything — the right shape for an agent whose whole job is reading untrusted web pages. Prompt injection picked up mid-run has nothing to act on.
 - **Search a little or a lot.** Depth levels span a quick pass to a large-scale investigation.
 
 ### Requirements / limitations
 
 - Node.js >= 22.19.0
-- An LLM with a 100k+ context window (bring your own key or use a local model)
+- An LLM with a 100k+ context window (an API key or a local model)
 - Internet access on a residential IP, since search, scraping, and YouTube transcripts get bot-blocked from datacenter/VPS/cloud IPs
 - pi. The pi extension uses the host's copy while the standalone CLI and agent skill install it as a dependency.
-- Cloudflare and similar systems block scraping on some sites. A run flags pages it can't reach and compensates with a wide pool of search results to cite.
+- Cloudflare and similar systems block scraping on some sites. A run flags unreachable pages and compensates with a wide pool of search results to cite.
 
 ### Channels
 

package/agent-skill/pi-research/SKILL.md

@@ -2,7 +2,7 @@
 name: pi-research
 description: "Multi-agent web research for any coding agent. Use for anything requiring current information from the internet — news, trends, product/service/company facts, statistics, people, places, events, APIs, library behavior, or 'what is X / how does X work' questions. Also use to search a local knowledge store of previously researched findings before doing live research. Powered by the pi-research engine (search, stealth scraping, security databases, Stack Exchange). Not for reading local files, running project commands, or analyzing this repository's own source."
 license: MIT
-metadata: { "author": "Lincoln Deen", "version": "1.5.3", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
+metadata: { "author": "Lincoln Deen", "version": "1.6.0", "package": "@lincoln504/pi-research", "engine": "pi-research-sdk", "openclaw": { "requires": { "bins": ["node"] } } }
 allowed-tools: Bash(node:*)
 ---
 

package/dist/cli.mjs

@@ -7273,11 +7273,10 @@ function assertValidEmbeddingVectors(vectors, expectedCount, declaredDim) {
   }
 }
 async function addDocumentsToStore(table, docs, embedder, isClosing, workspace, isGlobal) {
-  if (docs.length === 0) return;
+  if (docs.length === 0) return "empty";
   if (isClosing()) {
     logger.warn("[store] Ignoring addDocuments during close");
-    metrics.increment("knowledge_store_add_documents_total", 1, { status: "ignored_closing" });
-    return;
+    return "ignored_closing";
   }
   const startTime = Date.now();
   try {
@@ -7297,14 +7296,13 @@ async function addDocumentsToStore(table, docs, embedder, isClosing, workspace,
     await table.add(data);
     const duration = Date.now() - startTime;
     metrics.observe("knowledge_store_add_documents_duration_ms", duration);
-    metrics.increment("knowledge_store_add_documents_total", 1, { status: "success" });
     metrics.increment("knowledge_store_chunks_added_total", docs.length);
     logger.log(`[store] Added ${docs.length} chunk(s) for ${docs[0]?.url} [workspace=${workspace}, global=${isGlobal}]`);
+    return "written";
   } catch (err) {
     const duration = Date.now() - startTime;
     metrics.observe("knowledge_store_add_documents_duration_ms", duration, { status: "error" });
-    metrics.increment("knowledge_store_add_documents_total", 1, { status: "error" });
-    logger.error("[store] Failed to add documents:", err);
+    logger.debug(`[store] add attempt failed after ${duration}ms: ${err instanceof Error ? err.message : String(err)}`);
     throw err;
   }
 }
@@ -7876,18 +7874,19 @@ var init_store = __esm({
         try {
           if (this.isClosing) {
             logger.warn(`[store] Skipping addDocuments \u2014 store is closing`);
+            metrics.increment("knowledge_store_add_documents_total", 1, { status: "ignored_closing" });
             return;
           }
           let table = await this.getFreshTable();
           const workspace = this.getWorkspace();
-          await this.withEmbedderReconnect(async (embedder) => {
+          const outcome = await this.withEmbedderReconnect(async (embedder) => {
             let retryCount = 0;
             const MAX_RETRIES = 5;
             const BASE_DELAY = 100;
             while (retryCount <= MAX_RETRIES) {
               try {
                 const isGlobal = this.options.knowledgeMode === "global";
-                await addDocumentsToStore(
+                return await addDocumentsToStore(
                   table,
                   docs,
                   embedder,
@@ -7895,7 +7894,6 @@ var init_store = __esm({
                   workspace,
                   isGlobal
                 );
-                return;
               } catch (err) {
                 if (isLanceCommitConflict(err)) {
                   retryCount++;
@@ -7909,7 +7907,17 @@ var init_store = __esm({
                 throw err;
               }
             }
+            throw new Error("[store] addDocuments retry loop exited without result");
           });
+          if (outcome !== "empty") {
+            metrics.increment("knowledge_store_add_documents_total", 1, {
+              status: outcome === "ignored_closing" ? "ignored_closing" : "success"
+            });
+          }
+        } catch (err) {
+          metrics.increment("knowledge_store_add_documents_total", 1, { status: "error" });
+          logger.error("[store] Failed to add documents:", err);
+          throw err;
         } finally {
           this.activeWrites.delete(writePromise);
           writeResolve();
@@ -12436,6 +12444,101 @@ function getClientAgent() {
   return clientAgent;
 }
 
+// src/infrastructure/browser/browser-error-utils.ts
+init_circuit_breaker();
+function isTransientSocketError(error) {
+  if (!error || typeof error !== "object") return false;
+  const err = error;
+  return typeof err.message === "string" && (err.message.includes("ECONNREFUSED") || err.message.includes("ECONNRESET") || err.message.includes("socket hang up") || err.message.includes("EPIPE") || err.message.includes("ETIMEDOUT") || err.message.includes("timed out") || err.message.includes("pool busy") || err.message.includes("unreachable") || // WorkerPoolManager throws this during the window between forceSchedulerRestart
+  // clearing the scheduler reference and the old pool's 1500ms drain completing.
+  // It is a transient state — retry after a short delay succeeds once drain finishes.
+  err.message.includes("Worker pool is shutting down") || // Poolifier throws this when pool.execute() is called while the pool is being
+  // destroyed (e.g., during scheduler restart). Treat as transient — the pool will
+  // be ready for new tasks once destruction completes and a new pool is initialized.
+  err.message.includes("Cannot execute a task on destroying pool") || err.message.includes("destroying pool") || // The leader answered 503 because it is shutting down after losing
+  // leadership. Retrying re-elects and reaches the new leader, so this is
+  // transient in exactly the same sense as a pool drain.
+  err.message.includes("Browser pool leader is draining"));
+}
+function isCloudflareBlockError(error) {
+  if (!error || typeof error !== "object") return false;
+  const err = error;
+  return typeof err.message === "string" && (err.message.includes("Fetch blocked: Cloudflare") || err.message.includes("Cloudflare challenge"));
+}
+function isTaskTimeoutError(error) {
+  if (!error || typeof error !== "object") return false;
+  const err = error;
+  return typeof err.message === "string" && (err.message.includes("timed out after") || err.message.includes("Search task timed out") || err.message.includes("Scrape task timed out") || err.message.includes("Health check timed out") || err.message.includes("[BrowserClient] Request to"));
+}
+function isServerDrainingError(error) {
+  if (!error || typeof error !== "object") return false;
+  const err = error;
+  return typeof err.message === "string" && err.message.includes("Browser pool leader is draining");
+}
+function isPoolShutdownError(error) {
+  if (!error || typeof error !== "object") return false;
+  const err = error;
+  return typeof err.message === "string" && (err.message.includes("Worker pool is shutting down") || err.message.includes("Cannot execute a task on destroying pool") || err.message.includes("destroying pool") || // worker-pool-manager throws this when auto-recovery is still swapping the
+  // pool. Its own text says "please retry", but until it was listed here no
+  // classifier matched it, so the first attempt rethrew instead of retrying —
+  // and auto-recovery fires only after 3 consecutive worker failures, i.e.
+  // exactly when a burst of tasks is in flight and would all die together.
+  err.message.includes("Worker pool is being reset") || // poolifier's wording once destroy() has COMPLETED (vs "destroying pool"
+  // while it is in progress). A queued closure holding a pool captured before
+  // an auto-recovery destroy dispatches into exactly this window; it is the
+  // same transient swap the "being reset" string covers, not a hard fault.
+  err.message.includes("Cannot execute a task on not started pool"));
+}
+var BENIGN_NAVIGATION_FAILURE = /page\.goto: (?:Timeout \d+ *ms exceeded|net::|NS_ERROR_|SEC_ERROR_|MOZILLA_PKIX_ERROR_)/;
+function isBenignScrapeFailure(error) {
+  const msg2 = error instanceof Error ? error.message : String(error);
+  return msg2.startsWith("Fetch blocked:") || // bot-protection interstitial (Cloudflare, DDoS-Guard, …)
+  msg2.startsWith("Fetch returned stub:") || // nav-only / near-empty page
+  msg2.startsWith("Could not extract content from PDF") || // malformed/encrypted/scanned-only PDF
+  msg2.startsWith("PDF too large") || // exceeds the deliberate 100MB policy cap
+  msg2.startsWith("Unsupported download:") || // attachment-served non-PDF (zip/xlsx/…) — nothing to extract
+  BENIGN_NAVIGATION_FAILURE.test(msg2) || // remote TLS/DNS/timeout — the URL could not be loaded
+  msg2.startsWith("HTML response too large") || // fetch-layer 25MB policy cap — same class as the PDF cap
+  msg2.startsWith("Browser HTML too large") || // browser-layer 25MB policy cap
+  /^HTTP \d{3}\b/.test(msg2) || // remote 4xx/5xx status
+  isCloudflareBlockError(error) || isTaskTimeoutError(error) || isPoolShutdownError(error);
+}
+var sessionCircuitBreakers = /* @__PURE__ */ new Map();
+var MAX_SESSION_BREAKERS = 256;
+var DEFAULT_BREAKER_CONFIG = {
+  // Raised from 3→8: parallel search bursts of 20 queries frequently produce
+  // simultaneous timeouts on slow/CF-protected sites. A threshold of 3 caused
+  // the breaker to open mid-burst, blocking all subsequent requests for the round.
+  failureThreshold: 8,
+  // Raised from 15s→45s: enough time for one full search burst + retry window
+  // to complete before the breaker attempts to half-open.
+  resetTimeoutMs: 45e3,
+  name: "BrowserPool",
+  isTransientError: (error) => {
+    if (isPoolShutdownError(error)) return false;
+    if (isServerDrainingError(error)) return false;
+    if (isCloudflareBlockError(error)) return false;
+    if (isTaskTimeoutError(error)) return false;
+    return isTransientSocketError(error);
+  }
+};
+function getBrowserCircuitBreaker(sessionId) {
+  let breaker2 = sessionCircuitBreakers.get(sessionId);
+  if (!breaker2) {
+    if (sessionCircuitBreakers.size >= MAX_SESSION_BREAKERS) {
+      const oldest = sessionCircuitBreakers.keys().next().value;
+      if (oldest !== void 0) sessionCircuitBreakers.delete(oldest);
+    }
+    breaker2 = new CircuitBreaker({ ...DEFAULT_BREAKER_CONFIG, name: `BrowserPool-${sessionId}` });
+    sessionCircuitBreakers.set(sessionId, breaker2);
+  }
+  return breaker2;
+}
+function clearSessionCircuitBreaker(sessionId) {
+  sessionCircuitBreakers.delete(sessionId);
+}
+var browserCircuitBreaker = new CircuitBreaker(DEFAULT_BREAKER_CONFIG);
+
 // src/infrastructure/browser/browser-client.ts
 var CLIENT_PATIENCE_MARGIN_MS = 15e3;
 var CLIENT_TIMEOUT_CAP_MS = 3e5;
@@ -12597,12 +12700,16 @@ var BrowserClient = class {
           errorType = "unknown";
         }
         const error = new Error(errorMsg);
-        logger.error(`[BrowserClient] Request to http://127.0.0.1:${this.port}${path20} failed:`, errorMsg);
-        errorTracker.trackError(error, {
-          component: "browser-manager",
-          operation,
-          errorType
-        });
+        if (isTransientSocketError(error)) {
+          logger.debug(`[BrowserClient] Transient failure talking to http://127.0.0.1:${this.port}${path20} (caller will retry): ${errorMsg}`);
+        } else {
+          logger.error(`[BrowserClient] Request to http://127.0.0.1:${this.port}${path20} failed:`, errorMsg);
+          errorTracker.trackError(error, {
+            component: "browser-manager",
+            operation,
+            errorType
+          });
+        }
         reject(error);
       });
       req.write(JSON.stringify(data));
@@ -12644,89 +12751,6 @@ init_logger();
 init_http_body();
 import * as http5 from "node:http";
 import * as crypto5 from "node:crypto";
-
-// src/infrastructure/browser/browser-error-utils.ts
-init_circuit_breaker();
-function isTransientSocketError(error) {
-  if (!error || typeof error !== "object") return false;
-  const err = error;
-  return typeof err.message === "string" && (err.message.includes("ECONNREFUSED") || err.message.includes("ECONNRESET") || err.message.includes("socket hang up") || err.message.includes("EPIPE") || err.message.includes("ETIMEDOUT") || err.message.includes("timed out") || err.message.includes("pool busy") || err.message.includes("unreachable") || // WorkerPoolManager throws this during the window between forceSchedulerRestart
-  // clearing the scheduler reference and the old pool's 1500ms drain completing.
-  // It is a transient state — retry after a short delay succeeds once drain finishes.
-  err.message.includes("Worker pool is shutting down") || // Poolifier throws this when pool.execute() is called while the pool is being
-  // destroyed (e.g., during scheduler restart). Treat as transient — the pool will
-  // be ready for new tasks once destruction completes and a new pool is initialized.
-  err.message.includes("Cannot execute a task on destroying pool") || err.message.includes("destroying pool") || // The leader answered 503 because it is shutting down after losing
-  // leadership. Retrying re-elects and reaches the new leader, so this is
-  // transient in exactly the same sense as a pool drain.
-  err.message.includes("Browser pool leader is draining"));
-}
-function isCloudflareBlockError(error) {
-  if (!error || typeof error !== "object") return false;
-  const err = error;
-  return typeof err.message === "string" && (err.message.includes("Fetch blocked: Cloudflare") || err.message.includes("Cloudflare challenge"));
-}
-function isTaskTimeoutError(error) {
-  if (!error || typeof error !== "object") return false;
-  const err = error;
-  return typeof err.message === "string" && (err.message.includes("timed out after") || err.message.includes("Search task timed out") || err.message.includes("Scrape task timed out") || err.message.includes("Health check timed out") || err.message.includes("[BrowserClient] Request to"));
-}
-function isServerDrainingError(error) {
-  if (!error || typeof error !== "object") return false;
-  const err = error;
-  return typeof err.message === "string" && err.message.includes("Browser pool leader is draining");
-}
-function isPoolShutdownError(error) {
-  if (!error || typeof error !== "object") return false;
-  const err = error;
-  return typeof err.message === "string" && (err.message.includes("Worker pool is shutting down") || err.message.includes("Cannot execute a task on destroying pool") || err.message.includes("destroying pool") || // worker-pool-manager throws this when auto-recovery is still swapping the
-  // pool. Its own text says "please retry", but until it was listed here no
-  // classifier matched it, so the first attempt rethrew instead of retrying —
-  // and auto-recovery fires only after 3 consecutive worker failures, i.e.
-  // exactly when a burst of tasks is in flight and would all die together.
-  err.message.includes("Worker pool is being reset") || // poolifier's wording once destroy() has COMPLETED (vs "destroying pool"
-  // while it is in progress). A queued closure holding a pool captured before
-  // an auto-recovery destroy dispatches into exactly this window; it is the
-  // same transient swap the "being reset" string covers, not a hard fault.
-  err.message.includes("Cannot execute a task on not started pool"));
-}
-var sessionCircuitBreakers = /* @__PURE__ */ new Map();
-var MAX_SESSION_BREAKERS = 256;
-var DEFAULT_BREAKER_CONFIG = {
-  // Raised from 3→8: parallel search bursts of 20 queries frequently produce
-  // simultaneous timeouts on slow/CF-protected sites. A threshold of 3 caused
-  // the breaker to open mid-burst, blocking all subsequent requests for the round.
-  failureThreshold: 8,
-  // Raised from 15s→45s: enough time for one full search burst + retry window
-  // to complete before the breaker attempts to half-open.
-  resetTimeoutMs: 45e3,
-  name: "BrowserPool",
-  isTransientError: (error) => {
-    if (isPoolShutdownError(error)) return false;
-    if (isServerDrainingError(error)) return false;
-    if (isCloudflareBlockError(error)) return false;
-    if (isTaskTimeoutError(error)) return false;
-    return isTransientSocketError(error);
-  }
-};
-function getBrowserCircuitBreaker(sessionId) {
-  let breaker2 = sessionCircuitBreakers.get(sessionId);
-  if (!breaker2) {
-    if (sessionCircuitBreakers.size >= MAX_SESSION_BREAKERS) {
-      const oldest = sessionCircuitBreakers.keys().next().value;
-      if (oldest !== void 0) sessionCircuitBreakers.delete(oldest);
-    }
-    breaker2 = new CircuitBreaker({ ...DEFAULT_BREAKER_CONFIG, name: `BrowserPool-${sessionId}` });
-    sessionCircuitBreakers.set(sessionId, breaker2);
-  }
-  return breaker2;
-}
-function clearSessionCircuitBreaker(sessionId) {
-  sessionCircuitBreakers.delete(sessionId);
-}
-var browserCircuitBreaker = new CircuitBreaker(DEFAULT_BREAKER_CONFIG);
-
-// src/infrastructure/browser/browser-server.ts
 var _authSecret = null;
 function getOrCreateAuthSecret() {
   if (!_authSecret) {
@@ -12868,6 +12892,8 @@ var BrowserServer = class {
               logger.warn("[BrowserServer] Request blocked by bot protection:", error instanceof Error ? error.message : String(error));
             } else if (isPoolShutdownError(error)) {
               logger.debug("[BrowserServer] Request abandoned during shutdown:", error instanceof Error ? error.message : String(error));
+            } else if (isBenignScrapeFailure(error)) {
+              logger.debug("[BrowserServer] Expected per-URL scrape outcome:", error instanceof Error ? error.message.split("\n")[0] : String(error));
             } else {
               logger.error("[BrowserServer] Error handling request:", error);
             }
@@ -18060,17 +18086,6 @@ import * as dnsCb from "node:dns";
 import * as net3 from "node:net";
 import crypto9 from "node:crypto";
 import { NodeHtmlMarkdown } from "node-html-markdown";
-function isBenignScrapeFailure(error) {
-  const msg2 = error instanceof Error ? error.message : String(error);
-  return msg2.startsWith("Fetch blocked:") || // bot-protection interstitial (Cloudflare, DDoS-Guard, …)
-  msg2.startsWith("Fetch returned stub:") || // nav-only / near-empty page
-  msg2.startsWith("Could not extract content from PDF") || // malformed/encrypted/scanned-only PDF
-  msg2.startsWith("PDF too large") || // exceeds the deliberate 100MB policy cap
-  msg2.startsWith("HTML response too large") || // fetch-layer 25MB policy cap — same class as the PDF cap
-  msg2.startsWith("Browser HTML too large") || // browser-layer 25MB policy cap
-  /^HTTP \d{3}\b/.test(msg2) || // remote 4xx/5xx status
-  isCloudflareBlockError(error) || isTaskTimeoutError(error) || isPoolShutdownError(error);
-}
 function getRandomUserAgent() {
   try {
     const index = crypto9.randomInt(0, USER_AGENTS.length);
@@ -21971,9 +21986,14 @@ async function doMint(identifiers, opts) {
   const requestKey = opts.requestKey || process.env["PI_RESEARCH_YOUTUBE_POTOKEN_REQUEST_KEY"] || DEFAULT_REQUEST_KEY;
   const fetchImpl = opts.fetchImpl ?? fetch;
   const started = Date.now();
-  const bgMod = await import("bgutils-js");
-  const BG = bgMod.BG ?? bgMod.default;
-  const { buildURL: buildURL2, getHeaders } = bgMod;
+  const [bgGuard, bgWebPo, bgUtils] = await Promise.all([
+    import("bgutils-js/botguard"),
+    import("bgutils-js/webpo"),
+    import("bgutils-js/utils")
+  ]);
+  const { getChallenge, BotGuardClient } = bgGuard;
+  const { WebPoMinter } = bgWebPo;
+  const { buildURL: buildURL2, getHeaders } = bgUtils;
   const { JSDOM } = await import("jsdom");
   const dom = new JSDOM('<!DOCTYPE html><html lang="en"><body></body></html>', {
     url: "https://www.youtube.com/"
@@ -22004,24 +22024,24 @@ async function doMint(identifiers, opts) {
   };
   bridge();
   try {
-    const bgConfig = {
-      fetch: (input, init) => fetchImpl(input, {
+    const challenge = await getChallenge({
+      requestKey,
+      fetchFunction: ((input, init) => fetchImpl(input, {
         ...init,
         signal: init?.signal ?? opts.signal
-      }),
-      globalObj: globalThis,
-      identifier: identifiers[0],
-      requestKey
-    };
-    const challenge = await BG.Challenge.create(bgConfig);
+      }))
+    });
     if (!challenge) throw new Error("BotGuard challenge unavailable");
     const interpreter = challenge.interpreterJavascript?.privateDoNotAccessOrElseSafeScriptWrappedValue;
     if (!interpreter) throw new Error("BotGuard interpreter script missing");
     new Function(interpreter)();
-    const botguard = await BG.BotGuardClient.create({
+    if (!challenge.program || !challenge.globalName) {
+      throw new Error("BotGuard challenge missing program/globalName");
+    }
+    const botguard = await BotGuardClient.create({
       program: challenge.program,
       globalName: challenge.globalName,
-      globalObj: globalThis
+      globalObject: globalThis
     });
     const webPoSignalOutput = [];
     const botguardResponse = await botguard.snapshot({ webPoSignalOutput });
@@ -22039,7 +22059,7 @@ async function doMint(identifiers, opts) {
     if (!integrityToken) {
       throw new Error("Integrity token rejected by attestation server (likely a non-residential IP)");
     }
-    const minter = await BG.WebPoMinter.create({ integrityToken }, webPoSignalOutput);
+    const minter = await WebPoMinter.create({ integrityToken }, webPoSignalOutput);
     const tokens = /* @__PURE__ */ new Map();
     for (const id of identifiers) {
       tokens.set(id, await minter.mintAsWebsafeString(id));

package/dist/thread-worker.mjs

@@ -1516,11 +1516,25 @@ function isPoolShutdownError(error) {
   // same transient swap the "being reset" string covers, not a hard fault.
   err.message.includes("Cannot execute a task on not started pool"));
 }
-var DEFAULT_BREAKER_CONFIG, browserCircuitBreaker;
+function isBenignScrapeFailure(error) {
+  const msg = error instanceof Error ? error.message : String(error);
+  return msg.startsWith("Fetch blocked:") || // bot-protection interstitial (Cloudflare, DDoS-Guard, …)
+  msg.startsWith("Fetch returned stub:") || // nav-only / near-empty page
+  msg.startsWith("Could not extract content from PDF") || // malformed/encrypted/scanned-only PDF
+  msg.startsWith("PDF too large") || // exceeds the deliberate 100MB policy cap
+  msg.startsWith("Unsupported download:") || // attachment-served non-PDF (zip/xlsx/…) — nothing to extract
+  BENIGN_NAVIGATION_FAILURE.test(msg) || // remote TLS/DNS/timeout — the URL could not be loaded
+  msg.startsWith("HTML response too large") || // fetch-layer 25MB policy cap — same class as the PDF cap
+  msg.startsWith("Browser HTML too large") || // browser-layer 25MB policy cap
+  /^HTTP \d{3}\b/.test(msg) || // remote 4xx/5xx status
+  isCloudflareBlockError(error) || isTaskTimeoutError(error) || isPoolShutdownError(error);
+}
+var BENIGN_NAVIGATION_FAILURE, DEFAULT_BREAKER_CONFIG, browserCircuitBreaker;
 var init_browser_error_utils = __esm({
   "src/infrastructure/browser/browser-error-utils.ts"() {
     "use strict";
     init_circuit_breaker();
+    BENIGN_NAVIGATION_FAILURE = /page\.goto: (?:Timeout \d+ *ms exceeded|net::|NS_ERROR_|SEC_ERROR_|MOZILLA_PKIX_ERROR_)/;
     DEFAULT_BREAKER_CONFIG = {
       // Raised from 3→8: parallel search bursts of 20 queries frequently produce
       // simultaneous timeouts on slow/CF-protected sites. A threshold of 3 caused
@@ -1565,17 +1579,6 @@ import * as dnsCb from "node:dns";
 import * as net from "node:net";
 import crypto from "node:crypto";
 import { NodeHtmlMarkdown } from "node-html-markdown";
-function isBenignScrapeFailure(error) {
-  const msg = error instanceof Error ? error.message : String(error);
-  return msg.startsWith("Fetch blocked:") || // bot-protection interstitial (Cloudflare, DDoS-Guard, …)
-  msg.startsWith("Fetch returned stub:") || // nav-only / near-empty page
-  msg.startsWith("Could not extract content from PDF") || // malformed/encrypted/scanned-only PDF
-  msg.startsWith("PDF too large") || // exceeds the deliberate 100MB policy cap
-  msg.startsWith("HTML response too large") || // fetch-layer 25MB policy cap — same class as the PDF cap
-  msg.startsWith("Browser HTML too large") || // browser-layer 25MB policy cap
-  /^HTTP \d{3}\b/.test(msg) || // remote 4xx/5xx status
-  isCloudflareBlockError(error) || isTaskTimeoutError(error) || isPoolShutdownError(error);
-}
 function getRandomUserAgent() {
   try {
     const index = crypto.randomInt(0, USER_AGENTS.length);
@@ -2014,7 +2017,7 @@ import { platform as platform2 } from "node:os";
 
 // src/infrastructure/browser/thread-worker-messaging.ts
 init_log_utils();
-import { appendFileSync as appendFileSync2 } from "node:fs";
+import { appendFileSync as appendFileSync2, promises as fsp } from "node:fs";
 
 // src/utils/safe-unref.ts
 function safeUnref(timer) {
@@ -2286,6 +2289,66 @@ function seedSubresourceVerdict(validatedUrl) {
   } catch {
   }
 }
+function isEvictedBodyError(err) {
+  const msg = err instanceof Error ? err.message : String(err);
+  return /was evicted|Response body is unavailable/i.test(msg);
+}
+var PDF_REFETCH_TIMEOUT_MS = 9e4;
+async function refetchPdfBody(page, url, validateForWorker, signal) {
+  await validateForWorker(url);
+  if (signal?.aborted) throw new Error("Aborted");
+  const requestPromise = page.context().request.get(url, {
+    maxRedirects: 0,
+    timeout: PDF_REFETCH_TIMEOUT_MS
+  });
+  requestPromise.catch(() => {
+  });
+  const response = await (signal ? Promise.race([
+    requestPromise,
+    new Promise((_, reject) => {
+      if (signal.aborted) return reject(new Error("Aborted"));
+      signal.addEventListener("abort", () => reject(new Error("Aborted")), { once: true });
+    })
+  ]) : requestPromise);
+  const status = Number(response.status?.() ?? 0);
+  if (!response.ok?.()) throw new Error(`HTTP ${status}`);
+  const declared = parseInt(String((await response.headersArray?.() ?? []).find?.((h) => String(h.name).toLowerCase() === "content-length")?.value ?? ""), 10);
+  if (Number.isFinite(declared) && declared > MAX_PDF_SIZE) {
+    throw new Error(`PDF too large (${Math.round(declared / 1024 / 1024)}MB, max 100MB)`);
+  }
+  const buffer = await response.body();
+  if (buffer.byteLength > MAX_PDF_SIZE) {
+    throw new Error(`PDF too large (${Math.round(buffer.byteLength / 1024 / 1024)}MB, max 100MB)`);
+  }
+  logToDebugFile2("DEBUG", `[Worker-${workerId2}] Re-fetched evicted PDF body (${buffer.byteLength} bytes) for: ${url}`);
+  return buffer;
+}
+async function readPdfDownload(download, url, page, pendingAddrChecks, getPoisonedError) {
+  const failure = await download.failure?.().catch(() => null);
+  if (failure) throw new Error(`Fetch blocked: download failed (${failure})`);
+  const suggested = String(download.suggestedFilename?.() ?? "");
+  const downloadUrl = String(download.url?.() ?? url);
+  const looksLikePdf = suggested.toLowerCase().endsWith(".pdf") || downloadUrl.toLowerCase().split("?")[0].endsWith(".pdf");
+  if (!looksLikePdf) {
+    throw new Error(`Unsupported download: ${suggested || downloadUrl} is not a PDF`);
+  }
+  const filePath = await download.path();
+  if (!filePath) throw new Error("Fetch blocked: download produced no readable file");
+  const { size } = await fsp.stat(filePath);
+  if (size > MAX_PDF_SIZE) {
+    const sizeMB = Math.round(size / 1024 / 1024);
+    throw new Error(`PDF too large (${sizeMB}MB, max 100MB)`);
+  }
+  const buffer = await fsp.readFile(filePath);
+  await Promise.allSettled(pendingAddrChecks);
+  const poisoned = getPoisonedError();
+  if (poisoned) throw poisoned;
+  await download.delete?.().catch(() => {
+  });
+  await page.close();
+  logToDebugFile2("DEBUG", `[Worker-${workerId2}] Recovered attachment-served PDF (${buffer.byteLength} bytes) for: ${url}`);
+  return { contentType: "application/pdf", bufferB64: buffer.toString("base64"), jitter: 0 };
+}
 async function executeScrapeTask(_context, url, signal) {
   const page = await createPageSafe(_context);
   const SCRAPE_TIMEOUT = parseTimeoutMs(process.env["PI_RESEARCH_SCRAPE_TIMEOUT_MS"], 15e3);
@@ -2302,6 +2365,7 @@ async function executeScrapeTask(_context, url, signal) {
   let poisonedError = null;
   const pendingAddrChecks = [];
   let oversizedPdfError = null;
+  let pendingDownload = null;
   try {
     logToDebugFile2("DEBUG", `[Worker-${workerId2}] Starting scrape for: ${url}`);
     const { validateUrlForSSRF: validateForWorker, validateUrlForSSRFSync: validateForWorkerSync } = await Promise.resolve().then(() => (init_scraper_utils(), scraper_utils_exports));
@@ -2368,7 +2432,25 @@ async function executeScrapeTask(_context, url, signal) {
       } catch {
       }
     });
-    const response = await page.goto(url, { waitUntil: "domcontentloaded" });
+    page.on("download", (d) => {
+      pendingDownload ??= d;
+    });
+    let response;
+    try {
+      response = await page.goto(url, { waitUntil: "domcontentloaded" });
+    } catch (navErr) {
+      const navMsg = navErr instanceof Error ? navErr.message : String(navErr);
+      const isDownloadNav = /download is starting/i.test(navMsg);
+      if (!isDownloadNav) throw navErr;
+      if (!pendingDownload) {
+        await Promise.race([
+          page.waitForEvent("download", { timeout: 5e3 }).catch(() => null),
+          new Promise((r) => setTimeout(r, 5e3))
+        ]);
+      }
+      if (!pendingDownload) throw navErr;
+      return await readPdfDownload(pendingDownload, url, page, pendingAddrChecks, () => poisonedError);
+    }
     const serverAddr = await response?.serverAddr().catch(() => null);
     if (serverAddr?.ipAddress) {
       const ip = serverAddr.ipAddress;
@@ -2386,7 +2468,13 @@ async function executeScrapeTask(_context, url, signal) {
         throw new Error(`PDF too large (${sizeMB}MB, max 100MB)`);
       }
       if (oversizedPdfError) throw oversizedPdfError;
-      const buffer = await response.body();
+      let buffer;
+      try {
+        buffer = await response.body();
+      } catch (bodyErr) {
+        if (!isEvictedBodyError(bodyErr)) throw bodyErr;
+        buffer = await refetchPdfBody(page, finalUrl, validateForWorker, signal);
+      }
       if (buffer.byteLength > MAX_PDF_SIZE) {
         const sizeMB = Math.round(buffer.byteLength / 1024 / 1024);
         throw new Error(`PDF too large (${sizeMB}MB, max 100MB)`);
@@ -2749,7 +2837,7 @@ function logToDebugFile3(level, ...args) {
         return String(arg);
       }).join(" "))
     };
-    import("node:fs/promises").then((fsp) => fsp.appendFile(logFile, `${JSON.stringify(entry)}
+    import("node:fs/promises").then((fsp2) => fsp2.appendFile(logFile, `${JSON.stringify(entry)}
 `)).catch(() => {
     });
   } catch {
@@ -2888,7 +2976,7 @@ async function initBrowser() {
   return initPromise;
 }
 async function acquireTaskContext() {
-  const contextPromise = browser.newContext();
+  const contextPromise = browser.newContext({ acceptDownloads: true });
   contextPromise.catch((err) => logToDebugFile3("DEBUG", `[ThreadWorker] Background browser context rejection: ${err.message}`));
   let contextTimeoutId;
   let contextRaceLost = false;