crawlforge-mcp-server 6.7.0 → 6.9.0

package/CLAUDE.md

@@ -141,7 +141,7 @@ npm run docker:prod         # Run production container
 - **ActionExecutor**: Browser automation engine (Playwright-based)
 - **ResearchOrchestrator**: Multi-stage research with query expansion and synthesis
 - **AgentOrchestrator**: Powers the `agent` tool — NL prompt → autonomous PLAN→GATHER→ACT→DECIDE→SHAPE loop with three orchestrator-enforced hard stops (maxSteps≤10, maxUrls≤20, wall-clock) never delegated to the LLM; degraded no-LLM-key path (D2, v4.6.0)
-- **StealthBrowserManager**: Stealth mode scraping with anti-detection; Camoufox (Firefox) engine added in v4.0.0
+- **StealthBrowserManager**: Stealth mode scraping with anti-detection; Camoufox (Firefox) engine added in v4.0.0. Engine selection goes through the one resolver `resolveStealthEngine(requested)` → `{engine, fallbackWarning}`; `'auto'` (the default for `scrape.escalate_engine` and `stealth_mode.engine` since the 2026-09 stealth review Phase 2) prefers Camoufox and falls back to Chromium with a caller-visible warning when its binary is absent. `'playwright'` is the tool layer's public name for Chromium. Camoufox personas are pinned to the installed binary's Firefox major so the UA's `rv:` and `Firefox/` tokens agree — see `docs/STEALTH_REVIEW_2026-09.md`
 - **LocalizationManager**: Multi-language content and localization
 - **ChangeTracker**: Content change tracking over time
 - **SnapshotManager**: Website snapshots and version history
@@ -221,8 +221,25 @@ MAX_PAGES_PER_CRAWL=100
 RESPECT_ROBOTS_TXT=true
 ROBOTS_CACHE_TTL_MS=3600000          # how long a parsed robots.txt stays good for
 CRAWLFORGE_BLOCKED_HOSTS=            # comma-separated; extends the permanent opt-out blocklist
+CRAWLFORGE_STEALTH_ENGINE=           # which engine `auto` resolves to on THIS deployment:
+                                     # chromium|playwright pins it to Chromium; unset or anything
+                                     # else keeps the default preference for Camoufox. A caller
+                                     # naming an engine always wins. There is no correct global
+                                     # value — which engine clears a wall depends on the exit IP
+                                     # (residential favours Camoufox, the hosted datacenter IP
+                                     # favours Chromium, measured twice).
+CRAWLFORGE_STEALTH_PROXIES=          # comma-separated proxy URLs used by the escalation stage,
+                                     # stealth_mode, browser_session, scrape_with_actions and the
+                                     # deep_research fallback when the caller passes none (a proxy
+                                     # passed on the call wins). CrawlForge supplies no proxies, and
+                                     # a datacenter one does not defeat these detectors. Distinct
+                                     # from the PROXY_ROTATION_* family, which belongs to
+                                     # localization. The agent's automatic stealth retry
+                                     # (stealth review Phase 3) reads it too.
 ```
 
+Camoufox is an **optional** dependency and its transitive `language-tags@2.1.0` requires Node ≥22. npm silently drops an optional subtree that fails an engine check, so on Node 20 it is simply absent — `'auto'` then resolves to Chromium and says so in `fallbackWarning`. `engines.node` stays at `>=20.16.0` deliberately; bumping it is breaking and is a release decision.
+
 Web Bot Auth (optional, off by default — requests go out unsigned when unset):
 
 ```bash
@@ -260,7 +277,7 @@ Key mechanisms for security-conscious future sessions:
 - **endpointGuard** (`src/core/endpointGuard.js`): Hard allow-list of `{crawlforge.dev, www.crawlforge.dev, api.crawlforge.dev}` for the server's own backend calls; HTTPS required; fail-closed. Localhost only in creator mode (v3.0.18).
 - **Action allowlist** (`src/core/ActionExecutor.js`): `scrape_with_actions` accepts only 7 action types: `wait`, `click`, `type`, `press`, `scroll`, `screenshot`, `executeJavaScript`. `executeJavaScript` throws unless `ALLOW_JAVASCRIPT_EXECUTION=true` is set at deploy time (off by default).
 - **Elicitation** (`src/core/ElicitationHelper.js`): User confirmation requested for `deep_research` (>50 URLs), `batch_scrape` (sync, >25 URLs), `crawl_deep` (projected >500 pages), `extract_structured` (schema has >3 required fields, no LLM configured), and credit-low situations. Fail-open if client does not support elicitation.
-- **Browser sandboxing**: Standard pool retains OS sandbox. Stealth Chromium uses `--no-sandbox` + `--disable-web-security` (deliberate fingerprint-spoofing trade-off). Camoufox (Firefox, v4.0.0) is the alternative — see `docs/stealth-engines.md`.
+- **Browser sandboxing**: Standard pool retains OS sandbox. Stealth Chromium uses `--no-sandbox` (deliberate fingerprint-spoofing trade-off); `--disable-web-security` was removed from it in Phase 1 of the 2026-09 stealth review — a page reads it in one line — but the non-stealth render browser (`BrowserProcessor`) still carries both flags. Camoufox (Firefox, v4.0.0) is the alternative — see `docs/stealth-engines.md`.
 
 See `docs/sandboxing-and-approvals.md` for the full reference.
 

package/README.md

@@ -206,7 +206,7 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `scrape_with_actions` | 5 | Browser automation chains |
 | `generate_llms_txt` | 5 | Generate AI interaction guidelines |
 | `stealth_mode` | 5 | Anti-detection browser management |
-| `agent` | 8 | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) |
+| `agent` | 8 (+5 per stealth retry that gets the page, max 2) | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM). A page that walls the plain fetch is retried in the stealth browser automatically — URLs you name first — and that evidence is marked `via: "stealth"`; each retry that runs adds 5 |
 | `deep_research` | 10 | Multi-stage research with source verification |
 
 Ten tools (`scrape`, `fetch_url`, `extract_content`, `crawl_deep`, `batch_scrape`, `stealth_mode`, `scrape_with_actions`, `process_document`, `deep_research`, `extract_embedded_state`) accept `max_inline_chars` (default 40,000; env `CRAWLFORGE_MAX_INLINE_CHARS`): a result over it comes back as a `preview` plus a `result_handle` for `read_result`, with the full result kept for 1 hour under `~/.crawlforge/results/` on your own machine — nothing is uploaded.
@@ -267,6 +267,12 @@ export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape"   # unset = all tools; unkno
 export RESEARCH_STEALTH_ENGINE="auto"      # auto (default) | camoufox | chromium
 export RESEARCH_STEALTH_FALLBACK="true"    # set to "false" to disable entirely
 export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research run
+
+# Optional: your own proxies for the stealth paths that have no caller to ask —
+# the scrape escalation stage, the deep_research retry, the agent, browser_session.
+# Comma-separated; a proxy passed on a call always wins. CrawlForge supplies none.
+export CRAWLFORGE_STEALTH_PROXIES="http://user:pass@gw.provider.net:8080"
+# Unrelated to the PROXY_ROTATION_* variables, which belong to the localization tool.
 ```
 
 ### MCP Spec Features
@@ -297,6 +303,18 @@ ollama pull llama3.2
 #    extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })
 ```
 
+### Stealth engines and proxies
+
+Every stealth entry point — `scrape` with `escalate: true`, `stealth_mode`, `browser_session`, `scrape_with_actions` and the `deep_research` retry — now defaults to **`auto`**: use Camoufox (Firefox anti-detect) when its binary is installed, fall back to Chromium when it is not, with the reason in the result's `warnings[]`. Every result names the engine that actually ran. Naming an engine explicitly behaves as it always has — `camoufox` fails rather than falling back, and `chromium` and `playwright` are the same engine (on `scrape`'s `escalate_engine`, spell it `playwright`).
+
+On `browser_session` and `scrape_with_actions` the engine applies only with `stealth: true` — their ordinary path is the standard Chromium pool.
+
+`auto` is not free. Measured once each on an Apple Silicon Mac on 2026-09-21 (launch plus a context and a blank page): Chromium 148 ms and 253 MB, Camoufox 946 ms and 667 MB — roughly **+0.8 s and +400 MB per stealth call**. Pin `engine: "playwright"` for high-volume work on sites that do not block.
+
+**CrawlForge supplies no proxies**, and a datacenter proxy does not fix a block: Cloudflare scores the IP's ASN and the TLS/HTTP2 handshake before any JavaScript runs, so no browser-side patch compensates for a datacenter address. Bring your own residential exit with `CRAWLFORGE_STEALTH_PROXIES` (comma-separated URLs), which the escalation stage, `stealth_mode`, `browser_session`, `scrape_with_actions`, the `deep_research` retry and the `agent` tool's automatic stealth retry use when the caller passes none; a proxy passed on the call always wins. With a proxy, Camoufox derives its timezone, locale and geolocation from the exit IP.
+
+Full detail, measurements and sources: [docs/stealth-engines.md](docs/stealth-engines.md).
+
 ### Stealth extraction for `deep_research` (Camoufox)
 
 `deep_research` automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a **real fingerprinted browser**, then re-extracts from the rendered HTML. It's bounded (`RESEARCH_MAX_STEALTH_RETRIES`, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.
@@ -307,7 +325,7 @@ Engine selection (`RESEARCH_STEALTH_ENGINE`):
 - **`camoufox`** — force Camoufox.
 - **`chromium`** — force the Chromium stealth engine.
 
-Headless Chromium **cannot** clear modern challenges (Cloudflare Turnstile, DataDome) — **Camoufox can**. In testing it recovered Quora and Trustpilot pages that were otherwise fully blocked. To enable it, install the optional dependency and run its one-time binary fetch:
+Camoufox gets through walls that stop headless Chromium: in a benchmark run on 2026-09-21 from a residential IP it was the only engine to clear Cloudflare Turnstile (indeed.com) and Akamai (harrods.com), both of which blocked Chromium stealth. Neither engine cleared DataDome or an interactive Turnstile, so this is a better fetch path, not a bypass. To enable it, install the optional dependency and run its one-time binary fetch:
 
 ```bash
 # Camoufox is declared as an optional dependency, so a normal install already pulls it.
@@ -320,7 +338,7 @@ npx camoufox fetch
 
 Without the Camoufox binary, `deep_research` silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with `RESEARCH_STEALTH_FALLBACK=false`.
 
-> **Note:** Hard IP-reputation blocks (e.g. Reddit's edge `403`) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide. See [docs/stealth-engines.md](docs/stealth-engines.md) for details.
+> **Note:** Hard IP-reputation blocks (e.g. Reddit's edge `403`) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide — point the server at your own with `CRAWLFORGE_STEALTH_PROXIES`. See [docs/stealth-engines.md](docs/stealth-engines.md) for details.
 
 ### Manual Configuration
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "6.7.0",
+  "version": "6.9.0",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
@@ -28,6 +28,8 @@
     "docker:build": "docker build -t crawlforge .",
     "docker:dev": "docker-compose up crawlforge-dev",
     "docker:prod": "docker-compose up crawlforge-prod",
+    "bench:stealth": "node scripts/stealth-bench.mjs",
+    "bench:stealth:ci": "node scripts/stealth-bench.mjs --ci",
     "eval:selection": "node scripts/tool-selection-eval.mjs",
     "usage:report": "node scripts/tool-usage-report.mjs"
   },
@@ -155,6 +157,6 @@
     "@hono/node-server": "2.0.12",
     "hono": "^4.12.4",
     "dompurify": "^3.4.0",
-    "adm-zip": "^0.6.0"
+    "adm-zip": "^0.6.1"
   }
 }

package/server.js

@@ -29,7 +29,7 @@ import { GenerateLLMsTxtTool } from "./src/tools/llmstxt/generateLLMsTxt.js";
 import { ScrapeTemplateTool } from "./src/tools/templates/ScrapeTemplateTool.js"; // D3.3
 import { UnifiedScrapeTool, SCRAPE_INPUT_SHAPE } from "./src/tools/scrape/unifiedScrape.js"; // D4 D1
 import { AgentTool } from "./src/tools/agent/agent.js"; // D4 D2
-import { StealthBrowserManager } from "./src/core/StealthBrowserManager.js";
+import { StealthBrowserManager, resolveStealthEngine } from "./src/core/StealthBrowserManager.js";
 import { LocalizationManager } from "./src/core/LocalizationManager.js";
 // Stealth scrape: format conversion + the pre-fetch compliance gate (G5/G6/G7)
 import * as cheerio from "cheerio";
@@ -108,7 +108,7 @@ if (configErrors.length > 0 && config.server.nodeEnv === 'production') {
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "6.7.0",
+  version: "6.9.0",
   description: "Production-ready MCP server with 31 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, stateful browser sessions with element refs, deep research, structured extraction, embedded JavaScript state extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",
@@ -243,23 +243,35 @@ const deepResearchTool = new DeepResearchTool();
 const trackChangesTool = new TrackChangesTool();
 const generateLLMsTxtTool = new GenerateLLMsTxtTool();
 const scrapeTemplateTool = new ScrapeTemplateTool(); // D3.3
+// Phase 3: the escalation stage for `scrape`'s escalate:true and the agent's
+// automatic stealth retry (stealth review Phase 3) — one function, so both
+// go through the same gate, engine resolver and server-level proxy list.
+// Injected so the tool modules never import StealthBrowserManager (that would pull a
+// browser dependency into every unit test that loads `scrape`). Same gate
+// and same browser the stealth_mode tool drives — no new evasion, and the
+// engine name is resolved here, beside its sibling, by the one resolver
+// every stealth entry point shares.
+const stealthEscalation = async ({ url, engine, respectRobots }) => {
+  const warnings = await stealthComplianceGate(url, respectRobots);
+  // "auto" prefers camoufox and falls back to chromium when its binary is
+  // missing. Mapping here instead ("camoufox" or else chromium) collapsed
+  // auto to chromium and hid the fallback; the resolver says which engine it
+  // landed on and why, and the caller is told.
+  const resolved = await resolveStealthEngine(engine);
+  if (resolved.fallbackWarning) warnings.push(resolved.fallbackWarning);
+  const scraped = await stealthBrowserManager.scrapeWithStealth({
+    url,
+    engine: resolved.engine
+  });
+  // The RESOLVED engine: `scrape` reports it as stealth.engine, where the
+  // requested "auto" would tell the caller nothing about what ran.
+  return { ...scraped, engine: resolved.engine, warnings };
+};
 const unifiedScrapeTool = new UnifiedScrapeTool({
   actionExecutor: scrapeWithActionsTool.actionExecutor, // D4 D1 (+v4.8 screenshot reuses the shared browser pool)
-  // Phase 3: the escalation stage for `scrape`'s escalate:true, injected so
-  // the tool module never imports StealthBrowserManager (that would pull a
-  // browser dependency into every unit test that loads `scrape`). Same gate
-  // and same browser the stealth_mode tool drives — no new evasion, and the
-  // "playwright" → chromium engine mapping stays here with its sibling.
-  escalateScrape: async ({ url, engine, respectRobots }) => {
-    const warnings = await stealthComplianceGate(url, respectRobots);
-    const scraped = await stealthBrowserManager.scrapeWithStealth({
-      url,
-      engine: engine === 'camoufox' ? 'camoufox' : 'chromium'
-    });
-    return { ...scraped, engine, warnings };
-  }
+  escalateScrape: stealthEscalation
 });
-const agentTool = new AgentTool(); // D4 D2
+const agentTool = new AgentTool({ escalateFetch: stealthEscalation }); // D4 D2 + stealth review Phase 3
 const stealthBrowserManager = new StealthBrowserManager();
 const localizationManager = new LocalizationManager();
 
@@ -913,7 +925,7 @@ registerToolIfEnabled("read_result", {
 
 // Tool: scrape_with_actions
 registerToolIfEnabled("scrape_with_actions", {
-  description: "Use this when you must interact with a page before scraping - login, click buttons, fill forms, scroll, or wait for dynamic content to load - for SPAs, login-gated content, or multi-step flows. Actions: snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript, select (dropdowns), hover, navigate. Start a chain with {type:\"snapshot\"} to list the page's interactive elements with stable refs (@e1, @e2 ...), then target those refs in later actions instead of guessing CSS selectors; navigation invalidates refs, so snapshot again after one. Set browserOptions.stealth:true to run the chain in the stealth browser. robots.txt is respected on every navigation. Screenshots from this tool are stored as crawlforge://screenshot/{actionId} resources. Not for pages that render without interaction (scrape) and not as the first attempt on a blocked site (stealth_mode operation:\"scrape\"). Cost: 5 credits. Example: scrape_with_actions({url: \"https://app.com/dashboard\", actions: [{type:\"snapshot\"},{type:\"type\",selector:\"@e2\",text:\"user@a.com\"},{type:\"click\",selector:\"@e4\"}]})",
+  description: "Use this when you must interact with a page before scraping - login, click buttons, fill forms, scroll, or wait for dynamic content to load - for SPAs, login-gated content, or multi-step flows. Actions: snapshot, wait, click, type, press, scroll, screenshot, executeJavaScript, select (dropdowns), hover, navigate. Start a chain with {type:\"snapshot\"} to list the page's interactive elements with stable refs (@e1, @e2 ...), then target those refs in later actions instead of guessing CSS selectors; navigation invalidates refs, so snapshot again after one. Set browserOptions.stealth:true to run the chain in the stealth browser, and browserOptions.engine to pick its engine (\"auto\" by default - camoufox when it is installed, Chromium otherwise, and the result says which ran). robots.txt is respected on every navigation. Screenshots from this tool are stored as crawlforge://screenshot/{actionId} resources. Not for pages that render without interaction (scrape) and not as the first attempt on a blocked site (stealth_mode operation:\"scrape\"). Cost: 5 credits. Example: scrape_with_actions({url: \"https://app.com/dashboard\", actions: [{type:\"snapshot\"},{type:\"type\",selector:\"@e2\",text:\"user@a.com\"},{type:\"click\",selector:\"@e4\"}]})",
   annotations: { title: "Scrape with Browser Actions", readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: true },
   inputSchema: {
     url: z.string().url().describe("The URL to scrape"),
@@ -985,7 +997,8 @@ registerToolIfEnabled("scrape_with_actions", {
       viewportWidth: z.number().min(800).max(1920).default(1280),
       viewportHeight: z.number().min(600).max(1080).default(720),
       timeout: z.number().min(10000).max(120000).default(30000),
-      stealth: z.boolean().default(false).describe("Run the action chain in the stealth browser (randomized fingerprint, WebRTC/canvas spoofing) instead of the standard browser pool. Renders JavaScript; it does not solve challenges.")
+      stealth: z.boolean().default(false).describe("Run the action chain in the stealth browser (randomized fingerprint, WebRTC/canvas spoofing) instead of the standard browser pool. Renders JavaScript; it does not solve challenges."),
+      engine: z.enum(["auto", "chromium", "camoufox", "playwright"]).default("auto").describe("Stealth engine for the chain, with stealth:true. \"auto\" (default) runs camoufox when it is installed and Chromium otherwise; the result's `engine` says which ran. \"camoufox\" is Firefox-based with a higher anti-detect score; \"chromium\" (= \"playwright\") forces Chromium. Refused without stealth:true, where the browser is always Chromium.")
     }).optional().describe("Browser configuration options"),
     extractionOptions: z.object({
       selectors: z.record(z.string()).optional(),
@@ -1028,13 +1041,14 @@ registerToolIfEnabled("scrape_with_actions", {
 
 // Tool: browser_session
 registerToolIfEnabled("browser_session", {
-  description: "Use this to drive a browser across several calls, keeping the page, its cookies and its login in between. The loop is: open a session on a URL, snapshot it to list the interactive elements with stable refs (@e1, @e2 ...), act on those refs, read the content, close. Because the page stays open you can look before each step instead of committing to a whole chain up front, so a wrong selector costs one call rather than all of them. Operations: open (url, stealth, ttl, activity_ttl, viewport), snapshot, act (the same action array as scrape_with_actions), read (formats), screenshot, close, list. Navigation invalidates refs, so snapshot again after one. robots.txt is respected on every navigation, and screenshots are stored as crawlforge://screenshot/{actionId} resources. A session expires 600s after it opens or 300s after its last use, whichever comes first, so close it when you are done. Not for a page that renders without interaction (scrape), and not for an interaction you can write out in advance - that is one scrape_with_actions call for 5. Cost: 3 credits to open; read 2; snapshot, act, screenshot, close and list 1 each. Example: browser_session({operation:\"open\", url:\"https://app.com/login\"}), then browser_session({operation:\"snapshot\", session_id:\"...\"})",
+  description: "Use this to drive a browser across several calls, keeping the page, its cookies and its login in between. The loop is: open a session on a URL, snapshot it to list the interactive elements with stable refs (@e1, @e2 ...), act on those refs, read the content, close. Because the page stays open you can look before each step instead of committing to a whole chain up front, so a wrong selector costs one call rather than all of them. Operations: open (url, stealth, engine, ttl, activity_ttl, viewport), snapshot, act (the same action array as scrape_with_actions), read (formats), screenshot, close, list. Navigation invalidates refs, so snapshot again after one. robots.txt is respected on every navigation, and screenshots are stored as crawlforge://screenshot/{actionId} resources. A session expires 600s after it opens or 300s after its last use, whichever comes first, so close it when you are done. Not for a page that renders without interaction (scrape), and not for an interaction you can write out in advance - that is one scrape_with_actions call for 5. Cost: 3 credits to open; read 2; snapshot, act, screenshot, close and list 1 each. Example: browser_session({operation:\"open\", url:\"https://app.com/login\"}), then browser_session({operation:\"snapshot\", session_id:\"...\"})",
   annotations: { title: "Browser Session", readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true },
   inputSchema: {
     operation: z.enum(["open", "snapshot", "act", "read", "screenshot", "close", "list"]).describe("open a session, observe it, act on it, read it, or close it"),
     session_id: z.string().optional().describe("The id returned by operation:\"open\". Required by every operation except open and list"),
     url: z.string().url().optional().describe("open: the URL to load the session on"),
     stealth: z.boolean().default(false).describe("open: run the session in the stealth browser"),
+    engine: z.enum(["auto", "chromium", "camoufox", "playwright"]).default("auto").describe("open: stealth engine for the session, with stealth:true. \"auto\" (default) runs camoufox when it is installed and Chromium otherwise; every operation echoes the `engine` that actually ran. \"camoufox\" is Firefox-based with a higher anti-detect score; \"chromium\" (= \"playwright\") forces Chromium. Refused without stealth:true, where the browser is always Chromium."),
     ttl: z.number().min(30).max(3600).optional().describe("open: seconds the session may live at most (default 600)"),
     activity_ttl: z.number().min(10).max(3600).optional().describe("open: seconds the session may sit idle (default 300)"),
     viewport: z.object({
@@ -1201,7 +1215,7 @@ registerToolIfEnabled("scrape", {
 
 // Tool: agent (D4 D2 — autonomous NL prompt → search/navigate/extract)
 registerToolIfEnabled("agent", {
-  description: "Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web - no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:\"pro\" uses deep multi-source research. Hard limits: maxSteps<=10, maxUrls<=20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Not for a URL you already have (scrape) or a question one search answers (search_web). Cost: 8 credits, scales with maxUrls. Example: agent({prompt:\"What are the top 5 MCP servers in 2025?\", maxUrls:10})",
+  description: "Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web - no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:\"pro\" uses deep multi-source research. Hard limits: maxSteps<=10, maxUrls<=20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Not for a URL you already have (scrape) or a question one search answers (search_web). Pages that block a plain fetch are retried in the stealth browser automatically (at most 2 a run, URLs you name first; evidence marked via:\"stealth\"). Cost: 18 credits at most - 8, plus 5 per stealth retry that gets the page; a retry that is blocked again is free. Example: agent({prompt:\"What are the top 5 MCP servers in 2025?\", maxUrls:10})",
   annotations: { title: "Agent (Autonomous)", readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: true },
   inputSchema: {
     prompt: z.string().min(1).max(2000).describe("Natural-language task or question"),
@@ -1337,7 +1351,7 @@ function stealthScrapeFormats(formats, scraped) {
 
 // Tool: stealth_mode
 registerToolIfEnabled("stealth_mode", {
-  description: "Use this when a site blocks normal scraping - Cloudflare, Datadome, or other bot-detection systems. Renders in a Playwright browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing. operation:\"scrape\" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. The create_context -> create_page -> cleanup operations remain for multi-step work. robots.txt is respected on every navigation. Not a first choice: try scrape first and switch here after a 403/429/CAPTCHA/challenge page or an empty shell. Cost: 5 credits per browser operation; configure, enable, disable, get_stats and cleanup cost 1. Example: stealth_mode({operation:\"scrape\", url:\"https://example.com\", formats:[\"markdown\",\"links\"]})",
+  description: "Use this when a site blocks normal scraping - Cloudflare, Datadome, or other bot-detection systems. Renders in a real browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing - Camoufox (Firefox) when it is installed, Chromium otherwise, and the result names the one that ran. operation:\"scrape\" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. The create_context -> create_page -> cleanup operations remain for multi-step work. robots.txt is respected on every navigation. Not a first choice: try scrape first and switch here after a 403/429/CAPTCHA/challenge page or an empty shell. Cost: 5 credits per browser operation; configure, enable, disable, get_stats and cleanup cost 1. Example: stealth_mode({operation:\"scrape\", url:\"https://example.com\", formats:[\"markdown\",\"links\"]})",
   annotations: { title: "Stealth Mode", readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true },
   inputSchema: {
     operation: z.enum(['scrape', 'configure', 'enable', 'disable', 'create_context', 'create_page', 'get_stats', 'cleanup']).default('configure').describe("Stealth operation to perform"),
@@ -1379,7 +1393,7 @@ registerToolIfEnabled("stealth_mode", {
         hardwareSpoofing: z.boolean().default(true)
       }).optional()
     }).optional().describe("Stealth browser configuration with anti-detection settings"),
-    engine: z.enum(["playwright", "camoufox"]).optional().default("playwright").describe("Browser engine: \"playwright\" (Chromium, default) or \"camoufox\" (Firefox-based, higher anti-detect score — install with npm install camoufox)"),
+    engine: z.enum(["auto", "chromium", "camoufox", "playwright"]).optional().default("auto").describe("Browser engine: \"auto\" (default — camoufox when it is installed, Chromium otherwise, and the result says which), \"camoufox\" (Firefox-based, higher anti-detect score; fails if not installed), or \"chromium\" (\"playwright\" is the same engine under its old name)"),
     contextId: z.string().optional().describe("Browser context ID for page operations"),
     urlToTest: z.string().url().optional().describe("URL to navigate to when creating a page"),
     url: z.string().url().optional().describe("URL to scrape — required for operation:\"scrape\""),
@@ -1399,12 +1413,16 @@ registerToolIfEnabled("stealth_mode", {
         // Gate first: a disallowed URL must never launch a browser.
         const warnings = await stealthComplianceGate(url, respect_robots);
 
+        // "auto" prefers camoufox and falls back to chromium when its binary
+        // is missing; "playwright" is this tool's old public name for chromium.
+        // The resolver owns both, and reports a fallback the caller can see.
+        const resolvedEngine = await resolveStealthEngine(engine);
+        if (resolvedEngine.fallbackWarning) warnings.push(resolvedEngine.fallbackWarning);
+
         const wantsScreenshot = formats.includes('screenshot');
         const scraped = await stealthBrowserManager.scrapeWithStealth({
           url,
-          // "playwright" is this tool's public name for the chromium engine
-          // (the manager and the CLI both call it chromium).
-          engine: engine === 'camoufox' ? 'camoufox' : 'chromium',
+          engine: resolvedEngine.engine,
           wait_for: wait_for || 0,
           screenshot: wantsScreenshot,
           stealthConfig
@@ -1421,6 +1439,7 @@ registerToolIfEnabled("stealth_mode", {
           url: scraped.url,
           title: scraped.title,
           status: verdict.status,
+          engine: resolvedEngine.engine,
           content: stealthScrapeFormats(formats, scraped)
         };
         if (verdict.blocked) result.blocked = verdict.blocked;
@@ -1457,10 +1476,11 @@ registerToolIfEnabled("stealth_mode", {
       case 'create_context': {
         // Forward the tool-level engine the same way operation:"scrape" does —
         // without it, create_context always ran on chromium whatever the caller
-        // asked for ("playwright" is this tool's public name for chromium).
+        // asked for. Resolved, so "auto" reaches the manager as a real engine.
+        const contextEngine = await resolveStealthEngine(engine);
         const contextData = await stealthBrowserManager.createStealthContext({
           ...(stealthConfig || {}),
-          engine: engine === 'camoufox' ? 'camoufox' : 'chromium'
+          engine: contextEngine.engine
         });
         // The full fingerprint is ~4 KB of canvas noise arrays and WebGL
         // extension lists no caller acts on. Summarise by default; verbose:true
@@ -1468,6 +1488,10 @@ registerToolIfEnabled("stealth_mode", {
         result = {
           contextId: contextData.contextId,
           created: true,
+          // Which engine the context is on, and why, when "auto" had to fall
+          // back: every page made from it inherits that engine.
+          engine: contextEngine.engine,
+          ...(contextEngine.fallbackWarning ? { warnings: [contextEngine.fallbackWarning] } : {}),
           fingerprint: verbose
             ? contextData.fingerprint
             : stealthBrowserManager.summarizeFingerprint(contextData.fingerprint)

package/src/constants/config.js

@@ -454,6 +454,28 @@ export function isStealthConfigured() {
   );
 }
 
+// The server operator's own exit IPs for every stealth path, from
+// CRAWLFORGE_STEALTH_PROXIES — a comma-separated list in proxyRotation's URL
+// format (http/https/socks4/socks5, credentials percent-encoded in the URL).
+//
+// Deliberately a function reading process.env rather than a field on `config`
+// above: a value frozen at import time cannot be changed for a single run, and
+// every consumer (the escalation stage, the deep_research fallback, the agent,
+// browser_session) resolves its proxy mid-process.
+//
+// Separate from the PROXY_ROTATION_* family under localization.proxy on
+// purpose. Those answer "which country should this page be fetched from", which
+// is the localization tool's whole job; this one answers "which address does
+// this server go out from", which is about not being the same datacentre IP a
+// wall has already seen. Overloading one on the other would make either
+// setting silently change the other's behaviour.
+export function serverStealthProxies() {
+  return (process.env.CRAWLFORGE_STEALTH_PROXIES || '')
+    .split(',')
+    .map((entry) => entry.trim())
+    .filter(Boolean);
+}
+
 // Get localization configuration
 export function getLocalizationConfig() {
   return config.localization;

package/src/core/AgentOrchestrator.js

@@ -4,7 +4,8 @@
  * Design: hardcoded 3-action state machine.
  *   PLAN   — one SamplingClient call to decompose prompt into search queries
  *   GATHER — search_web (≤maxUrls results total)
- *   ACT    — fetchAndParse + relevance gate per URL
+ *   ACT    — fetchAndParse + relevance gate per URL; a walled page is retried
+ *            once in the stealth browser (capped, see tools/agent/escalation.js)
  *   DECIDE — loop or answer (step/URL/time hard stops; never LLM-trusted)
  *   SHAPE  — schema→ExtractWithLlm prose→synthesis via SamplingClient
  *
@@ -20,10 +21,21 @@
 import { fetchAndParse } from '../tools/extract/_fetchAndParse.js';
 import { SamplingClient } from './SamplingClient.js';
 import { fenceUntrusted } from '../utils/untrustedContent.js';
+import { stealthDocumentVerdict as documentVerdict } from '../utils/stealthVerdict.js';
+import { aBrowserMightPass } from '../tools/scrape/escalation.js';
+import { AGENT_MAX_ESCALATIONS } from '../tools/agent/escalation.js';
 
 const DEFAULT_WALL_CLOCK_MS = 120_000;
 const DEFAULT_MAX_STEPS = 5;
 const DEFAULT_MAX_URLS = 10;
+// A stealth retry launches a browser and waits out a challenge; one started
+// with less than this left on the wall clock would overrun the hard stop.
+const ESCALATION_MIN_REMAINING_MS = 20_000;
+
+/** A refusal (robots.txt, blocklist) is a decision, not a wall — a browser must not retry it. */
+function isRefusal(err) {
+  return err?.code === 'ROBOTS_DISALLOWED' || err?.code === 'HOST_BLOCKED';
+}
 
 // ── Helpers ───────────────────────────────────────────────────────────────────
 
@@ -168,6 +180,11 @@ export class AgentOrchestrator {
    * @param {object|null} options.mcpServer  - McpServer instance (for SamplingClient)
    * @param {object}      options.searchConfig - passed to SearchWebTool constructor
    * @param {object}      options.llmConfig    - passed to ExtractWithLlm constructor
+   * @param {Function}    [options.escalateFetch] - the stealth stage, injected from
+   *   server.js (same function `scrape`'s escalate:true uses):
+   *   ({ url, engine, respectRobots }) => { html, url, title, text, status, engine, warnings? }
+   *   Absent in unit tests and builds without a browser: ACT then keeps the
+   *   plain-fetch-or-snippet behaviour.
    */
   constructor(options = {}) {
     this._mcpServer = options.mcpServer || null;
@@ -177,6 +194,7 @@ export class AgentOrchestrator {
     this._searchTool = null;
     this._extractWithLlm = null;
     this._researchOrchestrator = null;
+    this._escalateFetch = options.escalateFetch || null;
   }
 
   /** Set MCP server (called by agent.js after construction). */
@@ -237,6 +255,9 @@ export class AgentOrchestrator {
    * @param {number}    [params.maxSteps]  - Max ACT iterations (≤10)
    * @param {number}    [params.maxUrls]   - Max URLs to fetch (≤20)
    * @param {number}    [params.wallClockMs] - Wall-clock budget in ms
+   * @param {object}    [params.usage]     - out-param: `usage.escalations` counts
+   *   stealth retries that ran (the cap), `usage.charged` those that got the
+   *   page (the bill), so the tool can bill them even if run() throws
    * @returns {Promise<object>}
    */
   async run(params) {
@@ -247,8 +268,11 @@ export class AgentOrchestrator {
       model = 'default',
       maxSteps = DEFAULT_MAX_STEPS,
       maxUrls = DEFAULT_MAX_URLS,
-      wallClockMs = DEFAULT_WALL_CLOCK_MS
+      wallClockMs = DEFAULT_WALL_CLOCK_MS,
+      usage = {}
     } = params;
+    usage.escalations = 0;
+    usage.charged = 0;
 
     const startTime = Date.now();
     const deadline = () => (Date.now() - startTime) >= wallClockMs;
@@ -365,6 +389,10 @@ export class AgentOrchestrator {
       if (!urlQueue.includes(url)) urlQueue.push(url);
       if (!priorityUrls.includes(url)) priorityUrls.push(url);
     }
+    // The URLs the caller named — seeds and sites written into the prompt.
+    // Snapshotted before GATHER adds a voted live root to priorityUrls: that
+    // root is the agent's own discovery, not the caller's.
+    const namedUrls = new Set(priorityUrls);
     const searchResults = [];
     /** url -> { title, text }: the engine's excerpt, evidence of last resort for a page that cannot be fetched. */
     const excerpts = new Map();
@@ -441,32 +469,93 @@ export class AgentOrchestrator {
     const evidence = [];
     let urlsFetched = 0;
     let step = 0;
+    const warnings = [];
 
     for (const url of urlQueue) {
       if (urlsFetched >= capUrls || step >= capSteps || deadline()) break;
       urlsFetched++;
 
+      // The plain fetch reads an error document instead of throwing on it, so
+      // a wall served as a 403 can be told apart from a 404 (the same
+      // arrangement `scrape` has).
+      let plain = null;
+      let plainError = null;
+      let verdict = null;
       try {
-        const { textContent, finalUrl } = await fetchAndParse(url, { timeoutMs: 10000 });
-        if (!isRelevant(textContent, prompt)) continue;
-        step++;
-        const sr = searchResults.find(s => s.url === url);
-        evidence.push({
-          url: finalUrl,
-          title: sr ? sr.title : '',
-          text: truncate(textContent),
-          step
-        });
-      } catch {
-        // The page is out of reach (challenge page, 403, timeout) but the
-        // search engine's excerpt of it is not: npmjs.com's "Latest version:
-        // 15.0.0" sat in the snippet while every fetch of the package page was
-        // challenged, and the run ended with no evidence at all (R17,
-        // 2026-09-04). Keep a relevant snippet as evidence, labelled as one.
-        const excerpt = excerpts.get(url);
-        if (excerpt && isRelevant(excerpt.text, prompt)) {
-          evidence.push({ url, title: excerpt.title, text: excerpt.text, snippet: true, step });
+        plain = await fetchAndParse(url, { timeoutMs: 10000, errorDocuments: true });
+        verdict = documentVerdict(
+          { url: plain.finalUrl, title: plain.$('title').first().text().trim(), text: plain.textContent, html: plain.html, status: plain.status },
+          { fetcher: 'a plain fetch', rendered: false, contentReturned: false }
+        );
+      } catch (err) {
+        plainError = err;
+      }
+
+      const ok2xx = plain && plain.status >= 200 && plain.status < 300;
+      if (ok2xx && !verdict.blocked) {
+        if (isRelevant(plain.textContent, prompt)) {
+          step++;
+          const sr = searchResults.find(s => s.url === url);
+          evidence.push({
+            url: plain.finalUrl,
+            title: sr ? sr.title : '',
+            text: truncate(plain.textContent),
+            step
+          });
+          continue;
         }
+        // A real page that is simply off-topic: nothing to retry.
+        if (verdict.success) continue;
+      }
+
+      // The page is out of reach: a wall, a 403/429, an empty shell, a
+      // timeout. A search excerpt may still carry the fact (npmjs.com's
+      // "Latest version: 15.0.0" sat in the snippet while every fetch of the
+      // package page was challenged, R17 2026-09-04).
+      const excerpt = excerpts.get(url);
+      const relevantExcerpt = excerpt && isRelevant(excerpt.text, prompt) ? excerpt : null;
+
+      // Stealth retry (review Phase 3). A URL the caller named always gets it
+      // first; a discovered one only when there is no relevant excerpt to fall
+      // back on. Never for a refusal, a 404 or a 5xx; bounded by its own cap
+      // and by the time left before the wall-clock stop.
+      const mightPass = plainError ? !isRefusal(plainError) : aBrowserMightPass(verdict, plain.status);
+      const canEscalate = this._escalateFetch &&
+        usage.escalations < Math.min(AGENT_MAX_ESCALATIONS, capUrls) &&
+        (wallClockMs - (Date.now() - startTime)) >= ESCALATION_MIN_REMAINING_MS;
+      if (mightPass && canEscalate && (namedUrls.has(url) || !relevantExcerpt)) {
+        usage.escalations++;
+        try {
+          const stealth = await this._escalateFetch({ url: plain?.finalUrl || url, engine: 'auto' });
+          if (Array.isArray(stealth.warnings)) warnings.push(...stealth.warnings);
+          const text = (stealth.text || '').trim();
+          const rendered = documentVerdict(
+            { url: stealth.url || url, title: stealth.title || '', text, html: stealth.html || '', status: stealth.status ?? null },
+            { waitedMs: stealth.gracedMs || 0, fetcher: 'the stealth browser', rendered: true, contentReturned: false }
+          );
+          // Billed only when the browser got past the wall: a retry that met
+          // the wall again, or threw, costs nothing (owner decision).
+          if (rendered.success) usage.charged++;
+          if (rendered.success && isRelevant(text, prompt)) {
+            step++;
+            const sr = searchResults.find(s => s.url === url);
+            evidence.push({
+              url: stealth.url || url,
+              title: stealth.title || (sr ? sr.title : ''),
+              text: truncate(text),
+              via: 'stealth',
+              step
+            });
+            continue;
+          }
+          warnings.push(`stealth retry of ${url} ${rendered.success ? 'returned a page unrelated to the task' : `did not get the page either${rendered.blocked?.vendor ? ` (${rendered.blocked.vendor})` : ''}`}`);
+        } catch (err) {
+          warnings.push(`stealth retry of ${url} failed: ${err.message}`);
+        }
+      }
+
+      if (relevantExcerpt) {
+        evidence.push({ url, title: relevantExcerpt.title, text: relevantExcerpt.text, snippet: true, step });
       }
     }
 
@@ -501,7 +590,10 @@ export class AgentOrchestrator {
         evidence: [],
         answer: null,
         steps: step,
-        urls_fetched: urlsFetched
+        urls_fetched: urlsFetched,
+        stealth_retries: usage.escalations,
+        stealth_retries_charged: usage.charged,
+        ...(warnings.length ? { warnings } : {})
       };
     }
 
@@ -520,11 +612,14 @@ export class AgentOrchestrator {
           answer: result.success ? result.data : null,
           structured: true,
           search_results: searchResults,
-          evidence: evidence.map(e => ({ url: e.url })),
+          evidence: evidence.map(e => (e.via ? { url: e.url, via: e.via } : { url: e.url })),
           degraded: !result.success,
           reason: result.success ? undefined : result.error,
           steps: step,
-          urls_fetched: urlsFetched
+          urls_fetched: urlsFetched,
+          stealth_retries: usage.escalations,
+        stealth_retries_charged: usage.charged,
+          ...(warnings.length ? { warnings } : {})
         };
       } catch (err) {
         // Fall through to prose synthesis
@@ -648,11 +743,14 @@ export class AgentOrchestrator {
       success: true,
       answer,
       search_results: searchResults,
-      evidence: degraded ? evidence : evidence.map(e => (e.snippet ? { url: e.url, snippet: true } : { url: e.url })),
+      evidence: degraded ? evidence : evidence.map(e => (e.snippet ? { url: e.url, snippet: true } : e.via ? { url: e.url, via: e.via } : { url: e.url })),
       degraded,
       reason: degradedReason,
       steps: step,
       urls_fetched: urlsFetched,
+      stealth_retries: usage.escalations,
+        stealth_retries_charged: usage.charged,
+      ...(warnings.length ? { warnings } : {}),
       provenance: {
         checked: !synthesisFailed,
         unverified,