crawlforge-mcp-server 6.8.0 → 6.9.0

package/CLAUDE.md

@@ -234,7 +234,8 @@ CRAWLFORGE_STEALTH_PROXIES=          # comma-separated proxy URLs used by the es
                                      # passed on the call wins). CrawlForge supplies no proxies, and
                                      # a datacenter one does not defeat these detectors. Distinct
                                      # from the PROXY_ROTATION_* family, which belongs to
-                                     # localization. The agent tool does not browse yet (Phase 3).
+                                     # localization. The agent's automatic stealth retry
+                                     # (stealth review Phase 3) reads it too.
 ```
 
 Camoufox is an **optional** dependency and its transitive `language-tags@2.1.0` requires Node ≥22. npm silently drops an optional subtree that fails an engine check, so on Node 20 it is simply absent — `'auto'` then resolves to Chromium and says so in `fallbackWarning`. `engines.node` stays at `>=20.16.0` deliberately; bumping it is breaking and is a release decision.

package/README.md

@@ -206,7 +206,7 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `scrape_with_actions` | 5 | Browser automation chains |
 | `generate_llms_txt` | 5 | Generate AI interaction guidelines |
 | `stealth_mode` | 5 | Anti-detection browser management |
-| `agent` | 8 | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) |
+| `agent` | 8 (+5 per stealth retry that gets the page, max 2) | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM). A page that walls the plain fetch is retried in the stealth browser automatically — URLs you name first — and that evidence is marked `via: "stealth"`; each retry that runs adds 5 |
 | `deep_research` | 10 | Multi-stage research with source verification |
 
 Ten tools (`scrape`, `fetch_url`, `extract_content`, `crawl_deep`, `batch_scrape`, `stealth_mode`, `scrape_with_actions`, `process_document`, `deep_research`, `extract_embedded_state`) accept `max_inline_chars` (default 40,000; env `CRAWLFORGE_MAX_INLINE_CHARS`): a result over it comes back as a `preview` plus a `result_handle` for `read_result`, with the full result kept for 1 hour under `~/.crawlforge/results/` on your own machine — nothing is uploaded.
@@ -311,7 +311,7 @@ On `browser_session` and `scrape_with_actions` the engine applies only with `ste
 
 `auto` is not free. Measured once each on an Apple Silicon Mac on 2026-09-21 (launch plus a context and a blank page): Chromium 148 ms and 253 MB, Camoufox 946 ms and 667 MB — roughly **+0.8 s and +400 MB per stealth call**. Pin `engine: "playwright"` for high-volume work on sites that do not block.
 
-**CrawlForge supplies no proxies**, and a datacenter proxy does not fix a block: Cloudflare scores the IP's ASN and the TLS/HTTP2 handshake before any JavaScript runs, so no browser-side patch compensates for a datacenter address. Bring your own residential exit with `CRAWLFORGE_STEALTH_PROXIES` (comma-separated URLs), which the escalation stage, `stealth_mode`, `browser_session`, `scrape_with_actions` and the `deep_research` retry use when the caller passes none; a proxy passed on the call always wins. (The `agent` tool does not browse yet, so it is not in that list — it joins when Phase 3 gives it an escalation path.) With a proxy, Camoufox derives its timezone, locale and geolocation from the exit IP.
+**CrawlForge supplies no proxies**, and a datacenter proxy does not fix a block: Cloudflare scores the IP's ASN and the TLS/HTTP2 handshake before any JavaScript runs, so no browser-side patch compensates for a datacenter address. Bring your own residential exit with `CRAWLFORGE_STEALTH_PROXIES` (comma-separated URLs), which the escalation stage, `stealth_mode`, `browser_session`, `scrape_with_actions`, the `deep_research` retry and the `agent` tool's automatic stealth retry use when the caller passes none; a proxy passed on the call always wins. With a proxy, Camoufox derives its timezone, locale and geolocation from the exit IP.
 
 Full detail, measurements and sources: [docs/stealth-engines.md](docs/stealth-engines.md).
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "6.8.0",
+  "version": "6.9.0",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
@@ -157,6 +157,6 @@
     "@hono/node-server": "2.0.12",
     "hono": "^4.12.4",
     "dompurify": "^3.4.0",
-    "adm-zip": "^0.6.0"
+    "adm-zip": "^0.6.1"
   }
 }

package/server.js

@@ -108,7 +108,7 @@ if (configErrors.length > 0 && config.server.nodeEnv === 'production') {
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "6.8.0",
+  version: "6.9.0",
   description: "Production-ready MCP server with 31 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, stateful browser sessions with element refs, deep research, structured extraction, embedded JavaScript state extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",
@@ -243,32 +243,35 @@ const deepResearchTool = new DeepResearchTool();
 const trackChangesTool = new TrackChangesTool();
 const generateLLMsTxtTool = new GenerateLLMsTxtTool();
 const scrapeTemplateTool = new ScrapeTemplateTool(); // D3.3
+// Phase 3: the escalation stage for `scrape`'s escalate:true and the agent's
+// automatic stealth retry (stealth review Phase 3) — one function, so both
+// go through the same gate, engine resolver and server-level proxy list.
+// Injected so the tool modules never import StealthBrowserManager (that would pull a
+// browser dependency into every unit test that loads `scrape`). Same gate
+// and same browser the stealth_mode tool drives — no new evasion, and the
+// engine name is resolved here, beside its sibling, by the one resolver
+// every stealth entry point shares.
+const stealthEscalation = async ({ url, engine, respectRobots }) => {
+  const warnings = await stealthComplianceGate(url, respectRobots);
+  // "auto" prefers camoufox and falls back to chromium when its binary is
+  // missing. Mapping here instead ("camoufox" or else chromium) collapsed
+  // auto to chromium and hid the fallback; the resolver says which engine it
+  // landed on and why, and the caller is told.
+  const resolved = await resolveStealthEngine(engine);
+  if (resolved.fallbackWarning) warnings.push(resolved.fallbackWarning);
+  const scraped = await stealthBrowserManager.scrapeWithStealth({
+    url,
+    engine: resolved.engine
+  });
+  // The RESOLVED engine: `scrape` reports it as stealth.engine, where the
+  // requested "auto" would tell the caller nothing about what ran.
+  return { ...scraped, engine: resolved.engine, warnings };
+};
 const unifiedScrapeTool = new UnifiedScrapeTool({
   actionExecutor: scrapeWithActionsTool.actionExecutor, // D4 D1 (+v4.8 screenshot reuses the shared browser pool)
-  // Phase 3: the escalation stage for `scrape`'s escalate:true, injected so
-  // the tool module never imports StealthBrowserManager (that would pull a
-  // browser dependency into every unit test that loads `scrape`). Same gate
-  // and same browser the stealth_mode tool drives — no new evasion, and the
-  // engine name is resolved here, beside its sibling, by the one resolver
-  // every stealth entry point shares.
-  escalateScrape: async ({ url, engine, respectRobots }) => {
-    const warnings = await stealthComplianceGate(url, respectRobots);
-    // "auto" prefers camoufox and falls back to chromium when its binary is
-    // missing. Mapping here instead ("camoufox" or else chromium) collapsed
-    // auto to chromium and hid the fallback; the resolver says which engine it
-    // landed on and why, and the caller is told.
-    const resolved = await resolveStealthEngine(engine);
-    if (resolved.fallbackWarning) warnings.push(resolved.fallbackWarning);
-    const scraped = await stealthBrowserManager.scrapeWithStealth({
-      url,
-      engine: resolved.engine
-    });
-    // The RESOLVED engine: `scrape` reports it as stealth.engine, where the
-    // requested "auto" would tell the caller nothing about what ran.
-    return { ...scraped, engine: resolved.engine, warnings };
-  }
+  escalateScrape: stealthEscalation
 });
-const agentTool = new AgentTool(); // D4 D2
+const agentTool = new AgentTool({ escalateFetch: stealthEscalation }); // D4 D2 + stealth review Phase 3
 const stealthBrowserManager = new StealthBrowserManager();
 const localizationManager = new LocalizationManager();
 
@@ -1212,7 +1215,7 @@ registerToolIfEnabled("scrape", {
 
 // Tool: agent (D4 D2 — autonomous NL prompt → search/navigate/extract)
 registerToolIfEnabled("agent", {
-  description: "Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web - no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:\"pro\" uses deep multi-source research. Hard limits: maxSteps<=10, maxUrls<=20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Not for a URL you already have (scrape) or a question one search answers (search_web). Cost: 8 credits, scales with maxUrls. Example: agent({prompt:\"What are the top 5 MCP servers in 2025?\", maxUrls:10})",
+  description: "Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web - no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:\"pro\" uses deep multi-source research. Hard limits: maxSteps<=10, maxUrls<=20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Not for a URL you already have (scrape) or a question one search answers (search_web). Pages that block a plain fetch are retried in the stealth browser automatically (at most 2 a run, URLs you name first; evidence marked via:\"stealth\"). Cost: 18 credits at most - 8, plus 5 per stealth retry that gets the page; a retry that is blocked again is free. Example: agent({prompt:\"What are the top 5 MCP servers in 2025?\", maxUrls:10})",
   annotations: { title: "Agent (Autonomous)", readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: true },
   inputSchema: {
     prompt: z.string().min(1).max(2000).describe("Natural-language task or question"),

package/src/core/AgentOrchestrator.js

@@ -4,7 +4,8 @@
  * Design: hardcoded 3-action state machine.
  *   PLAN   — one SamplingClient call to decompose prompt into search queries
  *   GATHER — search_web (≤maxUrls results total)
- *   ACT    — fetchAndParse + relevance gate per URL
+ *   ACT    — fetchAndParse + relevance gate per URL; a walled page is retried
+ *            once in the stealth browser (capped, see tools/agent/escalation.js)
  *   DECIDE — loop or answer (step/URL/time hard stops; never LLM-trusted)
  *   SHAPE  — schema→ExtractWithLlm prose→synthesis via SamplingClient
  *
@@ -20,10 +21,21 @@
 import { fetchAndParse } from '../tools/extract/_fetchAndParse.js';
 import { SamplingClient } from './SamplingClient.js';
 import { fenceUntrusted } from '../utils/untrustedContent.js';
+import { stealthDocumentVerdict as documentVerdict } from '../utils/stealthVerdict.js';
+import { aBrowserMightPass } from '../tools/scrape/escalation.js';
+import { AGENT_MAX_ESCALATIONS } from '../tools/agent/escalation.js';
 
 const DEFAULT_WALL_CLOCK_MS = 120_000;
 const DEFAULT_MAX_STEPS = 5;
 const DEFAULT_MAX_URLS = 10;
+// A stealth retry launches a browser and waits out a challenge; one started
+// with less than this left on the wall clock would overrun the hard stop.
+const ESCALATION_MIN_REMAINING_MS = 20_000;
+
+/** A refusal (robots.txt, blocklist) is a decision, not a wall — a browser must not retry it. */
+function isRefusal(err) {
+  return err?.code === 'ROBOTS_DISALLOWED' || err?.code === 'HOST_BLOCKED';
+}
 
 // ── Helpers ───────────────────────────────────────────────────────────────────
 
@@ -168,6 +180,11 @@ export class AgentOrchestrator {
    * @param {object|null} options.mcpServer  - McpServer instance (for SamplingClient)
    * @param {object}      options.searchConfig - passed to SearchWebTool constructor
    * @param {object}      options.llmConfig    - passed to ExtractWithLlm constructor
+   * @param {Function}    [options.escalateFetch] - the stealth stage, injected from
+   *   server.js (same function `scrape`'s escalate:true uses):
+   *   ({ url, engine, respectRobots }) => { html, url, title, text, status, engine, warnings? }
+   *   Absent in unit tests and builds without a browser: ACT then keeps the
+   *   plain-fetch-or-snippet behaviour.
    */
   constructor(options = {}) {
     this._mcpServer = options.mcpServer || null;
@@ -177,6 +194,7 @@ export class AgentOrchestrator {
     this._searchTool = null;
     this._extractWithLlm = null;
     this._researchOrchestrator = null;
+    this._escalateFetch = options.escalateFetch || null;
   }
 
   /** Set MCP server (called by agent.js after construction). */
@@ -237,6 +255,9 @@ export class AgentOrchestrator {
    * @param {number}    [params.maxSteps]  - Max ACT iterations (≤10)
    * @param {number}    [params.maxUrls]   - Max URLs to fetch (≤20)
    * @param {number}    [params.wallClockMs] - Wall-clock budget in ms
+   * @param {object}    [params.usage]     - out-param: `usage.escalations` counts
+   *   stealth retries that ran (the cap), `usage.charged` those that got the
+   *   page (the bill), so the tool can bill them even if run() throws
    * @returns {Promise<object>}
    */
   async run(params) {
@@ -247,8 +268,11 @@ export class AgentOrchestrator {
       model = 'default',
       maxSteps = DEFAULT_MAX_STEPS,
       maxUrls = DEFAULT_MAX_URLS,
-      wallClockMs = DEFAULT_WALL_CLOCK_MS
+      wallClockMs = DEFAULT_WALL_CLOCK_MS,
+      usage = {}
     } = params;
+    usage.escalations = 0;
+    usage.charged = 0;
 
     const startTime = Date.now();
     const deadline = () => (Date.now() - startTime) >= wallClockMs;
@@ -365,6 +389,10 @@ export class AgentOrchestrator {
       if (!urlQueue.includes(url)) urlQueue.push(url);
       if (!priorityUrls.includes(url)) priorityUrls.push(url);
     }
+    // The URLs the caller named — seeds and sites written into the prompt.
+    // Snapshotted before GATHER adds a voted live root to priorityUrls: that
+    // root is the agent's own discovery, not the caller's.
+    const namedUrls = new Set(priorityUrls);
     const searchResults = [];
     /** url -> { title, text }: the engine's excerpt, evidence of last resort for a page that cannot be fetched. */
     const excerpts = new Map();
@@ -441,32 +469,93 @@ export class AgentOrchestrator {
     const evidence = [];
     let urlsFetched = 0;
     let step = 0;
+    const warnings = [];
 
     for (const url of urlQueue) {
       if (urlsFetched >= capUrls || step >= capSteps || deadline()) break;
       urlsFetched++;
 
+      // The plain fetch reads an error document instead of throwing on it, so
+      // a wall served as a 403 can be told apart from a 404 (the same
+      // arrangement `scrape` has).
+      let plain = null;
+      let plainError = null;
+      let verdict = null;
       try {
-        const { textContent, finalUrl } = await fetchAndParse(url, { timeoutMs: 10000 });
-        if (!isRelevant(textContent, prompt)) continue;
-        step++;
-        const sr = searchResults.find(s => s.url === url);
-        evidence.push({
-          url: finalUrl,
-          title: sr ? sr.title : '',
-          text: truncate(textContent),
-          step
-        });
-      } catch {
-        // The page is out of reach (challenge page, 403, timeout) but the
-        // search engine's excerpt of it is not: npmjs.com's "Latest version:
-        // 15.0.0" sat in the snippet while every fetch of the package page was
-        // challenged, and the run ended with no evidence at all (R17,
-        // 2026-09-04). Keep a relevant snippet as evidence, labelled as one.
-        const excerpt = excerpts.get(url);
-        if (excerpt && isRelevant(excerpt.text, prompt)) {
-          evidence.push({ url, title: excerpt.title, text: excerpt.text, snippet: true, step });
+        plain = await fetchAndParse(url, { timeoutMs: 10000, errorDocuments: true });
+        verdict = documentVerdict(
+          { url: plain.finalUrl, title: plain.$('title').first().text().trim(), text: plain.textContent, html: plain.html, status: plain.status },
+          { fetcher: 'a plain fetch', rendered: false, contentReturned: false }
+        );
+      } catch (err) {
+        plainError = err;
+      }
+
+      const ok2xx = plain && plain.status >= 200 && plain.status < 300;
+      if (ok2xx && !verdict.blocked) {
+        if (isRelevant(plain.textContent, prompt)) {
+          step++;
+          const sr = searchResults.find(s => s.url === url);
+          evidence.push({
+            url: plain.finalUrl,
+            title: sr ? sr.title : '',
+            text: truncate(plain.textContent),
+            step
+          });
+          continue;
         }
+        // A real page that is simply off-topic: nothing to retry.
+        if (verdict.success) continue;
+      }
+
+      // The page is out of reach: a wall, a 403/429, an empty shell, a
+      // timeout. A search excerpt may still carry the fact (npmjs.com's
+      // "Latest version: 15.0.0" sat in the snippet while every fetch of the
+      // package page was challenged, R17 2026-09-04).
+      const excerpt = excerpts.get(url);
+      const relevantExcerpt = excerpt && isRelevant(excerpt.text, prompt) ? excerpt : null;
+
+      // Stealth retry (review Phase 3). A URL the caller named always gets it
+      // first; a discovered one only when there is no relevant excerpt to fall
+      // back on. Never for a refusal, a 404 or a 5xx; bounded by its own cap
+      // and by the time left before the wall-clock stop.
+      const mightPass = plainError ? !isRefusal(plainError) : aBrowserMightPass(verdict, plain.status);
+      const canEscalate = this._escalateFetch &&
+        usage.escalations < Math.min(AGENT_MAX_ESCALATIONS, capUrls) &&
+        (wallClockMs - (Date.now() - startTime)) >= ESCALATION_MIN_REMAINING_MS;
+      if (mightPass && canEscalate && (namedUrls.has(url) || !relevantExcerpt)) {
+        usage.escalations++;
+        try {
+          const stealth = await this._escalateFetch({ url: plain?.finalUrl || url, engine: 'auto' });
+          if (Array.isArray(stealth.warnings)) warnings.push(...stealth.warnings);
+          const text = (stealth.text || '').trim();
+          const rendered = documentVerdict(
+            { url: stealth.url || url, title: stealth.title || '', text, html: stealth.html || '', status: stealth.status ?? null },
+            { waitedMs: stealth.gracedMs || 0, fetcher: 'the stealth browser', rendered: true, contentReturned: false }
+          );
+          // Billed only when the browser got past the wall: a retry that met
+          // the wall again, or threw, costs nothing (owner decision).
+          if (rendered.success) usage.charged++;
+          if (rendered.success && isRelevant(text, prompt)) {
+            step++;
+            const sr = searchResults.find(s => s.url === url);
+            evidence.push({
+              url: stealth.url || url,
+              title: stealth.title || (sr ? sr.title : ''),
+              text: truncate(text),
+              via: 'stealth',
+              step
+            });
+            continue;
+          }
+          warnings.push(`stealth retry of ${url} ${rendered.success ? 'returned a page unrelated to the task' : `did not get the page either${rendered.blocked?.vendor ? ` (${rendered.blocked.vendor})` : ''}`}`);
+        } catch (err) {
+          warnings.push(`stealth retry of ${url} failed: ${err.message}`);
+        }
+      }
+
+      if (relevantExcerpt) {
+        evidence.push({ url, title: relevantExcerpt.title, text: relevantExcerpt.text, snippet: true, step });
       }
     }
 
@@ -501,7 +590,10 @@ export class AgentOrchestrator {
         evidence: [],
         answer: null,
         steps: step,
-        urls_fetched: urlsFetched
+        urls_fetched: urlsFetched,
+        stealth_retries: usage.escalations,
+        stealth_retries_charged: usage.charged,
+        ...(warnings.length ? { warnings } : {})
       };
     }
 
@@ -520,11 +612,14 @@ export class AgentOrchestrator {
           answer: result.success ? result.data : null,
           structured: true,
           search_results: searchResults,
-          evidence: evidence.map(e => ({ url: e.url })),
+          evidence: evidence.map(e => (e.via ? { url: e.url, via: e.via } : { url: e.url })),
           degraded: !result.success,
           reason: result.success ? undefined : result.error,
           steps: step,
-          urls_fetched: urlsFetched
+          urls_fetched: urlsFetched,
+          stealth_retries: usage.escalations,
+        stealth_retries_charged: usage.charged,
+          ...(warnings.length ? { warnings } : {})
         };
       } catch (err) {
         // Fall through to prose synthesis
@@ -648,11 +743,14 @@ export class AgentOrchestrator {
       success: true,
       answer,
       search_results: searchResults,
-      evidence: degraded ? evidence : evidence.map(e => (e.snippet ? { url: e.url, snippet: true } : { url: e.url })),
+      evidence: degraded ? evidence : evidence.map(e => (e.snippet ? { url: e.url, snippet: true } : e.via ? { url: e.url, via: e.via } : { url: e.url })),
       degraded,
       reason: degradedReason,
       steps: step,
       urls_fetched: urlsFetched,
+      stealth_retries: usage.escalations,
+        stealth_retries_charged: usage.charged,
+      ...(warnings.length ? { warnings } : {}),
       provenance: {
         checked: !synthesisFailed,
         unverified,

package/src/core/AuthManager.js

@@ -14,6 +14,7 @@ import { logger } from '../utils/Logger.js';
 import { maskSecrets } from '../utils/secretMask.js';
 import { scrapeFormatSurcharge } from '../tools/scrape/formats.js';
 import { scrapeEscalationSurcharge } from '../tools/scrape/escalation.js';
+import { agentEscalationSurcharge } from '../tools/agent/escalation.js';
 import { searchQueryCount } from '../tools/search/batchSearch.js';
 import { redactionSurcharge, REDACT_PII_MODEL_CREDITS } from '../server/redaction.js';
 // Stamped on every usage report so support can tell which client version
@@ -686,6 +687,14 @@ class AuthManager {
       return costs.search_web * searchQueryCount(params?.queries) + redaction;
     }
 
+    // agent retries a walled page in the stealth browser automatically, at
+    // most AGENT_MAX_ESCALATIONS times a run, 5 each. The projection is that
+    // ceiling; the tool reports what actually ran (setActualCost), so a run
+    // with no retry still costs 8. The rule lives in tools/agent/escalation.js.
+    if (tool === 'agent') {
+      return costs.agent + agentEscalationSurcharge(params) + redaction;
+    }
+
     return (costs[tool] ?? 1) + redaction;
   }
 
@@ -761,7 +770,7 @@ class AuthManager {
         const agentUrls = params?.maxUrls || 10;
         const isPro = params?.model === 'pro';
         projected = Math.max(base, base + Math.ceil(agentUrls / 5) + (isPro ? 5 : 0));
-        note = `Lower-bound estimate. Scales with maxUrls (${agentUrls}).${isPro ? ' pro model adds deep-research cost.' : ''} External LLM billed separately.`;
+        note = `Lower-bound estimate. Scales with maxUrls (${agentUrls}).${isPro ? ' pro model adds deep-research cost.' : ' Includes the ceiling for automatic stealth retries of walled pages (5 each, at most 2 a run); the actual charge drops by 5 for each retry that did not run or did not get the page.'} External LLM billed separately.`;
         break;
       }
       default:

package/src/skills/agent-skills/crawlforge-deep-research/SKILL.md

@@ -31,7 +31,7 @@ skill (`scrape` / `extract_content`) instead.
 |------|------|------|
 | A ranked list of result URLs + snippets | `search_web` | 5 |
 | Reddit posts, comments, or a full thread | `reddit_search` | 5 |
-| A direct answer, agent decides what to read | `agent` | 8 (scales) |
+| A direct answer, agent decides what to read | `agent` | 8 (+5 per automatic stealth retry, max 2) |
 | A synthesized, multi-source, cited report | `deep_research` | 10+ (scales) |
 
 A common pipeline: `search_web` to find sources → `batch_scrape` (see

package/src/skills/agent-skills/crawlforge-stealth-browsing/SKILL.md

@@ -93,8 +93,8 @@ scored. Route through your own residential proxy — CrawlForge supplies none.
 
 Server-wide, set `CRAWLFORGE_STEALTH_PROXIES` (comma-separated proxy URLs): the
 `scrape` escalation stage, `stealth_mode`, `browser_session`,
-`scrape_with_actions` and the `deep_research` retry use it when no proxy is
-passed on the call. (`agent` does not browse yet, so it is not a reader.) Per call, pass
+`scrape_with_actions`, the `deep_research` retry and the `agent` tool's automatic
+stealth retry use it when no proxy is passed on the call. Per call, pass
 `stealthConfig.proxyRotation`, which always wins:
 
 ```json

package/src/skills/agent-skills/crawlforge-stealth-browsing/references/engine-selection.md

@@ -69,8 +69,8 @@ them default to `auto`.
 - `proxyRotation { enabled, proxies[], rotationInterval }` — your own proxies,
   per call. Server-wide, `CRAWLFORGE_STEALTH_PROXIES` (comma-separated URLs of
   the same form) covers the paths with no caller to ask: the `scrape` escalation
-  stage, `stealth_mode`, `browser_session`, `scrape_with_actions` and the
-  `deep_research` retry — not `agent`, which has no browsing path yet. A call's
+  stage, `stealth_mode`, `browser_session`, `scrape_with_actions`, the
+  `deep_research` retry and the `agent` tool's automatic stealth retry. A call's
   own `proxyRotation` always wins. CrawlForge supplies no proxies, and a datacenter
   proxy does not help — Cloudflare scores the ASN before it serves a challenge.
   Unrelated to the `PROXY_ROTATION_*` variables, which belong to `localization`.

package/src/tools/agent/agent.js

@@ -9,6 +9,11 @@ import { z } from 'zod';
 import { AgentOrchestrator } from '../../core/AgentOrchestrator.js';
 import { ElicitationHelper } from '../../core/ElicitationHelper.js';
 import { getToolConfig } from '../../constants/config.js';
+import { setActualCost } from '../../server/requestContext.js';
+import { AGENT_ESCALATION_CREDITS } from './escalation.js';
+
+/** The agent's published price before any stealth retry. */
+export const AGENT_BASE_CREDITS = 8;
 
 export const AgentInputSchema = z.object({
   prompt: z.string().min(1).max(2000).describe('Natural-language task or question'),
@@ -24,7 +29,10 @@ export class AgentTool {
     this._orchestrator = new AgentOrchestrator({
       mcpServer: null,
       searchConfig: getToolConfig('search_web') || {},
-      llmConfig: options.llmConfig || {}
+      llmConfig: options.llmConfig || {},
+      // The stealth stage, injected from server.js so this module never
+      // imports a browser (see AgentOrchestrator's constructor).
+      escalateFetch: options.escalateFetch || null
     });
     this._elicitation = new ElicitationHelper({});
   }
@@ -58,14 +66,26 @@ export class AgentTool {
       }
     }
 
-    return this._orchestrator.run({
-      prompt: validated.prompt,
-      urls: validated.urls,
-      schema: validated.schema,
-      model: validated.model,
-      maxSteps: validated.maxSteps,
-      maxUrls: validated.maxUrls
-    });
+    // Priced like `scrape`'s escalation: getToolCost projects the ceiling
+    // (base + every retry the cap allows), and the charge is lowered to the
+    // retries that got the page — including when run() throws. A retry that
+    // met the wall again is free.
+    const usage = { escalations: 0, charged: 0 };
+    try {
+      return await this._orchestrator.run({
+        prompt: validated.prompt,
+        urls: validated.urls,
+        schema: validated.schema,
+        model: validated.model,
+        maxSteps: validated.maxSteps,
+        maxUrls: validated.maxUrls,
+        usage
+      });
+    } finally {
+      if (validated.model !== 'pro') {
+        setActualCost(AGENT_BASE_CREDITS + AGENT_ESCALATION_CREDITS * usage.charged);
+      }
+    }
   }
 
   async destroy() {

package/src/tools/agent/escalation.js

@@ -0,0 +1,40 @@
+/**
+ * escalation.js — the one declaration of the `agent` tool's automatic stealth
+ * retry (stealth review Phase 3, decision 4: automatic, capped).
+ *
+ * When ACT's plain fetch comes back walled (challenge page, 403/429, empty
+ * shell, timeout), the agent retries the URL once through the same stealth
+ * stage `scrape`'s escalate:true uses. It is not opt-in, so it is capped per
+ * run and priced per retry that gets the page; one that meets the wall again
+ * is free.
+ *
+ * The cap and the price live here so the orchestrator, the tool wrapper and
+ * `AuthManager.getToolCost` read one declaration — the arrangement
+ * `src/tools/scrape/escalation.js` has for `scrape`. Imports nothing, so
+ * AuthManager can read it without pulling in the orchestrator.
+ */
+
+/** What one stealth retry adds to the agent's price: the stealth browser's own. */
+export const AGENT_ESCALATION_CREDITS = 5;
+
+/**
+ * Most stealth retries one run may make. A second, independent bound on top
+ * of maxSteps/maxUrls/wall-clock: each retry is a browser launch (seconds,
+ * and hundreds of MB) and 5 credits, and the projection a caller is checked
+ * against has to be a finite ceiling.
+ */
+export const AGENT_MAX_ESCALATIONS = 2;
+
+/**
+ * The most escalation can add to one call — the ceiling getToolCost projects.
+ * Reads the RAW params (getToolCost runs before validation). The pro model
+ * delegates to ResearchOrchestrator and never runs this stage, so it adds 0.
+ *
+ * @param {object|undefined} params
+ * @returns {number}
+ */
+export function agentEscalationSurcharge(params) {
+  if (params?.model === 'pro') return 0;
+  const maxUrls = Number.isFinite(params?.maxUrls) && params.maxUrls >= 1 ? Math.floor(params.maxUrls) : AGENT_MAX_ESCALATIONS;
+  return AGENT_ESCALATION_CREDITS * Math.min(AGENT_MAX_ESCALATIONS, maxUrls);
+}