crawlforge-mcp-server 6.9.0 → 6.10.0

package/CLAUDE.md

@@ -236,6 +236,10 @@ CRAWLFORGE_STEALTH_PROXIES=          # comma-separated proxy URLs used by the es
                                      # from the PROXY_ROTATION_* family, which belongs to
                                      # localization. The agent's automatic stealth retry
                                      # (stealth review Phase 3) reads it too.
+CRAWLFORGE_CLEARANCE_JAR=            # `off` disables the clearance jar (stealth review Phase 4):
+                                     # cf_clearance/__cf_bm/datadome cookies a stealth render earned
+                                     # are replayed to the next stealth context with the same engine,
+                                     # UA and proxy. Persisted at ~/.crawlforge/stealth-clearance.json.
 ```
 
 Camoufox is an **optional** dependency and its transitive `language-tags@2.1.0` requires Node ≥22. npm silently drops an optional subtree that fails an engine check, so on Node 20 it is simply absent — `'auto'` then resolves to Chromium and says so in `fallbackWarning`. `engines.node` stays at `>=20.16.0` deliberately; bumping it is breaking and is a release decision.

package/README.md

@@ -273,6 +273,9 @@ export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research
 # Comma-separated; a proxy passed on a call always wins. CrawlForge supplies none.
 export CRAWLFORGE_STEALTH_PROXIES="http://user:pass@gw.provider.net:8080"
 # Unrelated to the PROXY_ROTATION_* variables, which belong to the localization tool.
+
+# Optional: turn off the clearance jar (on by default) — see "Stealth engines and proxies"
+export CRAWLFORGE_CLEARANCE_JAR="off"
 ```
 
 ### MCP Spec Features
@@ -313,6 +316,8 @@ On `browser_session` and `scrape_with_actions` the engine applies only with `ste
 
 **CrawlForge supplies no proxies**, and a datacenter proxy does not fix a block: Cloudflare scores the IP's ASN and the TLS/HTTP2 handshake before any JavaScript runs, so no browser-side patch compensates for a datacenter address. Bring your own residential exit with `CRAWLFORGE_STEALTH_PROXIES` (comma-separated URLs), which the escalation stage, `stealth_mode`, `browser_session`, `scrape_with_actions`, the `deep_research` retry and the `agent` tool's automatic stealth retry use when the caller passes none; a proxy passed on the call always wins. With a proxy, Camoufox derives its timezone, locale and geolocation from the exit IP.
 
+**A challenge solved once is not solved again.** When a stealth render gets past a Cloudflare or DataDome wall, the vendor's clearance cookies (`cf_clearance`, `__cf_bm`, `datadome` — nothing else) are kept in `~/.crawlforge/stealth-clearance.json` (mode 0600) and replayed to the next stealth context with the same engine, User-Agent and proxy, for at most the cookie's own lifetime and never more than 24 hours. A render that still meets the wall drops them. Set `CRAWLFORGE_CLEARANCE_JAR=off` to disable.
+
 Full detail, measurements and sources: [docs/stealth-engines.md](docs/stealth-engines.md).
 
 ### Stealth extraction for `deep_research` (Camoufox)

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "6.9.0",
+  "version": "6.10.0",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 31 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",

package/server.js

@@ -108,7 +108,7 @@ if (configErrors.length > 0 && config.server.nodeEnv === 'production') {
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "6.9.0",
+  version: "6.10.0",
   description: "Production-ready MCP server with 31 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, stateful browser sessions with element refs, deep research, structured extraction, embedded JavaScript state extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",

package/src/core/ClearanceJar.js

@@ -0,0 +1,229 @@
+/**
+ * ClearanceJar — a solved bot-wall challenge is not solved again for its lifetime.
+ *
+ * Stealth review Phase 4 (docs/STEALTH_REVIEW_2026-09.md). Every stealth
+ * context used to start cold, so a Cloudflare or DataDome challenge the browser
+ * had just passed was served again on the next call to the same site. The
+ * vendors hand out a clearance cookie when a challenge passes; this keeps those
+ * cookies and gives them back to the next stealth context that would present
+ * the same identity.
+ *
+ * Three rules keep it honest:
+ *
+ * - **Only clearance cookies.** The allow-list below is the vendors' own
+ *   clearance and bot-management cookies. Nothing a site sets for a login, a
+ *   cart or a preference is ever kept, so a browser_session one caller logged
+ *   into cannot leak its session into another caller's context — on the hosted
+ *   instance those are different customers.
+ * - **Keyed on who solved it.** cf_clearance is bound to the IP and the User
+ *   Agent that earned it, and a TLS stack that is not the one that earned it is
+ *   scored too. The key is the engine, the exact User-Agent and the proxy
+ *   identity (server + username, never the password), so a clearance earned
+ *   through one exit is never presented from another. A rotating proxy behind
+ *   one fixed username can still change exit underneath a key; the vendor then
+ *   challenges again, the verdict reports a block, and the host is discarded.
+ * - **Bounded.** Expiry is the cookie's own, capped at MAX_TTL_MS; the jar
+ *   holds at most MAX_KEYS identities and MAX_COOKIES_PER_KEY cookies each, and
+ *   the file is rewritten only when its contents change.
+ *
+ * Persisted at ~/.crawlforge/stealth-clearance.json (mode 0600) so an MCP
+ * server that a client restarts per session keeps its clearances.
+ * CRAWLFORGE_CLEARANCE_JAR=off disables the jar entirely.
+ */
+
+import crypto from 'crypto';
+import fs from 'fs';
+import os from 'os';
+import path from 'path';
+
+/**
+ * Cloudflare's clearance (cf_clearance) and bot-management (__cf_bm) cookies,
+ * and DataDome's (datadome). Exact names: a prefix match would sweep in
+ * whatever else a site happens to name alike.
+ */
+export const CLEARANCE_COOKIE_NAMES = new Set(['cf_clearance', '__cf_bm', 'datadome']);
+
+/** No clearance outlives this, whatever expiry the vendor set. */
+export const MAX_TTL_MS = 24 * 60 * 60 * 1000;
+/** A session cookie (no expiry) is kept this long. */
+export const SESSION_TTL_MS = 30 * 60 * 1000;
+export const MAX_KEYS = 32;
+export const MAX_COOKIES_PER_KEY = 200;
+
+const DEFAULT_FILE = path.join(os.homedir(), '.crawlforge', 'stealth-clearance.json');
+
+/** Whether a cookie domain (".example.com" or "example.com") covers `host`. */
+function domainCovers(domain, host) {
+  const d = String(domain || '').replace(/^\./, '').toLowerCase();
+  const h = String(host || '').toLowerCase();
+  return Boolean(d) && (h === d || h.endsWith(`.${d}`));
+}
+
+export class ClearanceJar {
+  /**
+   * @param {{ file?: string|null, now?: () => number }} [options]
+   *   file: where the jar persists; null keeps it in memory only.
+   */
+  constructor({ file = DEFAULT_FILE, now = Date.now } = {}) {
+    this.file = file;
+    this.now = now;
+    /** @type {Map<string, Map<string, object>>} key → (domain|path|name → cookie) */
+    this.entries = null;
+  }
+
+  /**
+   * The identity a clearance is bound to, or null when it cannot be named — an
+   * unknown User-Agent cannot be matched, so nothing is stored or reused for it.
+   *
+   * @param {{ engine?: string|null, userAgent?: string|null, proxy?: {server?: string, username?: string}|null }} identity
+   * @returns {string|null}
+   */
+  static keyFor({ engine, userAgent, proxy } = {}) {
+    if (!engine || !userAgent) return null;
+    const exit = proxy?.server ? `${proxy.server}|${proxy.username || ''}` : 'direct';
+    // Hashed so the file names no proxy host, account or persona in clear.
+    return crypto.createHash('sha256').update(`${engine}\n${userAgent}\n${exit}`).digest('hex');
+  }
+
+  /** Clearance cookies stored for `key` that have not expired, Playwright-shaped. */
+  cookiesFor(key) {
+    if (!key) return [];
+    const bucket = this._load().get(key);
+    if (!bucket) return [];
+    const nowS = this.now() / 1000;
+    const live = [...bucket.values()].filter((c) => c.expires > nowS);
+    if (live.length !== bucket.size) {
+      this._replace(key, live);
+      this._save();
+    }
+    return live;
+  }
+
+  /**
+   * Keep the clearance cookies from `cookies` (the output of
+   * BrowserContext.cookies()) under `key`, replacing older copies.
+   * @returns {number} how many clearance cookies were kept
+   */
+  store(key, cookies = []) {
+    if (!key) return 0;
+    const nowMs = this.now();
+    const kept = [];
+    for (const c of cookies) {
+      if (!c || !CLEARANCE_COOKIE_NAMES.has(c.name) || !c.value || !c.domain) continue;
+      const vendorExpiryMs = c.expires > 0 ? c.expires * 1000 : nowMs + SESSION_TTL_MS;
+      const expiresMs = Math.min(vendorExpiryMs, nowMs + MAX_TTL_MS);
+      if (expiresMs <= nowMs) continue;
+      kept.push({
+        name: c.name,
+        value: c.value,
+        domain: c.domain,
+        path: c.path || '/',
+        expires: Math.floor(expiresMs / 1000),
+        httpOnly: Boolean(c.httpOnly),
+        secure: Boolean(c.secure),
+        sameSite: c.sameSite || 'Lax'
+      });
+    }
+    if (!kept.length) return 0;
+
+    const entries = this._load();
+    const bucket = entries.get(key) || new Map();
+    let changed = false;
+    for (const c of kept) {
+      const id = `${c.domain}|${c.path}|${c.name}`;
+      const old = bucket.get(id);
+      if (old && old.value === c.value && old.expires === c.expires) continue;
+      bucket.delete(id); // re-insert at the end: insertion order is recency
+      bucket.set(id, c);
+      changed = true;
+    }
+    if (!changed) return kept.length;
+    while (bucket.size > MAX_COOKIES_PER_KEY) bucket.delete(bucket.keys().next().value);
+    entries.delete(key);
+    entries.set(key, bucket);
+    while (entries.size > MAX_KEYS) entries.delete(entries.keys().next().value);
+    this._save();
+    return kept.length;
+  }
+
+  /**
+   * Drop every clearance for `host` under `key` — called when a stealth render
+   * still met the wall, so a clearance that no longer works is not replayed.
+   * @returns {number} how many cookies were dropped
+   */
+  discard(key, host) {
+    if (!key || !host) return 0;
+    const bucket = this._load().get(key);
+    if (!bucket) return 0;
+    let dropped = 0;
+    for (const [id, c] of bucket) {
+      if (domainCovers(c.domain, host)) {
+        bucket.delete(id);
+        dropped++;
+      }
+    }
+    if (dropped) {
+      if (!bucket.size) this.entries.delete(key);
+      this._save();
+    }
+    return dropped;
+  }
+
+  _replace(key, cookies) {
+    const entries = this._load();
+    if (!cookies.length) {
+      entries.delete(key);
+      return;
+    }
+    entries.set(key, new Map(cookies.map((c) => [`${c.domain}|${c.path}|${c.name}`, c])));
+  }
+
+  _load() {
+    if (this.entries) return this.entries;
+    this.entries = new Map();
+    if (!this.file) return this.entries;
+    let raw;
+    try {
+      raw = JSON.parse(fs.readFileSync(this.file, 'utf8'));
+    } catch {
+      return this.entries; // absent or unreadable: start empty
+    }
+    const nowS = this.now() / 1000;
+    for (const [key, cookies] of Object.entries(raw?.entries || {})) {
+      if (!Array.isArray(cookies)) continue;
+      const live = cookies.filter((c) => c && CLEARANCE_COOKIE_NAMES.has(c.name) && c.expires > nowS);
+      if (live.length) this._replace(key, live.slice(-MAX_COOKIES_PER_KEY));
+    }
+    while (this.entries.size > MAX_KEYS) this.entries.delete(this.entries.keys().next().value);
+    return this.entries;
+  }
+
+  _save() {
+    if (!this.file) return;
+    const out = { version: 1, entries: {} };
+    for (const [key, bucket] of this.entries) out.entries[key] = [...bucket.values()];
+    try {
+      fs.mkdirSync(path.dirname(this.file), { recursive: true, mode: 0o700 });
+      const tmp = `${this.file}.${process.pid}.tmp`;
+      fs.writeFileSync(tmp, JSON.stringify(out), { mode: 0o600 });
+      fs.renameSync(tmp, this.file);
+    } catch {
+      // Persistence is an optimisation: an unwritable home directory costs the
+      // next process a challenge, not this call its result.
+    }
+  }
+}
+
+let shared;
+
+/**
+ * The process-wide jar, shared by every StealthBrowserManager (server.js,
+ * BrowserProcessor and ResearchOrchestrator each construct their own), or null
+ * when CRAWLFORGE_CLEARANCE_JAR=off.
+ * @returns {ClearanceJar|null}
+ */
+export function sharedClearanceJar() {
+  if (String(process.env.CRAWLFORGE_CLEARANCE_JAR || '').toLowerCase() === 'off') return null;
+  shared ??= new ClearanceJar();
+  return shared;
+}

package/src/core/StealthBrowserManager.js

@@ -18,9 +18,10 @@ import { createRequire } from 'module';
 import HumanBehaviorSimulator from '../utils/HumanBehaviorSimulator.js';
 import { BrowserContextPool } from './BrowserContextPool.js';
 import { safeGoto } from '../utils/ssrfGuard.js';
-import { looksLikeInterstitial } from '../utils/challengeDetection.js';
+import { looksLikeInterstitial, detectChallengePage } from '../utils/challengeDetection.js';
 import { guardFirefoxPageErrors } from '../utils/firefoxPageErrorGuard.js';
 import { serverStealthProxies } from '../constants/config.js';
+import { ClearanceJar, sharedClearanceJar } from './ClearanceJar.js';
 
 // Grace given to a document that rendered no title and no text (see _waitOutEmptyDocument).
 export const EMPTY_DOCUMENT_GRACE_MS = 8000;
@@ -39,6 +40,15 @@ const CAMOUFOX_PROXY = Symbol('crawlforge.camoufoxProxy');
 // browser for the same reason as the proxy: camoufox fixes it at launch.
 const CAMOUFOX_LOCALE = Symbol('crawlforge.camoufoxLocale');
 
+// The User-Agent that camoufox browser presents — the persona pinned at launch,
+// null when camoufox drew its own and we therefore do not know it. The
+// clearance jar keys on it (see ClearanceJar), and camoufox fixes it at launch.
+const CAMOUFOX_USER_AGENT = Symbol('crawlforge.camoufoxUserAgent');
+
+// How long closeContext waits to read a context's cookies before giving up on
+// keeping its clearances: a wedged browser hangs rather than errors.
+const CLEARANCE_HARVEST_TIMEOUT_MS = 2000;
+
 const StealthConfigSchema = z.object({
   level: z.enum(['basic', 'medium', 'advanced']).default('medium'),
   randomizeFingerprint: z.boolean().default(true),
@@ -175,6 +185,12 @@ export class StealthBrowserManager {
     // Chromium that nobody reports reads as a camoufox result that failed.
     this._engineFallbackWarning = null;
     this._maxContexts = parseInt(process.env.MAX_BROWSER_CONTEXTS || '10', 10);
+    // Stealth review Phase 4: clearance cookies a solved challenge earned are
+    // replayed to the next context with the same identity. One process-wide jar
+    // by default (every manager instance shares it); null turns it off — the
+    // benchmark does, since a replayed clearance would measure the jar instead
+    // of the engine.
+    this.clearanceJar = options.clearanceJar !== undefined ? options.clearanceJar : sharedClearanceJar();
     this.contexts = this._createContextPool();
     // D2.2: fingerprints Map is capped at _maxContexts to prevent unbounded growth.
     // Oldest entries are evicted when the cap is exceeded (insertion order via Map).
@@ -525,6 +541,7 @@ export class StealthBrowserManager {
       });
       browser[CAMOUFOX_PROXY] = proxy;
       browser[CAMOUFOX_LOCALE] = locale;
+      browser[CAMOUFOX_USER_AGENT] = adapter.lastUserAgent ?? null;
       this.browser = browser;
       this._launchedEngine = 'camoufox';
       return this.browser;
@@ -727,6 +744,19 @@ export class StealthBrowserManager {
       contextOptions.proxy = proxy;
     }
 
+    // The identity a clearance is bound to: engine, the User-Agent this context
+    // presents, and the exit it goes out through. Read before the camoufox
+    // block below takes the User-Agent off the context options.
+    const jarKey = this.clearanceJar
+      ? ClearanceJar.keyFor({
+          engine: this._launchedEngine,
+          userAgent: this._launchedEngine === 'camoufox'
+            ? this.browser[CAMOUFOX_USER_AGENT]
+            : contextOptions.userAgent,
+          proxy
+        })
+      : null;
+
     if (this._launchedEngine === 'camoufox') {
       // camoufox's Firefox build predates the Browser.setDefaultViewport fields
       // playwright-core 1.62 sends (screenSize, isMobile, ...) and rejects
@@ -787,11 +817,19 @@ export class StealthBrowserManager {
 
     const context = await this.browser.newContext(contextOptions);
     const contextId = this.generateContextId();
-    
+
+    // Replay clearances this identity already earned. addCookies scopes each
+    // to its own domain, so a context bound for another site sends none of
+    // them. Best effort: a context without them just meets the challenge.
+    const clearances = jarKey ? this.clearanceJar.cookiesFor(jarKey) : [];
+    if (clearances.length) {
+      await context.addCookies(clearances).catch(() => {});
+    }
+
     // Apply stealth scripts and configurations
     await this.applyAdvancedStealthConfigurations(context, validatedConfig, fingerprint);
     
-    await this.contexts.set(contextId, { context, fingerprint, config: validatedConfig });
+    await this.contexts.set(contextId, { context, fingerprint, config: validatedConfig, jarKey });
     // D2.2: enforce LRU cap on fingerprints Map
     this._setFingerprint(contextId, fingerprint);
 
@@ -2770,6 +2808,13 @@ export class StealthBrowserManager {
         ? await page.screenshot({ encoding: 'base64', fullPage: false }).catch(() => null)
         : null;
 
+      // Still a wall after everything above: whatever clearance this context
+      // carried for the site did not work, so closeContext drops it instead of
+      // keeping it for the next call.
+      if (detectChallengePage({ title, html, text })) {
+        this._markBlocked(contextId, [url, page.url()]);
+      }
+
       return { success: true, url, title, text, html, screenshot: shot, status, gracedMs, engine: engineUsed, warnings };
     } finally {
       await this.closeContext(contextId).catch(() => {});
@@ -3050,11 +3095,54 @@ export class StealthBrowserManager {
    */
   async closeContext(contextId) {
     if (this.contexts.has(contextId)) {
+      await this._harvestClearances(contextId);
       await this.contexts.dispose(contextId);
       this.fingerprints.delete(contextId);
     }
   }
 
+  /**
+   * Record that a context's render still met a bot wall on these URLs' hosts.
+   * @param {string} contextId
+   * @param {string[]} urls
+   */
+  _markBlocked(contextId, urls) {
+    const contextData = this.contexts.get(contextId);
+    if (!contextData) return;
+    contextData.blockedHosts ??= new Set();
+    for (const url of urls) {
+      try {
+        contextData.blockedHosts.add(new URL(url).hostname);
+      } catch { /* about:blank or a malformed URL names no host */ }
+    }
+  }
+
+  /**
+   * Keep the clearance cookies a context earned, and drop the ones for any host
+   * it was still blocked on. Runs before the context is closed, and never
+   * throws or hangs the close it precedes.
+   * @param {string} contextId
+   */
+  async _harvestClearances(contextId) {
+    const contextData = this.contexts.get(contextId);
+    if (!this.clearanceJar || !contextData?.jarKey) return;
+    let timer;
+    try {
+      const cookies = await Promise.race([
+        contextData.context.cookies(),
+        new Promise((resolve) => { timer = setTimeout(() => resolve([]), CLEARANCE_HARVEST_TIMEOUT_MS); })
+      ]);
+      this.clearanceJar.store(contextData.jarKey, cookies);
+      for (const host of contextData.blockedHosts ?? []) {
+        this.clearanceJar.discard(contextData.jarKey, host);
+      }
+    } catch {
+      // A closed or crashed context has nothing to keep.
+    } finally {
+      clearTimeout(timer);
+    }
+  }
+
   /**
    * D2.2: LRU-capped fingerprint setter.
    * Evicts the oldest entry when the Map exceeds _maxContexts to prevent unbounded growth.
@@ -3451,6 +3539,9 @@ export class CamoufoxAdapter extends BrowserEngine {
     // reaches only `rv:` and about half of all launches announce a Firefox
     // that is not the one running — see _pinnedFingerprint.
     const fingerprint = this._pinnedFingerprint(camoufox);
+    // The UA this launch will present, for the clearance jar's key. Only known
+    // when we pinned the persona; camoufox's own draw is not visible here.
+    this.lastUserAgent = fingerprint?.navigator?.userAgent || null;
     if (fingerprint) {
       options.fingerprint = fingerprint;
       // Switches off exactly two things: camoufox's "you passed your own