crawlforge-mcp-server 4.6.5 → 4.7.0

package/README.md

@@ -67,9 +67,9 @@
 npm install -g crawlforge-mcp-server
 ```
 
-### 2. Setup Your API Key (optional for the free local tools)
+### 2. Setup Your API Key (required)
 
-The 15 free local tools work immediately with **no API key at all** — skip straight to step 3 if that's all you need. To unlock the metered premium tools (`search_web`, `crawl_deep`, `stealth_mode`, `agent`, …):
+Every tool requires a CrawlForge API key — new accounts get 1,000 free trial credits to start:
 
 ```bash
 npx crawlforge-setup
@@ -150,43 +150,38 @@ Restart Cursor to activate.
 
 ## 📊 Available Tools
 
-CrawlForge is **open-core**: 15 tools run locally on your machine and are **completely free — no API key required**. The metered premium tools cover real infrastructure (search fees, proxies, browser farms) and need an API key.
-
-**Free Local Tools** (0 credits, no API key needed)
-
-| Tool | What it does |
-|------|--------------|
-| `fetch_url` | Fetch content from any URL |
-| `extract_text` | Extract clean text from web pages |
-| `extract_links` | Get all links from a page |
-| `extract_metadata` | Extract page metadata (title, OG tags, schema.org) |
-| `scrape` | **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings. *The `screenshot` format is the one metered exception (2 credits — needs a server browser)* |
-| `scrape_structured` | Extract structured data with CSS selectors |
-| `scrape_template` | Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors |
-| `extract_content` | Enhanced content extraction |
-| `summarize_content` | Generate intelligent summaries |
-| `analyze_content` | Comprehensive content analysis |
-| `extract_structured` | LLM-powered schema-driven extraction (your own LLM key or local Ollama) |
-| `extract_with_llm` | Natural-language extraction. **Defaults to a local Ollama model — no API key, no API costs.** Pass `provider: "openai" \| "anthropic"` with the matching key for cloud models |
-| `process_document` | Multi-format document processing |
-| `list_ollama_models` | List the Ollama models installed locally (helps you pick a `model` for `extract_with_llm`) |
-| `get_batch_results` | Retrieve paginated results for a `batch_scrape` job by `batchId` |
-
-**Metered Premium Tools** (3–10 credits, API key required)
+CrawlForge requires a CrawlForge API key — **every tool is metered and consumes credits**. New accounts get **1,000 free trial credits** to start. Get a key at [crawlforge.dev/signup](https://www.crawlforge.dev/signup).
+
+**All Tools** (API key required)
 
 | Tool | Credits | What it does |
 |------|---------|--------------|
-| `map_site` | 3 | Discover and map website structure (optional `search=` ranks the discovered URLs) |
+| `fetch_url` | 1 | Fetch content from any URL |
+| `extract_text` | 1 | Extract clean text from web pages |
+| `extract_links` | 1 | Get all links from a page |
+| `extract_metadata` | 1 | Extract page metadata (title, OG tags, schema.org) |
+| `scrape_template` | 1 | Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors |
+| `list_ollama_models` | 1 | List the Ollama models installed locally (helps you pick a `model` for `extract_with_llm`) |
+| `get_batch_results` | 1 | Retrieve paginated results for a `batch_scrape` job by `batchId` |
+| `scrape` | 2 | **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings |
+| `scrape_structured` | 2 | Extract structured data with CSS selectors |
+| `extract_content` | 2 | Enhanced content extraction |
+| `map_site` | 2 | Discover and map website structure (optional `search=` ranks the discovered URLs) |
+| `process_document` | 2 | Multi-format document processing |
+| `localization` | 2 | Multi-language and geo-location management |
 | `track_changes` | 3 | Monitor content changes over time |
+| `analyze_content` | 3 | Comprehensive content analysis |
+| `extract_structured` | 3 | LLM-powered schema-driven extraction (your own LLM key or local Ollama) |
+| `extract_with_llm` | 3 | Natural-language extraction. Defaults to a local Ollama model; pass `provider: "openai" \| "anthropic"` with the matching key for cloud models (external LLM billed by your provider) |
+| `summarize_content` | 4 | Generate intelligent summaries |
+| `crawl_deep` | 4 | Deep crawl entire websites |
 | `search_web` | 5 | Search the web using Google Search API |
-| `crawl_deep` | 5 | Deep crawl entire websites |
 | `batch_scrape` | 5 | Process multiple URLs simultaneously |
 | `scrape_with_actions` | 5 | Browser automation chains |
 | `generate_llms_txt` | 5 | Generate AI interaction guidelines |
-| `localization` | 5 | Multi-language and geo-location management |
+| `stealth_mode` | 5 | Anti-detection browser management |
 | `agent` | 8 | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) |
 | `deep_research` | 10 | Multi-stage research with source verification |
-| `stealth_mode` | 10 | Anti-detection browser management |
 
 For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see [SKILL.md](SKILL.md).
 
@@ -194,7 +189,7 @@ For the full canonical capabilities reference (all tools, CLI commands, stealth
 
 ## 💳 Pricing
 
-**15 local tools are free forever — no API key, no credit card.** Credits only meter the premium tools that run on CrawlForge infrastructure.
+**Every tool is metered and requires an API key.** New accounts get 1,000 free trial credits — no credit card required to start.
 
 | Plan | Credits/Month | Best For |
 |------|---------------|----------|
@@ -229,11 +224,16 @@ export OLLAMA_DEFAULT_MODEL="llama3.2"             # default; any locally-pulled
 # Optional: Cloud LLM keys — only needed when you pass provider: "openai" or "anthropic"
 export OPENAI_API_KEY="sk-..."
 export ANTHROPIC_API_KEY="sk-ant-..."
+
+# Optional: deep_research stealth extraction fallback (v4.6.6) — see below
+export RESEARCH_STEALTH_ENGINE="auto"      # auto (default) | camoufox | chromium
+export RESEARCH_STEALTH_FALLBACK="true"    # set to "false" to disable entirely
+export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research run
 ```
 
 ### Local-LLM quickstart (`extract_with_llm` with Ollama)
 
-`extract_with_llm` defaults to a local Ollama model — no API key, no API costs, no data leaving your machine.
+`extract_with_llm` defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).
 
 ```bash
 # 1. Install Ollama:  https://ollama.com
@@ -247,6 +247,31 @@ ollama pull llama3.2
 #    extract_with_llm({ url: "https://example.com", prompt: "…", model: "llama3.2" })
 ```
 
+### Stealth extraction for `deep_research` (Camoufox)
+
+`deep_research` automatically retries sources that block the normal fetch path (Reddit, Quora, forums, and Cloudflare/DataDome-protected pages return HTTP 403) through a **real fingerprinted browser**, then re-extracts from the rendered HTML. It's bounded (`RESEARCH_MAX_STEALTH_RETRIES`, default 8, plus a per-page timeout) and lazy — the browser stack only loads when a source is actually blocked.
+
+Engine selection (`RESEARCH_STEALTH_ENGINE`):
+
+- **`auto`** (default) — prefer **Camoufox** (Firefox anti-detect), fall back to Chromium stealth, then plain fetch.
+- **`camoufox`** — force Camoufox.
+- **`chromium`** — force the Chromium stealth engine.
+
+Headless Chromium **cannot** clear modern challenges (Cloudflare Turnstile, DataDome) — **Camoufox can**. In testing it recovered Quora and Trustpilot pages that were otherwise fully blocked. To enable it, install the optional dependency and run its one-time binary fetch:
+
+```bash
+# Camoufox is declared as an optional dependency, so a normal install already pulls it.
+# If you installed with --no-optional, add it explicitly:
+npm install camoufox
+
+# One-time download of the Camoufox Firefox binary (~130 MB):
+npx camoufox fetch
+```
+
+Without the Camoufox binary, `deep_research` silently falls back to Chromium stealth and then to plain fetch — no errors, just lower recovery on heavily-protected sites. Disable the whole fallback with `RESEARCH_STEALTH_FALLBACK=false`.
+
+> **Note:** Hard IP-reputation blocks (e.g. Reddit's edge `403`) resist headless stealth from any IP and require residential/mobile proxies, which CrawlForge does not provide. See [docs/stealth-engines.md](docs/stealth-engines.md) for details.
+
 ### Manual Configuration
 
 Your configuration is stored at `~/.crawlforge/config.json`:
@@ -287,7 +312,7 @@ Once configured, use these tools in your AI assistant:
 - **Action allowlist**: `scrape_with_actions` accepts only 7 action types (`wait`, `click`, `type`, `press`, `scroll`, `screenshot`, `executeJavaScript`). No download, file-write, or arbitrary cross-page navigation primitives exist.
 - **JavaScript gate**: The `executeJavaScript` action throws by default. Set `ALLOW_JAVASCRIPT_EXECUTION=true` at deploy time to enable (not recommended in production).
 - **MCP Elicitation** (v3.6.0): Four tools request user confirmation before executing expensive operations — `deep_research` (>50 URLs), `batch_scrape` (sync mode, >25 URLs), `crawl_deep` (projected >500 pages), `extract_structured` (schema has >3 required fields with no LLM configured). Credit-low situations also elicit. Confirmation is best-effort: if the MCP client does not support elicitation the tool proceeds (fail-open).
-- **Per-tool credit gating**: Every tool is wrapped with `withAuth()`; metered tools check and deduct credits before execution (fail-closed since v3.0.18). Free local tools (cost 0) skip the credit path entirely.
+- **Per-tool credit gating**: Every tool is wrapped with `withAuth()` and is metered — credits are checked and deducted before execution, and a valid API key is required for every tool (fail-closed since v3.0.18).
 
 See [docs/sandboxing-and-approvals.md](docs/sandboxing-and-approvals.md) for the full reference.
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "4.6.5",
+  "version": "4.7.0",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 26 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
@@ -131,6 +131,9 @@
     "winston": "^3.11.0",
     "zod": "^3.23.8"
   },
+  "optionalDependencies": {
+    "camoufox": "^0.1.19"
+  },
   "devDependencies": {
     "@jest/globals": "^30.3.0",
     "c8": "^11.0.0",

package/server.js

@@ -68,15 +68,14 @@ if (!AuthManager.isAuthenticated() && !AuthManager.isCreatorMode()) {
       process.exit(1);
     }
   } else {
-    // Open-core Phase 2: no API key is fine — start in free-tier mode.
-    // Tier-0 tools (cost 0) run locally without a key; Tier-1 metered tools
-    // return a "not configured" error until a key is set.
+    // Every tool is metered and requires an API key — there is no free tier.
+    // The server still starts so the MCP client can list tools, but every
+    // tool call errors with "not configured" until a key is set.
     // Status → stderr; stdout is reserved for the MCP JSON-RPC stream.
-    console.error('ℹ️  CrawlForge running in free-tier mode (no API key configured).');
-    console.error('   Free local tools work out of the box. Premium tools (search_web,');
-    console.error('   crawl_deep, stealth_mode, agent, deep_research, …) need an API key:');
-    console.error('   get one at https://www.crawlforge.dev/signup, then run `npm run setup`');
-    console.error('   or set CRAWLFORGE_API_KEY.');
+    console.error('⚠️  No CrawlForge API key configured — all tools require a key.');
+    console.error('   Every tool (fetch_url, search_web, deep_research, …) is metered.');
+    console.error('   Get a key at https://www.crawlforge.dev/signup, then run `npm run setup`');
+    console.error('   or set CRAWLFORGE_API_KEY. Tool calls will error until a key is set.');
   }
 }
 
@@ -90,7 +89,7 @@ if (configErrors.length > 0 && config.server.nodeEnv === 'production') {
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "4.6.5",
+  version: "4.7.0",
   description: "Production-ready MCP server with 26 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png"

package/src/core/AuthManager.js

@@ -239,11 +239,6 @@ class AuthManager {
       return true;
     }
 
-    // Open-core Phase 2: Tier-0 tools cost 0 and run without an API key
-    if (estimatedCredits === 0) {
-      return true;
-    }
-
     if (!this.config) {
       throw new Error('CrawlForge not configured. Run setup first.');
     }
@@ -507,51 +502,53 @@ class AuthManager {
   /**
    * Get credit cost for a tool.
    *
-   * Open-core Phase 1 (docs/tier-map.md): this table is the single source of
-   * truth shared with the backend (crawlforge-website/src/lib/credits.ts).
-   * Tier 0 tools run locally on the user's machine and cost 0; Tier 1 tools
-   * are metered per COGS.
+   * Every tool is metered and requires an API key — there is no free tier.
+   * This table is the single source of truth shared with the backend
+   * (crawlforge-website/src/lib/credits.ts TOOL_CREDIT_COSTS).
    *
    * @param {string} tool
-   * @param {object} [params] — invocation params; only used for per-call
-   *        exceptions (scrape's screenshot format needs a server browser).
    */
-  getToolCost(tool, params) {
-    // Tier-0 exception: the screenshot format of `scrape` is browser-backed
-    if (tool === 'scrape' && Array.isArray(params?.formats) && params.formats.includes('screenshot')) {
-      return 2;
-    }
-
+  getToolCost(tool) {
     const costs = {
-      // Tier 0 — free, local (key optional)
-      fetch_url: 0,
-      extract_text: 0,
-      extract_links: 0,
-      extract_metadata: 0,
-      scrape_structured: 0,
-      scrape_template: 0,
-      extract_content: 0,
-      scrape: 0, // 2 if formats includes 'screenshot' (handled above)
-      summarize_content: 0,
-      analyze_content: 0,
-      extract_with_llm: 0,
-      extract_structured: 0,
-      process_document: 0,
-      list_ollama_models: 0,
-      get_batch_results: 0, // retrieval of an already-paid batch job
-
-      // Tier 1 — metered (costs reflect COGS)
-      map_site: 3,
+      // 1 credit
+      fetch_url: 1,
+      extract_text: 1,
+      extract_links: 1,
+      extract_metadata: 1,
+      scrape_template: 1,
+      list_ollama_models: 1,
+      get_batch_results: 1, // retrieval of an already-paid batch job
+
+      // 2 credits
+      scrape_structured: 2,
+      extract_content: 2,
+      map_site: 2,
+      process_document: 2,
+      localization: 2,
+      scrape: 2,
+
+      // 3 credits
       track_changes: 3,
-      generate_llms_txt: 5,
-      search_web: 5,
-      crawl_deep: 5,
-      batch_scrape: 5,
+      analyze_content: 3,
+      extract_structured: 3,
+      extract_with_llm: 3,
+
+      // 4 credits
+      summarize_content: 4,
+      crawl_deep: 4,
+
+      // 5 credits
+      stealth_mode: 5,
       scrape_with_actions: 5,
-      localization: 5,
+      batch_scrape: 5,
+      search_web: 5,
+      generate_llms_txt: 5,
+
+      // 8 credits
       agent: 8, // projectCost() scales with maxUrls
-      deep_research: 10,
-      stealth_mode: 10
+
+      // 10 credits
+      deep_research: 10
     };
 
     return costs[tool] ?? 1;
@@ -574,7 +571,7 @@ class AuthManager {
 
     // Override for tools whose cost scales with params
     let projected = base;
-    let note = base === 0 ? 'Free local tool — no credits charged.' : 'Fixed cost per invocation.';
+    let note = 'Fixed cost per invocation.';
 
     switch (toolName) {
       case 'batch_scrape': {
@@ -596,14 +593,11 @@ class AuthManager {
         break;
       }
       case 'extract_with_llm':
-        note = 'Free local tool. External LLM API call billed by your LLM provider, not in credits.';
+        note = 'External LLM API call billed by your LLM provider, separate from the credit cost.';
         break;
       case 'scrape': {
-        // Free local tool; only the browser-backed screenshot format is metered
         projected = base;
-        note = base > 0
-          ? 'screenshot format requires a server browser (2 credits). Other formats are free.'
-          : 'Free local tool — no credits charged. json format may incur external LLM cost.';
+        note = 'Fixed cost per invocation. json format may incur external LLM cost (billed by your provider).';
         break;
       }
       case 'agent': {
@@ -614,7 +608,7 @@ class AuthManager {
         break;
       }
       default:
-        note = base === 0 ? 'Free local tool — no credits charged.' : 'Fixed cost per invocation.';
+        note = 'Fixed cost per invocation.';
     }
 
     return { projected, note };

package/src/core/ResearchOrchestrator.js

@@ -35,6 +35,19 @@ export class ResearchOrchestrator extends EventEmitter {
       cacheEnabled = true,
       cacheTTL = 1800000, // 30 minutes
       researchApproach = 'broad',
+      // Stealth-browser fallback for sources that block the plain fetch/extract
+      // path (Reddit, Quora, forums → HTTP 403). On by default; bounded so it
+      // cannot blow the research time budget. Disable with
+      // RESEARCH_STEALTH_FALLBACK=false.
+      enableStealthFallback = process.env.RESEARCH_STEALTH_FALLBACK !== 'false',
+      maxStealthRetries = parseInt(process.env.RESEARCH_MAX_STEALTH_RETRIES || '8', 10),
+      // 'auto' (default) prefers Camoufox (Firefox anti-detect — beats
+      // Cloudflare/DataDome that headless Chromium can't) and falls back to
+      // Chromium stealth when Camoufox/its binary is unavailable. Force one
+      // with RESEARCH_STEALTH_ENGINE=camoufox|chromium.
+      stealthEngine = process.env.RESEARCH_STEALTH_ENGINE || 'auto',
+      stealthLevel = 'medium',
+      stealthTimeoutMs = 20000,
       searchConfig = {},
       crawlConfig = {},
       extractConfig = {},
@@ -49,6 +62,18 @@ export class ResearchOrchestrator extends EventEmitter {
     this.enableSourceVerification = enableSourceVerification;
     this.enableConflictDetection = enableConflictDetection;
 
+    // Stealth fallback config + lazy state (browser launched only on first block)
+    this.enableStealthFallback = enableStealthFallback;
+    this.maxStealthRetries = Math.max(0, maxStealthRetries);
+    this.stealthEngine = stealthEngine;
+    this.stealthLevel = stealthLevel;
+    this.stealthTimeoutMs = stealthTimeoutMs;
+    this._stealthManager = null;     // Chromium StealthBrowserManager (fallback engine)
+    this._stealthBrowser = null;     // Camoufox browser handle (preferred engine)
+    this._stealthEngineActive = null;
+    this._stealthInit = null;
+    this._stealthCount = 0;
+
     // Initialize tools
     this.searchTool = new SearchWebTool(searchConfig);
     this.crawlTool = new CrawlDeepTool(crawlConfig);
@@ -101,7 +126,9 @@ export class ResearchOrchestrator extends EventEmitter {
       llmAnalysisCalls: 0,
       semanticAnalysisTime: 0,
       queryExpansionTime: 0,
-      synthesisTime: 0
+      synthesisTime: 0,
+      stealthRetries: 0,
+      stealthRecovered: 0
     };
   }
 
@@ -203,6 +230,9 @@ export class ResearchOrchestrator extends EventEmitter {
     Object.keys(this.metrics).forEach(key => {
       this.metrics[key] = 0;
     });
+
+    // Reset per-run stealth-retry budget
+    this._stealthCount = 0;
   }
 
   /**
@@ -551,11 +581,38 @@ export class ResearchOrchestrator extends EventEmitter {
             }
 
             // Normalize content to string (extract_content returns {text: "..."}, fallback returns string)
-            const contentText = contentData && contentData.content
-              ? (typeof contentData.content === 'string'
-                  ? contentData.content
-                  : (contentData.content.text || ''))
+            const normalizeContent = (cd) => cd && cd.content
+              ? (typeof cd.content === 'string' ? cd.content : (cd.content.text || ''))
               : '';
+            let contentText = normalizeContent(contentData);
+
+            // Stealth fallback: high-value discussion sources (Reddit, Quora,
+            // forums) return HTTP 403 to the plain fetch/extract path. When the
+            // normal path produced no usable content, retry through a real
+            // fingerprinted browser and re-run extraction on the rendered HTML.
+            // Bounded by maxStealthRetries + a per-page timeout.
+            const blocked = !contentData || contentData.success === false || contentText.trim().length === 0;
+            if (blocked && this.enableStealthFallback && this._stealthCount < this.maxStealthRetries) {
+              this._stealthCount++;
+              this.metrics.stealthRetries++;
+              try {
+                const stealthHtml = await this._stealthFetchHtml(source.link);
+                if (stealthHtml) {
+                  contentData = await this.extractTool.execute({
+                    url: source.link,
+                    html: stealthHtml,
+                    options: { includeMetadata: true, includeStructuredData: true }
+                  });
+                  contentText = normalizeContent(contentData);
+                  if (contentData && contentData.success !== false && contentText.trim().length > 0) {
+                    this.metrics.stealthRecovered++;
+                    this.logActivity('stealth_recovery', { url: source.link });
+                  }
+                }
+              } catch (stealthError) {
+                this.logger.warn('Stealth fallback failed', { url: source.link, error: stealthError.message });
+              }
+            }
 
             // Only count and enhance sources that actually produced non-empty content.
             // Skip failed extractions and empty {text:""} results.
@@ -641,10 +698,134 @@ export class ResearchOrchestrator extends EventEmitter {
       }
     });
 
+    // Tear down the stealth browser as soon as the extraction stage is done —
+    // it is only needed here and would otherwise leak a Playwright handle.
+    await this._closeStealth();
+
     // Sort by relevance score (LLM or traditional)
     return detailedFindings.sort((a, b) => (b.relevanceScore || 0) - (a.relevanceScore || 0));
   }
 
+  /**
+   * Lazily launch the stealth browser once. The heavy browser stack is only
+   * loaded when a source actually blocks the plain path. Engine selection:
+   *   - 'camoufox'/'auto' → Camoufox (Firefox anti-detect). Loaded via the CJS
+   *     build (its ESM bundle has a broken dynamic-require). Beats Cloudflare/
+   *     DataDome challenges that patched headless Chromium can't pass.
+   *   - 'chromium', or any Camoufox failure under 'auto' → StealthBrowserManager.
+   */
+  async _getStealthBrowser() {
+    if (!this._stealthInit) {
+      this._stealthInit = (async () => {
+        if (this.stealthEngine === 'camoufox' || this.stealthEngine === 'auto') {
+          try {
+            const { createRequire } = await import('module');
+            const require = createRequire(import.meta.url);
+            const camoufox = require('camoufox'); // CJS build — ESM build is broken
+            await this._ensureCamoufoxLayout(camoufox);
+            this._stealthBrowser = await camoufox.Camoufox({ headless: true });
+            this._stealthEngineActive = 'camoufox';
+            this.logger.info('Stealth fallback using Camoufox (Firefox) engine');
+            return;
+          } catch (e) {
+            if (this.stealthEngine === 'camoufox') throw e; // explicit request → surface
+            this.logger.warn('Camoufox unavailable, falling back to Chromium stealth', { error: e.message });
+          }
+        }
+        const { StealthBrowserManager } = await import('./StealthBrowserManager.js');
+        this._stealthManager = new StealthBrowserManager();
+        await this._stealthManager.launchStealthBrowser({ level: this.stealthLevel });
+        this._stealthEngineActive = 'chromium';
+      })();
+    }
+    await this._stealthInit;
+  }
+
+  /**
+   * macOS packaging fix for camoufox-js: it expects properties.json in
+   * Camoufox.app/Contents/MacOS/, but the .app bundle ships it under
+   * Contents/Resources/. Bridge it so the launcher can boot. Best-effort.
+   */
+  async _ensureCamoufoxLayout(camoufox) {
+    if (process.platform !== 'darwin' || !camoufox?.INSTALL_DIR) return;
+    try {
+      const fs = await import('fs');
+      const path = await import('path');
+      const appDir = path.join(camoufox.INSTALL_DIR, 'Camoufox.app', 'Contents');
+      const target = path.join(appDir, 'MacOS', 'properties.json');
+      const source = path.join(appDir, 'Resources', 'properties.json');
+      if (!fs.existsSync(target) && fs.existsSync(source)) {
+        fs.copyFileSync(source, target);
+      }
+    } catch { /* best-effort; launch surfaces a real error if it matters */ }
+  }
+
+  /**
+   * Fetch a URL's fully-rendered HTML through the stealth browser. Returns the
+   * HTML string, or null if every attempt was blocked / empty.
+   *
+   * Cloudflare/DataDome challenges are probabilistic — the same URL may serve a
+   * challenge on one load and the real page on the next — so Camoufox retries a
+   * few times with a fresh page each attempt. Chromium can't clear these at all
+   * (proven), so it gets a single attempt to avoid burning the time budget.
+   */
+  async _stealthFetchHtml(url) {
+    await this._getStealthBrowser();
+    const attempts = this._stealthEngineActive === 'camoufox' ? 3 : 1;
+    for (let i = 0; i < attempts; i++) {
+      const html = await this._stealthFetchOnce(url);
+      if (html) return html;
+    }
+    return null;
+  }
+
+  /** One stealth navigation. Fresh page/context; judges blocked by rendered content. */
+  async _stealthFetchOnce(url) {
+    let page;
+    if (this._stealthEngineActive === 'camoufox') {
+      page = await this._stealthBrowser.newPage();
+    } else {
+      const { contextId } = await this._stealthManager.createStealthContext({ level: this.stealthLevel });
+      page = await this._stealthManager.createStealthPage(contextId);
+    }
+    try {
+      const resp = await page.goto(url, { waitUntil: 'domcontentloaded', timeout: this.stealthTimeoutMs });
+      // Do NOT bail on the initial HTTP status: anti-bot challenges (Cloudflare
+      // Turnstile) return 403 on the first response and only resolve to the
+      // real page after their JS runs. Let it settle, then judge by the
+      // *rendered* content instead.
+      await page.waitForLoadState('networkidle', { timeout: 8000 }).catch(() => {});
+      await page.waitForTimeout(2500).catch(() => {});
+      const html = await page.content();
+      const title = (await page.title().catch(() => '')) || '';
+      const bodyLen = await page.evaluate(() => document.body?.innerText?.trim().length || 0).catch(() => 0);
+
+      // Still a challenge/block page → treat as blocked.
+      const challengeTitle = /just a moment|checking your browser|attention required|verify you are human|access denied|^blocked$/i.test(title);
+      const status = resp ? resp.status() : 0;
+      if (challengeTitle) return null;
+      if (status >= 400 && bodyLen < 500) return null; // hard block (e.g. Reddit 403 shell)
+      if (bodyLen < 200) return null;                  // empty / interstitial
+      return html && html.length > 200 ? html : null;
+    } finally {
+      await page.close().catch(() => {});
+    }
+  }
+
+  /** Close the stealth browser and reset its lazy state (idempotent). */
+  async _closeStealth() {
+    try {
+      if (this._stealthBrowser) await this._stealthBrowser.close().catch(() => {});
+      if (this._stealthManager) await this._stealthManager.cleanup().catch(() => {});
+    } catch (e) {
+      this.logger.warn('Stealth browser cleanup failed', { error: e.message });
+    }
+    this._stealthBrowser = null;
+    this._stealthManager = null;
+    this._stealthEngineActive = null;
+    this._stealthInit = null;
+  }
+
   /**
    * Verify source credibility using multiple factors
    */
@@ -1484,7 +1665,10 @@ export class ResearchOrchestrator extends EventEmitter {
     try {
       // Stop any active research
       this.stopResearch();
-      
+
+      // Tear down the stealth browser if one was launched
+      await this._closeStealth();
+
       // Clear cache if available
       if (this.cache && typeof this.cache.clear === "function") {
         await this.cache.clear();
@@ -1522,9 +1706,11 @@ export class ResearchOrchestrator extends EventEmitter {
         llmAnalysisCalls: 0,
         semanticAnalysisTime: 0,
         queryExpansionTime: 0,
-        synthesisTime: 0
+        synthesisTime: 0,
+        stealthRetries: 0,
+        stealthRecovered: 0
       };
-      
+
     } catch (error) {
       // Silent cleanup - do not throw errors during cleanup
       console.warn("Warning during ResearchOrchestrator cleanup:", error.message);

package/src/server/withAuth.js

@@ -4,8 +4,8 @@
  * (OpenTelemetry spans + Prometheus counters) added in v3.2.0.
  *
  * Contract:
- *   - resolves toolCost once per call (params-aware; 0-cost Tier-0 tools skip
- *     the credit check and usage reports entirely — open-core Phase 2)
+ *   - resolves toolCost once per call; every tool is metered (no free tier),
+ *     so a valid API key is required for every invocation
  *   - try/finally guarantees a single `tool invocation` log line per call
  *   - log payload: { toolName, paramHash, durationMs, outcome, creditCost, creatorMode }
  *   - outcome ∈ { 'success' | 'error' | 'insufficient_credits' }
@@ -36,16 +36,12 @@ export function makeWithAuth({ authManager, logger, metrics = null }) {
       const startTime = Date.now();
       const paramHash = hashParams(params);
       const creatorMode = authManager.isCreatorMode();
-      // Params-aware: scrape's screenshot format is metered, other formats free
       const creditCost = creatorMode ? 0 : authManager.getToolCost(toolName, params);
-      // Open-core Phase 2: Tier-0 tools (cost 0) run locally for free — no
-      // credit check, no usage report, and no API key required.
-      const freeTier = creditCost === 0;
       let outcome = 'pending';
       let thrown = null;
 
       try {
-        if (!creatorMode && !freeTier) {
+        if (!creatorMode) {
           const hasCredits = await authManager.checkCredits(creditCost);
           if (!hasCredits) {
             outcome = 'insufficient_credits';
@@ -90,7 +86,7 @@ export function makeWithAuth({ authManager, logger, metrics = null }) {
           // Cost injection must never break the request path
         }
 
-        if (!creatorMode && !freeTier) {
+        if (!creatorMode) {
           await authManager.reportUsage(toolName, creditCost, params, 200, Date.now() - startTime);
         }
 
@@ -98,7 +94,7 @@ export function makeWithAuth({ authManager, logger, metrics = null }) {
       } catch (error) {
         outcome = 'error';
         thrown = error;
-        if (!creatorMode && !freeTier) {
+        if (!creatorMode) {
           await authManager.reportUsage(
             toolName,
             Math.max(1, Math.floor(creditCost * 0.5)),

package/src/tools/extract/extractContent.js

@@ -11,6 +11,11 @@ import { htmlToMarkdown } from '../../utils/htmlToMarkdown.js'; // D3.1
 
 const ExtractContentSchema = z.object({
   url: z.string().url(),
+  // Pre-rendered HTML to process directly instead of fetching `url` (e.g. a
+  // post-action page from scrape_with_actions, or a stealth-browser render in
+  // deep_research). Without this field Zod stripped it and the tool always
+  // re-fetched the URL — silently defeating any pre-fetched-HTML caller.
+  html: z.string().optional(),
   options: z.object({
     // Content extraction options
     useReadability: z.boolean().default(true),

package/src/tools/search/searchWeb.js

@@ -79,9 +79,10 @@ export class SearchWebTool {
     // Check for Creator Mode - allows search without API key for development/testing
     const isCreatorMode = isCreatorModeVerified();
 
-    // Open-core Phase 2: no API key is allowed at construction time (the server
-    // now starts in free-tier mode without one). The key requirement is
-    // enforced at execute() time instead, so Tier-0 tools keep working.
+    // The server can start without a key so the MCP client can list tools, so
+    // construction must not throw here. Every tool is metered and the key
+    // requirement is enforced before execute() runs (withAuth credit check)
+    // and again at execute() time below.
     if (!apiKey && !isCreatorMode) {
       this.searchAdapter = null;
       this.isCreatorModeFallback = false;
@@ -127,7 +128,7 @@ export class SearchWebTool {
       }
       // --- end SearXNG short-circuit ---
 
-      // Free-tier mode: search via the CrawlForge proxy needs an API key
+      // Search via the CrawlForge proxy needs an API key
       if (!this.searchAdapter) {
         throw new Error('CrawlForge API key is required for search functionality. Get one at https://www.crawlforge.dev/signup');
       }