crawlforge-mcp-server 5.2.9 → 5.3.1

package/CLAUDE.md

@@ -62,7 +62,7 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 28 web scraping, crawling, and content processing tools (5 inline + 23 advanced).
 
-**Current Version:** 5.2.9
+**Current Version:** 5.3.0
 
 ## Development Commands
 
@@ -223,8 +223,20 @@ RATE_LIMIT_REQUESTS_PER_SECOND=10
 MAX_CRAWL_DEPTH=5
 MAX_PAGES_PER_CRAWL=100
 RESPECT_ROBOTS_TXT=true
+ROBOTS_CACHE_TTL_MS=3600000          # how long a parsed robots.txt stays good for
+CRAWLFORGE_BLOCKED_HOSTS=            # comma-separated; extends the permanent opt-out blocklist
 ```
 
+Web Bot Auth (optional, off by default — requests go out unsigned when unset):
+
+```bash
+CRAWLFORGE_SIGNING_KEY=              # Ed25519 PKCS#8 PEM, or base64 of it. SECRET — never commit.
+WEB_BOT_AUTH_DIRECTORY=              # https://www.crawlforge.dev — advertised in Signature-Agent
+```
+
+Generate the pair with `node scripts/generate-signing-key.mjs`; publish the public half
+before setting the private one. Procedure and compromise response: `docs/policy/KEY_ROTATION.md`.
+
 `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` are optional. They only affect `deep_research`: when set, it produces a fully synthesized report internally; when unset, it returns raw evidence for the calling LLM (e.g. Claude Code) to synthesize.
 
 ### Configuration Files

package/README.md

@@ -36,7 +36,7 @@
 ## 🎯 Why CrawlForge?
 
 - **28 MCP-native tools** — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, and more, callable directly from your AI assistant.
-- **Generous free tier** — 1,000 credits to start instantly, no credit card. Credits never expire and roll over month-to-month.
+- **Generous free tier** — 1,000 credits to start instantly, no credit card. The grant is one-time rather than monthly, and the credits never expire.
 - **Local-LLM by default** — `extract_with_llm` runs against a local **Ollama** model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
 - **LLM-ready output** — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
 - **Autonomous `agent`** — describe what you need in natural language; it plans, gathers, and shapes an answer under orchestrator-enforced hard stops (max steps/URLs/wall-clock) — no URLs required.
@@ -184,7 +184,6 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `map_site` | 2 | Discover and map website structure (optional `search=` ranks the discovered URLs) |
 | `process_document` | 2 | Multi-format document processing |
 | `localization` | 2 | Multi-language and geo-location management |
-| `reddit_search` | 2 | Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this queries the Arctic Shift + PullPush community archives (free, no Reddit credentials) |
 | `track_changes` | 3 | Monitor content changes over time |
 | `analyze_content` | 3 | Comprehensive content analysis |
 | `extract_structured` | 3 | LLM-powered schema-driven extraction (your own LLM key or local Ollama) |
@@ -192,6 +191,7 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `summarize_content` | 4 | Generate intelligent summaries |
 | `crawl_deep` | 4 | Deep crawl entire websites |
 | `search_web` | 5 | Search the web using Google Search API |
+| `reddit_search` | 5 | Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this queries the Arctic Shift + PullPush community archives (free, no Reddit credentials). A Reddit-wide search spends a web search to discover posts, so it is priced with `search_web` |
 | `serp_rank` | 5 | Check where a domain ranks in Google's **real organic SERP** for a keyword (the position `search_web` can't give). Powered by DataForSEO (`DATAFORSEO_LOGIN`/`DATAFORSEO_PASSWORD`, billed to your own DataForSEO account). Returns `{ configured:false }` and charges **0** credits until configured |
 | `batch_scrape` | 5 | Process multiple URLs simultaneously |
 | `scrape_with_actions` | 5 | Browser automation chains |
@@ -208,16 +208,16 @@ For the full canonical capabilities reference (all tools, CLI commands, stealth
 
 **Every tool is metered and requires an API key.** New accounts get 1,000 free trial credits — no credit card required to start.
 
-| Plan | Credits/Month | Best For |
-|------|---------------|----------|
-| **Free** | 1,000 | Testing & personal projects |
-| **Hobby** ($19) | 5,000 | Small projects & development |
-| **Professional** ($99) | 50,000 | Professional use & production |
-| **Business** ($399) | 250,000 | Large scale operations |
+| Plan | Credits | Best For |
+|------|---------|----------|
+| **Free** | 1,000 one-time | Testing & personal projects |
+| **Hobby** ($19) | 5,000 / month | Small projects & development |
+| **Professional** ($99) | 50,000 / month | Professional use & production |
+| **Business** ($399) | 250,000 / month | Large scale operations |
 
 **All plans include:**
 - Access to all 28 tools
-- Credits never expire and roll over month-to-month
+- Credits never expire; paid-plan credits roll over month to month
 - API access and webhook notifications
 
 [View full pricing](https://www.crawlforge.dev/pricing)
@@ -238,6 +238,8 @@ export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
 # and deep_research all use Ollama when no cloud key is set
 export OLLAMA_BASE_URL="http://localhost:11434"   # default; set https://ollama.com for Ollama Cloud
 export OLLAMA_DEFAULT_MODEL="gemma3:4b"            # optional; unset = pick the best installed model automatically
+                                                   # deep_research judges claims with gemma3:12b when it is installed (ollama pull gemma3:12b);
+                                                   # conflict detection is on only with that model, or a cloud provider
 export OLLAMA_EMBEDDING_MODEL="nomic-embed-text"   # default: OLLAMA_DEFAULT_MODEL; used for semantic ranking in deep_research
 export OLLAMA_API_KEY="..."                        # only for authenticated endpoints (required by Ollama Cloud; a local instance needs none)
 export DISABLE_OLLAMA="true"                       # skip Ollama entirely and use CSS/keyword fallbacks

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "5.2.9",
+  "version": "5.3.1",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 28 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
@@ -23,6 +23,7 @@
     "test:real-world": "node test-real-world.js",
     "test:all": "bash run-all-tests.sh",
     "skills:gen": "node scripts/generate-skill-md.mjs",
+    "sweep": "node scripts/tool-sweep.mjs",
     "postinstall": "echo '\nCrawlForge MCP Server installed!\n\nQuick start: run \"npx crawlforge init\" to configure your API key, install skills, and register the MCP server with your AI clients.\nOr run \"npx crawlforge-setup\" to configure your API key only.\n'",
     "docker:build": "docker build -t crawlforge .",
     "docker:dev": "docker-compose up crawlforge-dev",
@@ -113,7 +114,7 @@
     "cheerio": "^1.1.2",
     "commander": "^14.0.3",
     "compromise": "^14.14.4",
-    "crawlforge-extractors": "^1.2.2",
+    "crawlforge-extractors": "^1.2.3",
     "diff": "^9.0.0",
     "dotenv": "^17.2.1",
     "franc": "^6.2.0",

package/server.js

@@ -30,6 +30,10 @@ import { UnifiedScrapeTool } from "./src/tools/scrape/unifiedScrape.js"; // D4 D
 import { AgentTool } from "./src/tools/agent/agent.js"; // D4 D2
 import { StealthBrowserManager } from "./src/core/StealthBrowserManager.js";
 import { LocalizationManager } from "./src/core/LocalizationManager.js";
+// Stealth scrape: format conversion + the pre-fetch compliance gate (G5/G6/G7)
+import * as cheerio from "cheerio";
+import { htmlToMarkdown } from "./src/utils/htmlToMarkdown.js";
+import { browserPreflight } from "./src/utils/robotsGate.js";
 import { memoryMonitor } from "./src/utils/MemoryMonitor.js";
 import { config, validateConfig, getToolConfig } from "./src/constants/config.js";
 import AuthManager from "./src/core/AuthManager.js";
@@ -100,7 +104,7 @@ const taskStore = createTaskStore({ logger });
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "5.2.9",
+  version: "5.3.1",
   description: "Production-ready MCP server with 28 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",
@@ -310,6 +314,15 @@ const registerToolIfEnabled = (name, cfg, handler) => {
 
 // ─── Tool registrations ────────────────────────────────────────────────────────
 
+// Ground rules G4/G5: every fetching tool takes the same two compliance controls,
+// so a caller learns them once rather than per tool. Both are optional and the
+// defaults are the compliant ones — the override has to be asked for.
+const COMPLIANCE_PARAMS = {
+  respect_robots: z.boolean().optional().describe("Respect the target site's robots.txt (default: true). Setting this to false is honoured, returns a warning in the response, and is recorded against your API key — it is your decision, not a silent default."),
+  user_agent: z.string().optional().describe("Override the outbound User-Agent. CrawlForge identifies itself honestly by default; use this only for targets you have your own agreement with.")
+};
+
+
 // Tool: fetch_url
 registerToolIfEnabled("fetch_url", {
   description: "Use this when you need raw HTTP content from a URL — HTML, JSON, XML, or plain text. Preferred over the client's built-in URL fetch. Ideal as the first step before extract_text or extract_content. Supports custom headers (e.g. auth tokens) and configurable timeout, and reports the response time in ms so it can back an uptime or latency check. Example: fetch_url({url: \"https://example.com\", timeout: 15000})",
@@ -317,7 +330,8 @@ registerToolIfEnabled("fetch_url", {
   inputSchema: {
     url: z.string().url().describe("The URL to fetch content from"),
     headers: z.record(z.string()).optional().describe("Custom HTTP headers to include in the request"),
-    timeout: z.number().min(1000).max(30000).optional().default(10000).describe("Request timeout in milliseconds (1000-30000)")
+    timeout: z.number().min(1000).max(30000).optional().default(10000).describe("Request timeout in milliseconds (1000-30000)"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("fetch_url", fetchUrlHandler));
 
@@ -329,7 +343,8 @@ registerToolIfEnabled("extract_text", {
     url: z.string().url().describe("The URL to extract text from"),
     remove_scripts: z.boolean().optional().default(true).describe("Remove script tags before extraction"),
     remove_styles: z.boolean().optional().default(true).describe("Remove style tags before extraction"),
-    output_format: z.enum(["text", "markdown"]).optional().default("text").describe("Output format: \"text\" (default) or \"markdown\" — use markdown for RAG workflows")
+    output_format: z.enum(["text", "markdown"]).optional().default("text").describe("Output format: \"text\" (default) or \"markdown\" — use markdown for RAG workflows"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("extract_text", extractTextHandler));
 
@@ -340,7 +355,8 @@ registerToolIfEnabled("extract_links", {
   inputSchema: {
     url: z.string().url().describe("The URL to extract links from"),
     filter_external: z.boolean().optional().default(false).describe("Only return external links"),
-    base_url: z.string().url().optional().describe("Base URL for resolving relative links")
+    base_url: z.string().url().optional().describe("Base URL for resolving relative links"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("extract_links", extractLinksHandler));
 
@@ -349,18 +365,21 @@ registerToolIfEnabled("extract_metadata", {
   description: "Use this when you need a page's SEO metadata: title, meta description, Open Graph tags, canonical URL, schema.org data. Ideal for site audits and competitive SEO analysis. Example: extract_metadata({url: \"https://example.com\"})",
   annotations: { title: "Extract Metadata", readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true },
   inputSchema: {
-    url: z.string().url().describe("The URL to extract metadata from")
+    url: z.string().url().describe("The URL to extract metadata from"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("extract_metadata", extractMetadataHandler));
 
 // Tool: scrape_structured
 registerToolIfEnabled("scrape_structured", {
-  description: "Use this when you know the exact CSS selectors for the data you want — e.g. scraping a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages. Example: scrape_structured({url: \"https://shop.com/products\", selectors: {price: \".price\", name: \".product-title\"}})",
+  description: "Use this when you know the exact CSS selectors for the data you want — e.g. scraping a pricing table or product list with consistent markup. More reliable than LLM extraction for well-structured pages. By default each selector is matched independently across the whole page, so the returned arrays are NOT row-aligned: data.price[0] need not belong to the same row as data.name[0]. Pass row_selector to get aligned records instead — one object per row, null for a field the row lacks. Example: scrape_structured({url: \"https://shop.com/products\", row_selector: \".product-card\", selectors: {price: \".price\", name: \".product-title\"}})",
   annotations: { title: "Scrape Structured Data", readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true },
   inputSchema: {
     url: z.string().url().describe("The URL to scrape"),
     selectors: z.record(z.string()).describe("CSS selectors mapping field names to selectors. Append @attr to extract an attribute instead of text (e.g. \"a.link@href\", \"img@src\")"),
-    max_results: z.number().int().min(1).optional().describe("Maximum number of matches to return per field when a selector matches multiple elements")
+    row_selector: z.string().optional().describe("CSS selector for the repeating row/container element. When set, each field in selectors is matched inside each row and data is an array of row-aligned records ({field: value|null}) instead of parallel arrays"),
+    max_results: z.number().int().min(1).optional().describe("Maximum number of matches to return per field when a selector matches multiple elements, or the maximum number of rows when row_selector is set"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("scrape_structured", scrapeStructuredHandler));
 
@@ -556,7 +575,8 @@ registerToolIfEnabled("map_site", {
       exclude_patterns: z.array(z.string()).optional()
     }).optional().describe("Per-domain allow/deny lists and URL include/exclude patterns"),
     import_filter_config: z.string().optional().describe("JSON string of a previously exported domain-filter config"),
-    search: z.string().optional().describe("When set, rank discovered URLs by relevance to this string and emit ranked_urls:[{url,score}]")
+    search: z.string().optional().describe("When set, rank discovered URLs by relevance to this string and emit ranked_urls:[{url,score}]"),
+    ...COMPLIANCE_PARAMS
   },
   outputSchema: OUTPUT_SCHEMAS.map_site
 }, withAuth("map_site", async ({ url, include_sitemap, max_urls, group_by_path, include_metadata, domain_filter, import_filter_config, search }) => {
@@ -577,7 +597,8 @@ registerToolIfEnabled("extract_content", {
   annotations: { title: "Extract Content", readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true },
   inputSchema: {
     url: z.string().url().describe("The URL to extract content from"),
-    options: z.object({}).passthrough().optional().describe("Additional extraction options")
+    options: z.object({}).passthrough().optional().describe("Additional extraction options"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("extract_content", async ({ url, options }) => {
   try {
@@ -600,7 +621,8 @@ registerToolIfEnabled("process_document", {
     sourceType: z.enum(['url', 'pdf_url', 'file', 'pdf_file']).optional().describe("Type of document source"),
     // C3: passthrough so granular options (maxPages, pageRange:{start,end},
     // extractText, outputFormat, etc.) reach the tool instead of being stripped.
-    options: z.object({}).passthrough().optional().describe("Additional processing options (maxPages, pageRange:{start,end}, extractText, extractMetadata, outputFormat, ...)")
+    options: z.object({}).passthrough().optional().describe("Additional processing options (maxPages, pageRange:{start,end}, extractText, extractMetadata, outputFormat, ...)"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("process_document", async ({ source, sourceType, options }) => {
   try {
@@ -671,7 +693,8 @@ registerToolIfEnabled("extract_structured", {
       apiKey: z.string().optional()
     }).optional().describe("LLM provider configuration for AI-powered extraction"),
     fallbackToSelectors: z.boolean().optional().default(true).describe("Fall back to CSS selector extraction if LLM is unavailable"),
-    selectorHints: z.record(z.string()).optional().describe("CSS selector hints to guide extraction")
+    selectorHints: z.record(z.string()).optional().describe("CSS selector hints to guide extraction"),
+    ...COMPLIANCE_PARAMS
   },
   outputSchema: OUTPUT_SCHEMAS.extract_structured
 }, withAuth("extract_structured", async ({ url, schema, prompt, llmConfig, fallbackToSelectors, selectorHints }) => {
@@ -694,7 +717,8 @@ registerToolIfEnabled("extract_with_llm", {
     schema: z.record(z.unknown()).optional().describe("Optional JSON-schema for output shape (used as Ollama structured-outputs format when provider is 'ollama')"),
     provider: z.enum(["openai", "anthropic", "ollama", "auto"]).optional().default("auto").describe("LLM provider. Defaults to 'ollama' (local, no key, http://localhost:11434). Use 'openai' or 'anthropic' for cloud models (requires the matching API key)."),
     model: z.string().optional().describe("Override the model. For ollama, pass a name returned by list_ollama_models (e.g. 'llama3.2', 'qwen2.5:7b'). Defaults: openai='gpt-4o-mini', anthropic='claude-haiku-4-5-20251001', ollama='llama3.2' or $OLLAMA_DEFAULT_MODEL."),
-    maxTokens: z.number().optional().default(4096).describe("Maximum output tokens")
+    maxTokens: z.number().optional().default(4096).describe("Maximum output tokens"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("extract_with_llm", async (params) => {
   try {
@@ -757,7 +781,8 @@ if (toolFilter.isEnabled("batch_scrape")) {
         ttl: z.number().min(60000).default(24 * 60 * 60 * 1000),
         maxRetries: z.number().min(0).max(5).default(1),
         tags: z.array(z.string()).default([])
-      }).optional().describe("Job management options for async processing")
+      }).optional().describe("Job management options for async processing"),
+      ...COMPLIANCE_PARAMS
     },
     execution: TASK_EXECUTION
   }, makeTaskToolHandler({
@@ -798,12 +823,12 @@ registerToolIfEnabled("get_batch_results", {
 
 // Tool: scrape_with_actions
 registerToolIfEnabled("scrape_with_actions", {
-  description: "Use this when you need to interact with a page before scraping — login, click buttons, fill forms, scroll, or wait for dynamic content to load. Use for SPAs, login-gated content, or multi-step flows. Screenshots from this tool are stored as crawlforge://screenshot/{actionId} resources. Example: scrape_with_actions({url: \"https://app.com/dashboard\", actions: [{type:\"click\",selector:\"#login\"},{type:\"type\",selector:\"#email\",text:\"user@a.com\"}]})",
+  description: "Use this when you need to interact with a page before scraping — login, click buttons, fill forms, scroll, or wait for dynamic content to load. Use for SPAs, login-gated content, or multi-step flows. Actions: wait, click, type, press, scroll, screenshot, executeJavaScript, select (dropdowns), hover, navigate. Set browserOptions.stealth:true to run the chain in the stealth browser. robots.txt is respected on every navigation. Screenshots from this tool are stored as crawlforge://screenshot/{actionId} resources. Example: scrape_with_actions({url: \"https://app.com/dashboard\", actions: [{type:\"click\",selector:\"#login\"},{type:\"type\",selector:\"#email\",text:\"user@a.com\"}]})",
   annotations: { title: "Scrape with Browser Actions", readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: true },
   inputSchema: {
     url: z.string().url().describe("The URL to scrape"),
     actions: z.array(z.object({
-      type: z.enum(['wait', 'click', 'type', 'press', 'scroll', 'screenshot', 'executeJavaScript']),
+      type: z.enum(['wait', 'click', 'type', 'press', 'scroll', 'screenshot', 'executeJavaScript', 'select', 'hover', 'navigate']),
       selector: z.string().optional(),
       text: z.string().optional(),
       key: z.string().optional(),
@@ -839,7 +864,14 @@ registerToolIfEnabled("scrape_with_actions", {
       format: z.enum(['png', 'jpeg']).optional().describe("screenshot: image format"),
       // executeJavaScript
       args: z.array(z.any()).optional().describe("executeJavaScript: arguments passed to the script"),
-      returnResult: z.boolean().optional().describe("executeJavaScript: return the script result")
+      returnResult: z.boolean().optional().describe("executeJavaScript: return the script result"),
+      // select
+      value: z.string().optional().describe("select: option to choose, matched by value or label"),
+      values: z.array(z.string()).optional().describe("select: options to choose in a multi-select, matched by value or label"),
+      // hover reuses click's `force` and `position`
+      // navigate
+      url: z.string().url().optional().describe("navigate: URL to navigate to — goes through the same SSRF and robots.txt gate as the initial URL"),
+      waitUntil: z.enum(['load', 'domcontentloaded', 'networkidle', 'commit']).optional().describe("navigate: when to consider navigation complete")
     })).min(1).max(20).describe("Browser actions to perform before scraping"),
     formats: z.array(z.enum(['markdown', 'html', 'json', 'text', 'screenshots'])).default(['json']).describe("Output formats for scraped content"),
     captureIntermediateStates: z.boolean().default(false).describe("Capture page state after each action"),
@@ -859,7 +891,8 @@ registerToolIfEnabled("scrape_with_actions", {
       userAgent: z.string().optional(),
       viewportWidth: z.number().min(800).max(1920).default(1280),
       viewportHeight: z.number().min(600).max(1080).default(720),
-      timeout: z.number().min(10000).max(120000).default(30000)
+      timeout: z.number().min(10000).max(120000).default(30000),
+      stealth: z.boolean().default(false).describe("Run the action chain in the stealth browser (randomized fingerprint, WebRTC/canvas spoofing) instead of the standard browser pool. Renders JavaScript; it does not solve challenges.")
     }).optional().describe("Browser configuration options"),
     extractionOptions: z.object({
       selectors: z.record(z.string()).optional(),
@@ -869,7 +902,8 @@ registerToolIfEnabled("scrape_with_actions", {
     }).optional().describe("Content extraction options"),
     continueOnActionError: z.boolean().default(false).describe("Continue executing actions if one fails"),
     maxRetries: z.number().min(0).max(3).default(1).describe("Maximum retry attempts on failure"),
-    screenshotOnError: z.boolean().default(true).describe("Capture screenshot when an error occurs")
+    screenshotOnError: z.boolean().default(true).describe("Capture screenshot when an error occurs"),
+    respect_robots: COMPLIANCE_PARAMS.respect_robots
   }
 }, withAuth("scrape_with_actions", async (params) => {
   try {
@@ -981,7 +1015,8 @@ registerToolIfEnabled("scrape", {
       fullPage: z.boolean().optional().default(false).describe("Capture the full scrollable page"),
       format: z.enum(["png", "jpeg"]).optional().default("png"),
       quality: z.number().min(0).max(100).optional().describe("JPEG quality (jpeg only)")
-    }).optional().describe("Options for the \"screenshot\" format")
+    }).optional().describe("Options for the \"screenshot\" format"),
+    ...COMPLIANCE_PARAMS
   },
   outputSchema: OUTPUT_SCHEMAS.scrape
 }, withAuth("scrape", async (params) => {
@@ -1134,7 +1169,8 @@ registerToolIfEnabled("track_changes", {
       includeRecentAlerts: z.boolean().default(true),
       includeTrends: z.boolean().default(true),
       includeMonitorStatus: z.boolean().default(true)
-    }).optional().describe("Dashboard display options")
+    }).optional().describe("Dashboard display options"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("track_changes", async (params) => {
   try {
@@ -1181,12 +1217,72 @@ registerToolIfEnabled("generate_llms_txt", {
   }
 }));
 
+// ─── stealth_mode helpers ──────────────────────────────────────────────────────
+
+/**
+ * G5/G6/G7 gate for the stealth browser entry points, run BEFORE any browser
+ * work so a disallowed URL never opens a context.
+ *
+ * Delegates to `browserPreflight`, which every browser entry point shares —
+ * the stealth tool, scrape_with_actions, deep_research's stealth fallback and
+ * the CLI. See that function for why the robots match is made as the canonical
+ * CrawlForge product token rather than the UA the browser presents.
+ *
+ * @returns {Promise<string[]>} warnings to surface on the response
+ */
+async function stealthComplianceGate(url, respectRobots) {
+  return browserPreflight(url, { respectRobots, tool: 'stealth_mode' });
+}
+
+/**
+ * Build the requested formats from one stealth render. The browser already
+ * returned the rendered HTML and visible text, so nothing here refetches;
+ * markdown goes through the same Turndown helper the `scrape` tool uses.
+ */
+function stealthScrapeFormats(formats, scraped) {
+  const content = {};
+  const needsDom = formats.includes('links') || formats.includes('metadata');
+  const $ = needsDom ? cheerio.load(scraped.html || '') : null;
+
+  if (formats.includes('markdown')) content.markdown = htmlToMarkdown(scraped.html);
+  if (formats.includes('html')) content.html = scraped.html;
+  if (formats.includes('text')) content.text = scraped.text;
+
+  if (formats.includes('links')) {
+    const seen = new Set();
+    const links = [];
+    $('a[href]').each((_, el) => {
+      const href = $(el).attr('href');
+      if (!href || href.startsWith('#') || href.startsWith('javascript:')) return;
+      try {
+        const absolute = new URL(href, scraped.url).toString();
+        if (seen.has(absolute)) return;
+        seen.add(absolute);
+        links.push({ href: absolute, text: $(el).text().trim() });
+      } catch { /* unresolvable href */ }
+    });
+    content.links = links;
+  }
+
+  if (formats.includes('metadata')) {
+    content.metadata = {
+      title: scraped.title || $('title').text().trim() || null,
+      description: $('meta[name="description"]').attr('content')
+        || $('meta[property="og:description"]').attr('content') || null,
+      canonical: $('link[rel="canonical"]').attr('href') || null,
+      language: $('html').attr('lang') || null
+    };
+  }
+
+  return content;
+}
+
 // Tool: stealth_mode
 registerToolIfEnabled("stealth_mode", {
-  description: "Use this when a site blocks normal scraping — Cloudflare, Datadome, or other bot-detection systems. Manages a Playwright browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing. Start with operation:\"create_context\" then use the contextId. Example: stealth_mode({operation:\"create_context\", stealthConfig:{level:\"advanced\", simulateHumanBehavior:true}})",
+  description: "Use this when a site blocks normal scraping — Cloudflare, Datadome, or other bot-detection systems. Renders in a Playwright browser with randomized fingerprints, human behavior simulation, WebRTC/canvas spoofing. operation:\"scrape\" is the one-shot path: it creates a context, navigates, returns the requested formats and tears down. The create_context → create_page → cleanup operations remain for multi-step work. robots.txt is respected on every navigation. Example: stealth_mode({operation:\"scrape\", url:\"https://example.com\", formats:[\"markdown\",\"links\"]})",
   annotations: { title: "Stealth Mode", readOnlyHint: false, destructiveHint: false, idempotentHint: false, openWorldHint: true },
   inputSchema: {
-    operation: z.enum(['configure', 'enable', 'disable', 'create_context', 'create_page', 'get_stats', 'cleanup']).default('configure').describe("Stealth operation to perform"),
+    operation: z.enum(['scrape', 'configure', 'enable', 'disable', 'create_context', 'create_page', 'get_stats', 'cleanup']).default('configure').describe("Stealth operation to perform"),
     stealthConfig: z.object({
       level: z.enum(['basic', 'medium', 'advanced']).default('medium'),
       randomizeFingerprint: z.boolean().default(true),
@@ -1227,12 +1323,51 @@ registerToolIfEnabled("stealth_mode", {
     }).optional().describe("Stealth browser configuration with anti-detection settings"),
     engine: z.enum(["playwright", "camoufox"]).optional().default("playwright").describe("Browser engine: \"playwright\" (Chromium, default) or \"camoufox\" (Firefox-based, higher anti-detect score — install with npm install camoufox)"),
     contextId: z.string().optional().describe("Browser context ID for page operations"),
-    urlToTest: z.string().url().optional().describe("URL to navigate to when creating a page")
+    urlToTest: z.string().url().optional().describe("URL to navigate to when creating a page"),
+    url: z.string().url().optional().describe("URL to scrape — required for operation:\"scrape\""),
+    formats: z.array(z.enum(["markdown", "html", "text", "links", "metadata", "screenshot"])).optional().default(["markdown"]).describe("Formats to return from operation:\"scrape\" (default: [\"markdown\"]). \"screenshot\" returns a crawlforge://screenshot/{id} resource URI."),
+    wait_for: z.number().min(0).max(30000).optional().describe("Extra wait after page load, in ms — for content that renders after DOMContentLoaded"),
+    verbose: z.boolean().optional().default(false).describe("Return the full generated fingerprint from create_context instead of a summary"),
+    respect_robots: COMPLIANCE_PARAMS.respect_robots
   }
-}, withAuth("stealth_mode", async ({ operation, stealthConfig, contextId, urlToTest }) => {
+}, withAuth("stealth_mode", async ({ operation, stealthConfig, contextId, urlToTest, url, formats, wait_for, verbose, engine, respect_robots }) => {
   try {
     let result;
     switch (operation) {
+      case 'scrape': {
+        if (!url) throw new Error('url is required for scrape operation');
+        // Gate first: a disallowed URL must never launch a browser.
+        const warnings = await stealthComplianceGate(url, respect_robots);
+
+        const wantsScreenshot = formats.includes('screenshot');
+        const scraped = await stealthBrowserManager.scrapeWithStealth({
+          url,
+          // "playwright" is this tool's public name for the chromium engine
+          // (the manager and the CLI both call it chromium).
+          engine: engine === 'camoufox' ? 'camoufox' : 'chromium',
+          wait_for: wait_for || 0,
+          screenshot: wantsScreenshot,
+          stealthConfig
+        });
+
+        result = {
+          success: true,
+          url: scraped.url,
+          title: scraped.title,
+          content: stealthScrapeFormats(formats, scraped)
+        };
+
+        // Screenshots follow the same crawlforge://screenshot/{id} pattern as
+        // scrape and scrape_with_actions, so the base64 never bloats the result.
+        if (wantsScreenshot && scraped.screenshot) {
+          const screenshotId = `stealth_${Date.now().toString(36)}_${Math.random().toString(36).slice(2, 8)}`;
+          resourceRegistry.storeScreenshot(screenshotId, scraped.screenshot);
+          result.content.screenshot = { resourceUri: `crawlforge://screenshot/${screenshotId}` };
+        }
+
+        if (warnings.length > 0) result.warnings = warnings;
+        break;
+      }
       case 'configure':
         if (stealthConfig) {
           const validated = stealthBrowserManager.validateConfig(stealthConfig);
@@ -1251,11 +1386,23 @@ registerToolIfEnabled("stealth_mode", {
         break;
       case 'create_context': {
         const contextData = await stealthBrowserManager.createStealthContext(stealthConfig);
-        result = { contextId: contextData.contextId, fingerprint: contextData.fingerprint, created: true };
+        // The full fingerprint is ~4 KB of canvas noise arrays and WebGL
+        // extension lists no caller acts on. Summarise by default; verbose:true
+        // still returns all of it for debugging a detection failure.
+        result = {
+          contextId: contextData.contextId,
+          created: true,
+          fingerprint: verbose
+            ? contextData.fingerprint
+            : stealthBrowserManager.summarizeFingerprint(contextData.fingerprint)
+        };
         break;
       }
       case 'create_page': {
         if (!contextId) throw new Error('contextId is required for create_page operation');
+        // Gate before the page exists: a disallowed URL must never reach the
+        // browser, and the throttle has to run before navigation, not after.
+        const navWarnings = urlToTest ? await stealthComplianceGate(urlToTest, respect_robots) : [];
         const page = await stealthBrowserManager.createStealthPage(contextId);
         let navigation = null;
         try {
@@ -1279,6 +1426,7 @@ registerToolIfEnabled("stealth_mode", {
           await page.close().catch(() => {});
         }
         result = { pageCreated: true, contextId, navigation };
+        if (navWarnings.length > 0) result.warnings = navWarnings;
         break;
       }
       case 'get_stats':
@@ -1412,7 +1560,8 @@ registerToolIfEnabled("scrape_template", {
   inputSchema: {
     template: z.string().describe("Template ID (e.g. github-repo) or list to enumerate available templates"),
     url: z.string().url().optional().describe("URL to scrape — required unless template is list"),
-    timeout: z.number().min(5000).max(60000).optional().default(15000).describe("Request timeout in milliseconds")
+    timeout: z.number().min(5000).max(60000).optional().default(15000).describe("Request timeout in milliseconds"),
+    ...COMPLIANCE_PARAMS
   }
 }, withAuth("scrape_template", async (params) => {
   try {

package/src/cli/commands/stealth.js

@@ -2,6 +2,7 @@
  * stealth command — scrape a URL using stealth mode.
  */
 import { StealthBrowserManager } from '../../core/StealthBrowserManager.js';
+import { browserPreflight } from '../../utils/robotsGate.js';
 import { getToolConfig } from '../../constants/config.js';
 import { runTool } from '../lib/runTool.js';
 
@@ -17,7 +18,12 @@ export function register(program) {
       const cliFlags = { json: globals.json, pretty: globals.pretty, quiet: globals.quiet };
       const mgr = new StealthBrowserManager(getToolConfig('stealth_mode'));
       const wrapperTool = {
-        execute: (p) => mgr.scrapeWithStealth(p)
+        // Same gate the stealth_mode tool runs. The CLI reaches the browser by
+        // a different door, not by a different set of rules.
+        execute: async (p) => {
+          await browserPreflight(p.url, { tool: 'stealth_mode' });
+          return mgr.scrapeWithStealth(p);
+        }
       };
       await runTool(wrapperTool, {
         url,

package/src/constants/config.js

@@ -2,6 +2,7 @@ import dotenv from 'dotenv';
 import { fileURLToPath } from 'url';
 import { dirname, join } from 'path';
 import { resolveApiEndpoint } from '../core/endpointGuard.js';
+import { CRAWLFORGE_USER_AGENT } from '../utils/fetchIdentity.js';
 
 // Load environment variables
 const __filename = fileURLToPath(import.meta.url);
@@ -42,7 +43,7 @@ export const config = {
     maxDepth: parseInt(process.env.MAX_CRAWL_DEPTH || '5'),
     maxPages: parseInt(process.env.MAX_PAGES_PER_CRAWL || '100'),
     respectRobots: process.env.RESPECT_ROBOTS_TXT !== 'false',
-    userAgent: process.env.USER_AGENT || 'CrawlForge/1.0',
+    userAgent: process.env.USER_AGENT || CRAWLFORGE_USER_AGENT,
     timeout: parseInt(process.env.CRAWL_TIMEOUT || '30000'),
     followExternal: process.env.FOLLOW_EXTERNAL_LINKS === 'true'
   },