crawlforge-mcp-server 4.5.0 → 4.6.1

package/CLAUDE.md

@@ -60,9 +60,9 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 ## Project Overview
 
-CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 23 web scraping, crawling, and content processing tools (5 inline + 18 advanced).
+CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 26 web scraping, crawling, and content processing tools (5 inline + 21 advanced).
 
-**Current Version:** 4.2.4
+**Current Version:** 4.6.0
 
 ## Development Commands
 
@@ -92,8 +92,10 @@ npm run dev
 # Test MCP protocol compliance
 npm test
 
-# Unit tests (262 tests, no live network)
+# Unit tests (400+ tests across tests/unit/, no live network)
 npm run test:unit
+# Phase D regressions live in tests/unit/phaseD-regressions.test.js (agent hard stops, unified scrape, map_site ranking)
+# Run a single test file:  node --test tests/unit/phaseD-regressions.test.js
 # Note: add --test-force-exit if the run appears to hang at the end — importing
 # StealthBrowserManager (d2-reliability.test.js) leaves a Playwright handle that
 # otherwise delays process exit ~100s. Tests themselves pass either way.
@@ -109,7 +111,9 @@ node test-real-world.js        # Test real-world usage scenarios
 node tests/integration/mcp-protocol-compliance.test.js
 
 # CLI (v4.1.0+, requires global install or npx)
-crawlforge --help              # Show all 15 subcommands
+crawlforge --help              # Show all subcommands
+crawlforge init                # API-key detection + skill install + idempotent MCP-stanza merge (v4.6.0)
+crawlforge init --all --yes    # Merge MCP config into Claude Code / Desktop / Cursor non-interactively
 crawlforge scrape https://example.com
 crawlforge batch --urls urls.txt --format markdown
 crawlforge install-skills --target claude-code
@@ -140,6 +144,7 @@ npm run docker:prod         # Run production container
 - **WebhookDispatcher**: Event notification system for job completion callbacks
 - **ActionExecutor**: Browser automation engine (Playwright-based)
 - **ResearchOrchestrator**: Multi-stage research with query expansion and synthesis
+- **AgentOrchestrator**: Powers the `agent` tool — NL prompt → autonomous PLAN→GATHER→ACT→DECIDE→SHAPE loop with three orchestrator-enforced hard stops (maxSteps≤10, maxUrls≤20, wall-clock) never delegated to the LLM; degraded no-LLM-key path (D2, v4.6.0)
 - **StealthBrowserManager**: Stealth mode scraping with anti-detection; Camoufox (Firefox) engine added in v4.0.0
 - **LocalizationManager**: Multi-language content and localization
 - **ChangeTracker**: Content change tracking over time
@@ -155,7 +160,9 @@ npm run docker:prod         # Run production container
 Tools are organized in subdirectories by category:
 
 - `advanced/` - BatchScrapeTool, ScrapeWithActionsTool
+- `agent/` - agent (AgentOrchestrator-driven autonomous tool, v4.6.0)
 - `basic/` - fetchUrl, extractText, extractLinks, extractMetadata, scrapeStructured
+- `scrape/` - unifiedScrape (single-fetch multi-format `scrape` tool, v4.6.0)
 - `crawl/` - crawlDeep, mapSite
 - `extract/` - analyzeContent, extractContent, extractStructured, extractWithLlm, listOllamaModels, processDocument, summarizeContent
 - `research/` - deepResearch
@@ -164,13 +171,18 @@ Tools are organized in subdirectories by category:
 - `tracking/` - trackChanges
 - `llmstxt/` - generateLLMsTxt
 
-### Available MCP Tools (23 total)
+### Available MCP Tools (26 total)
 
 **Basic Tools (server.js inline, 5):**
 fetch_url, extract_text, extract_links, extract_metadata, scrape_structured
 
-**Advanced Tools (18):**
-search_web, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template
+**Advanced Tools (21):**
+search_web, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template, scrape, agent
+
+**v4.6.0 additions (Phase D):**
+- `scrape` — single fetch + one cheerio load dispatching a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) + `onlyMainContent`; partial-success via per-format `warnings[]`. Cost: 2.
+- `agent` — NL prompt → autonomous research/extract, no URLs required (see AgentOrchestrator above). Cost: 8.
+- `map_site` gained an optional `search=` param that ranks discovered URLs (`ranked_urls:[{url,score}]`); default output unchanged.
 
 ### MCP Server Entry Point
 

package/README.md

@@ -9,7 +9,7 @@ Professional web scraping and content extraction server implementing the Model C
 
 ## 🎯 Features
 
-- **23 Professional Tools**: Web scraping, deep research, stealth browsing, content analysis, local-LLM extraction (Ollama)
+- **26 Professional Tools**: Web scraping, deep research, an autonomous `agent`, a unified multi-format `scrape`, stealth browsing, content analysis, local-LLM extraction (Ollama)
 - **Free Tier**: 1,000 credits to get started instantly
 - **MCP Compatible**: Works with Claude, Cursor, and other MCP-enabled AI tools
 - **Enterprise Ready**: Scale up with paid plans for production use
@@ -37,6 +37,8 @@ This will:
 
 **Don't have an API key?** Get one free at [https://www.crawlforge.dev/signup](https://www.crawlforge.dev/signup)
 
+> **One-step setup (v4.6.0+):** `crawlforge init` detects your API key, installs the agent skill, and idempotently merges the MCP config stanza into Claude Code, Claude Desktop, and Cursor. Use `crawlforge init --all --yes` to configure every detected client non-interactively.
+
 ### 3. Configure Your IDE (if not auto-configured)
 
 <details>
@@ -107,8 +109,10 @@ Restart Cursor to activate.
 - `extract_text` - Extract clean text from web pages
 - `extract_links` - Get all links from a page
 - `extract_metadata` - Extract page metadata
+- `scrape_template` - Structured data from well-known sites (Amazon, GitHub, LinkedIn, YouTube, Reddit, Hacker News, npm, and more) without writing selectors
 
 ### Advanced Tools (2-3 credits)
+- `scrape` - **Unified single-fetch, multi-format extraction.** Pass a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) plus `onlyMainContent`; one fetch serves every requested format with per-format partial-success warnings
 - `scrape_structured` - Extract structured data with CSS selectors
 - `search_web` - Search the web using Google Search API
 - `summarize_content` - Generate intelligent summaries
@@ -117,10 +121,12 @@ Restart Cursor to activate.
 - `extract_with_llm` - Natural-language extraction. **Defaults to a local Ollama model — no API key, no API costs.** Pass `provider: "openai" | "anthropic"` with the matching key for cloud models.
 - `list_ollama_models` - List the Ollama models installed locally (free; helps you pick a `model` for `extract_with_llm`)
 - `track_changes` - Monitor content changes over time
+- `get_batch_results` - Retrieve paginated results for a `batch_scrape` job by `batchId`
 
 ### Premium Tools (5-10 credits)
+- `agent` - **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM)
 - `crawl_deep` - Deep crawl entire websites
-- `map_site` - Discover and map website structure
+- `map_site` - Discover and map website structure (optional `search=` ranks the discovered URLs)
 - `batch_scrape` - Process multiple URLs simultaneously
 - `deep_research` - Multi-stage research with source verification
 - `stealth_mode` - Anti-detection browser management
@@ -132,6 +138,8 @@ Restart Cursor to activate.
 - `generate_llms_txt` - Generate AI interaction guidelines
 - `localization` - Multi-language and geo-location management
 
+For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see [SKILL.md](SKILL.md).
+
 ## 💳 Pricing
 
 | Plan | Credits/Month | Best For |
@@ -142,7 +150,7 @@ Restart Cursor to activate.
 | **Enterprise** | 250,000 | Large scale operations |
 
 **All plans include:**
-- Access to all 23 tools
+- Access to all 26 tools
 - Credits never expire and roll over month-to-month
 - API access and webhook notifications
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "4.5.0",
+  "version": "4.6.1",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 23 web scraping, crawling, and content processing tools. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. v4.0 adds Markdown-first output, pre-built site templates, Camoufox stealth engine, and cost transparency.",
   "main": "server.js",
   "bin": {
@@ -21,7 +21,7 @@
     "test:tools": "node test-tools.js",
     "test:real-world": "node test-real-world.js",
     "test:all": "bash run-all-tests.sh",
-    "postinstall": "echo '\n🎉 CrawlForge MCP Server installed!\n\nRun \"npx crawlforge-setup\" to configure your API key and get started.\n'",
+    "postinstall": "echo '\nCrawlForge MCP Server installed!\n\nQuick start: run \"npx crawlforge init\" to configure your API key, install skills, and register the MCP server with your AI clients.\nOr run \"npx crawlforge-setup\" to configure your API key only.\n'",
     "docker:build": "docker build -t crawlforge .",
     "docker:dev": "docker-compose up crawlforge-dev",
     "docker:prod": "docker-compose up crawlforge-prod"

package/server.js

@@ -24,6 +24,8 @@ import { DeepResearchTool } from "./src/tools/research/deepResearch.js";
 import { TrackChangesTool } from "./src/tools/tracking/trackChanges/index.js";
 import { GenerateLLMsTxtTool } from "./src/tools/llmstxt/generateLLMsTxt.js";
 import { ScrapeTemplateTool } from "./src/tools/templates/ScrapeTemplateTool.js"; // D3.3
+import { UnifiedScrapeTool } from "./src/tools/scrape/unifiedScrape.js"; // D4 D1
+import { AgentTool } from "./src/tools/agent/agent.js"; // D4 D2
 import { StealthBrowserManager } from "./src/core/StealthBrowserManager.js";
 import { LocalizationManager } from "./src/core/LocalizationManager.js";
 import { memoryMonitor } from "./src/utils/MemoryMonitor.js";
@@ -97,7 +99,7 @@ if (configErrors.length > 0 && config.server.nodeEnv === 'production') {
 const server = new McpServer({
   name: "crawlforge",
   version: "4.5.0",
-  description: "Production-ready MCP server with 24 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, change tracking, and local-LLM extraction via Ollama.",
+  description: "Production-ready MCP server with 26 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png"
 });
@@ -111,7 +113,7 @@ server.prompt("getting-started", {
       role: "user",
       content: {
         type: "text",
-        text: "You have access to CrawlForge MCP with 23 web scraping tools. Key tools:\n\n" +
+        text: "You have access to CrawlForge MCP with 26 web scraping tools. Key tools:\n\n" +
           "- fetch_url: Fetch raw HTML/content from any URL\n" +
           "- extract_text: Extract clean text from a webpage\n" +
           "- extract_content: Smart content extraction with readability\n" +
@@ -161,6 +163,8 @@ const deepResearchTool = new DeepResearchTool();
 const trackChangesTool = new TrackChangesTool();
 const generateLLMsTxtTool = new GenerateLLMsTxtTool();
 const scrapeTemplateTool = new ScrapeTemplateTool(); // D3.3
+const unifiedScrapeTool = new UnifiedScrapeTool(); // D4 D1
+const agentTool = new AgentTool(); // D4 D2
 const stealthBrowserManager = new StealthBrowserManager();
 const localizationManager = new LocalizationManager();
 
@@ -181,6 +185,7 @@ deepResearchTool.setMcpServer(server);
 batchScrapeTool.setMcpServer(server);
 crawlDeepTool.setMcpServer(server);
 extractStructuredTool.setMcpServer(server);
+agentTool.setMcpServer(server); // D4 D2: SamplingClient + Elicitation
 AuthManager.setElicitation(elicitation);
 
 // ─── D1.1 Resource Templates (MCP Resources) ─────────────────────────────────
@@ -433,14 +438,15 @@ server.registerTool("map_site", {
       include_patterns: z.array(z.string()).optional(),
       exclude_patterns: z.array(z.string()).optional()
     }).optional().describe("Per-domain allow/deny lists and URL include/exclude patterns"),
-    import_filter_config: z.string().optional().describe("JSON string of a previously exported domain-filter config")
+    import_filter_config: z.string().optional().describe("JSON string of a previously exported domain-filter config"),
+    search: z.string().optional().describe("When set, rank discovered URLs by relevance to this string and emit ranked_urls:[{url,score}]")
   }
-}, withAuth("map_site", async ({ url, include_sitemap, max_urls, group_by_path, include_metadata, domain_filter, import_filter_config }) => {
+}, withAuth("map_site", async ({ url, include_sitemap, max_urls, group_by_path, include_metadata, domain_filter, import_filter_config, search }) => {
   try {
     if (!url) {
       return { content: [{ type: "text", text: "URL parameter is required" }], isError: true };
     }
-    const result = await mapSiteTool.execute({ url, include_sitemap, max_urls, group_by_path, include_metadata, domain_filter, import_filter_config });
+    const result = await mapSiteTool.execute({ url, include_sitemap, max_urls, group_by_path, include_metadata, domain_filter, import_filter_config, search });
     return { content: [{ type: "text", text: JSON.stringify(result, null, 2) }] };
   } catch (error) {
     return { content: [{ type: "text", text: `Site mapping failed: ${error.message}` }], isError: true };
@@ -801,6 +807,53 @@ server.registerTool("deep_research", {
   }
 }));
 
+// Tool: scrape (D4 D1 — unified multi-format single-fetch)
+server.registerTool("scrape", {
+  description: "Use this when you need multiple content formats from a single URL in one call — e.g. markdown + links + metadata together. One fetch, no N-request fan-out. Formats: \"markdown\", \"html\", \"rawHtml\", \"text\", \"links\", \"metadata\", or {type:\"json\",schema,prompt} for LLM-structured extraction. onlyMainContent:true (default) strips boilerplate via Readability. Partial success: per-format warnings never fail the whole call. Example: scrape({url:\"https://example.com\", formats:[\"markdown\",\"links\",\"metadata\"]})",
+  annotations: { title: "Scrape (Multi-Format)", readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true },
+  inputSchema: {
+    url: z.string().url().describe("The URL to scrape"),
+    formats: z.array(z.union([
+      z.enum(["markdown", "html", "rawHtml", "text", "links", "metadata", "screenshot"]),
+      z.object({
+        type: z.literal("json"),
+        schema: z.record(z.any()).optional().describe("JSON schema for extraction"),
+        prompt: z.string().optional().describe("Extraction instruction for the LLM")
+      })
+    ])).min(1).optional().default(["markdown"]).describe("Formats to return (default: [\"markdown\"])"),
+    onlyMainContent: z.boolean().optional().default(true).describe("Strip boilerplate via Readability (default: true)"),
+    timeoutMs: z.number().min(1000).max(60000).optional().default(15000).describe("Fetch timeout in ms")
+  }
+}, withAuth("scrape", async (params) => {
+  try {
+    const result = await unifiedScrapeTool.execute(params);
+    return { content: [{ type: "text", text: JSON.stringify(result, null, 2) }] };
+  } catch (error) {
+    return { content: [{ type: "text", text: `Scrape failed: ${error.message}` }], isError: true };
+  }
+}));
+
+// Tool: agent (D4 D2 — autonomous NL prompt → search/navigate/extract)
+server.registerTool("agent", {
+  description: "Use this when you need an autonomous agent to research, navigate, and synthesise an answer from the web — no URLs required. The agent plans search queries, fetches and filters relevant pages, and returns a prose or structured answer. model:\"pro\" uses deep multi-source research. Hard limits: maxSteps≤10, maxUrls≤20, 120s wall-clock. Confirms before pro runs. Degraded-but-useful output if no LLM keys/Ollama. Example: agent({prompt:\"What are the top 5 MCP servers in 2025?\", maxUrls:10})",
+  annotations: { title: "Agent (Autonomous)", readOnlyHint: true, destructiveHint: false, idempotentHint: false, openWorldHint: true },
+  inputSchema: {
+    prompt: z.string().min(1).max(2000).describe("Natural-language task or question"),
+    urls: z.array(z.string().url()).max(20).optional().describe("Optional seed URLs to include (max 20)"),
+    schema: z.record(z.any()).optional().describe("Optional JSON schema for structured output"),
+    model: z.enum(["default", "pro"]).optional().default("default").describe("\"default\" = SamplingClient loop (no keys needed); \"pro\" = full ResearchOrchestrator"),
+    maxSteps: z.number().min(1).max(10).optional().default(5).describe("Max fetch iterations (hard cap: 10)"),
+    maxUrls: z.number().min(1).max(20).optional().default(10).describe("Max URLs to fetch (hard cap: 20)")
+  }
+}, withAuth("agent", async (params) => {
+  try {
+    const result = await agentTool.execute(params);
+    return { content: [{ type: "text", text: JSON.stringify(result, null, 2) }] };
+  } catch (error) {
+    return { content: [{ type: "text", text: `Agent failed: ${error.message}` }], isError: true };
+  }
+}));
+
 // Tool: track_changes
 server.registerTool("track_changes", {
   description: "Use this when you need to monitor a URL for content changes over time — e.g. competitor pricing, regulation updates, product availability. Start with operation:\"create_baseline\", then periodically use operation:\"compare\" to diff. Supports webhooks and scheduled monitoring. Example: track_changes({url: \"https://example.com/pricing\", operation: \"create_baseline\"})",
@@ -1224,9 +1277,10 @@ async function runServer() {
     "batch_scrape", "get_batch_results", "scrape_with_actions",
     "deep_research", "track_changes", "generate_llms_txt",
     "stealth_mode", "localization", "extract_structured", "extract_with_llm",
-    "list_ollama_models", "scrape_template"  // D3.3
+    "list_ollama_models", "scrape_template", // D3.3
+    "scrape", "agent"  // D4
   ];
-  console.error(`Tools available (24): ${allTools.join(", ")}`);
+  console.error(`Tools available (26): ${allTools.join(", ")}`);
 
   // Start memory monitoring in development
   if (config.server.nodeEnv === "development") {
@@ -1252,7 +1306,8 @@ async function gracefulShutdown(signal) {
     const toolsToCleanup = [
       batchScrapeTool, scrapeWithActionsTool, deepResearchTool,
       trackChangesTool, generateLLMsTxtTool, stealthBrowserManager,
-      localizationManager, extractStructuredTool
+      localizationManager, extractStructuredTool,
+      agentTool // D4 D2: may hold ResearchOrchestrator
     ].filter(tool => tool && (typeof tool.destroy === 'function' || typeof tool.cleanup === 'function'));
 
     console.error(`Cleaning up ${toolsToCleanup.length} tools...`);

package/src/cli/commands/init.js

@@ -0,0 +1,107 @@
+/**
+ * init command — one-shot setup: API key check + skill install + MCP stanza merge.
+ */
+import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
+import { join } from 'node:path';
+import { install } from '../../skills/installer.js';
+
+const HOME = process.env.HOME || process.env.USERPROFILE || '';
+
+function loadStoredApiKey() {
+  try {
+    const cfg = JSON.parse(readFileSync(join(HOME, '.crawlforge', 'config.json'), 'utf8'));
+    return cfg.apiKey || undefined;
+  } catch {
+    return undefined;
+  }
+}
+
+function mcpStanza(apiKey) {
+  const stanza = { command: 'npx', args: ['-y', 'crawlforge@latest', 'mcp'] };
+  if (apiKey) stanza.env = { CRAWLFORGE_API_KEY: apiKey };
+  return stanza;
+}
+
+function mergeClientConfig(configPath, apiKey) {
+  let existing = {};
+  if (existsSync(configPath)) {
+    try { existing = JSON.parse(readFileSync(configPath, 'utf8')); } catch { /* keep {} */ }
+  } else {
+    const dir = configPath.substring(0, configPath.lastIndexOf('/'));
+    if (dir) mkdirSync(dir, { recursive: true });
+  }
+  existing.mcpServers = existing.mcpServers || {};
+  existing.mcpServers.crawlforge = mcpStanza(apiKey);
+  writeFileSync(configPath, JSON.stringify(existing, null, 2) + '\n', 'utf8');
+  return configPath;
+}
+
+function resolveClientPaths(client) {
+  const paths = [];
+  if (!client || client === 'claude-code') {
+    paths.push({ label: 'Claude Code', path: join(HOME, '.claude.json') });
+  }
+  if (!client || client === 'claude-desktop') {
+    const desktopPath = process.platform === 'darwin'
+      ? join(HOME, 'Library', 'Application Support', 'Claude', 'claude_desktop_config.json')
+      : process.platform === 'win32'
+        ? join(process.env.APPDATA || join(HOME, 'AppData', 'Roaming'), 'Claude', 'claude_desktop_config.json')
+        : join(HOME, '.config', 'Claude', 'claude_desktop_config.json');
+    paths.push({ label: 'Claude Desktop', path: desktopPath });
+  }
+  if (!client || client === 'cursor') {
+    paths.push({ label: 'Cursor', path: join(HOME, '.cursor', 'mcp.json') });
+  }
+  return paths;
+}
+
+export function register(program) {
+  program
+    .command('init')
+    .description('Set up CrawlForge: verify API key, install skills, and register the MCP server with your AI clients')
+    .option('--all', 'Install skills to all targets and register all detected client configs')
+    .option('--client <name>', 'Target client to register: claude-code, claude-desktop, or cursor')
+    .option('--yes', 'Non-interactive — assume yes to all prompts')
+    .action(async (opts) => {
+      const out = (msg) => process.stderr.write(msg + '\n');
+
+      // 1. API key check
+      const apiKey = loadStoredApiKey() || process.env.CRAWLFORGE_API_KEY;
+      if (!apiKey) {
+        out('No CrawlForge API key found.');
+        out('Run: npx crawlforge-setup');
+        out('Then re-run: crawlforge init');
+        process.exit(1);
+      }
+      out('API key: found (' + apiKey.slice(0, 8) + '...)');
+
+      // 2. Install skills
+      const skillTarget = opts.all ? 'all' : 'claude-code';
+      try {
+        const results = await install({ target: skillTarget, force: false, cwd: process.cwd() });
+        if (results.installed.length > 0) {
+          out('Skills installed: ' + results.installed.length + ' file(s)');
+        } else {
+          out('Skills: already up to date (use crawlforge install-skills --force to overwrite)');
+        }
+      } catch (err) {
+        out('Warning: skill install failed — ' + err.message);
+      }
+
+      // 3. MCP stanza merge
+      const clientFilter = opts.client || (opts.all ? undefined : 'claude-code');
+      const targets = resolveClientPaths(clientFilter);
+
+      for (const { label, path: cfgPath } of targets) {
+        try {
+          mergeClientConfig(cfgPath, apiKey);
+          out('MCP registered: ' + label + ' (' + cfgPath + ')');
+        } catch (err) {
+          out('Warning: could not update ' + label + ' config — ' + err.message);
+        }
+      }
+
+      out('Done. Restart your AI client to pick up the crawlforge MCP server.');
+      process.exit(0);
+    });
+}

package/src/cli/index.js

@@ -58,6 +58,7 @@ import { register as registerTemplate } from './commands/template.js';
 import { register as registerMonitor } from './commands/monitor.js';
 import { register as registerInstallSkills } from './commands/install-skills.js';
 import { register as registerUninstallSkills } from './commands/uninstall-skills.js';
+import { register as registerInit } from './commands/init.js';
 
 // ─── MCP stdio server mode (backward compatibility) ──────────────────────────
 // Before v4.1.0 the `crawlforge` bin WAS the MCP server. v4.1.0 turned it into
@@ -136,6 +137,7 @@ registerTemplate(program);
 registerMonitor(program);
 registerInstallSkills(program);
 registerUninstallSkills(program);
+registerInit(program);
 
 // `crawlforge mcp` / `crawlforge serve` — explicitly start the MCP server over
 // stdio. Extra args (e.g. --http) are read directly by server.js from argv.

package/src/core/AgentOrchestrator.js

@@ -0,0 +1,302 @@
+/**
+ * AgentOrchestrator — autonomous NL-prompt → search/navigate/extract → answer.
+ *
+ * Design: hardcoded 3-action state machine.
+ *   PLAN   — one SamplingClient call to decompose prompt into search queries
+ *   GATHER — search_web (≤maxUrls results total)
+ *   ACT    — fetchAndParse + relevance gate per URL
+ *   DECIDE — loop or answer (step/URL/time hard stops; never LLM-trusted)
+ *   SHAPE  — schema→ExtractWithLlm prose→synthesis via SamplingClient
+ *
+ * Hard stops (enforced here, not by the LLM):
+ *   1. maxSteps iterations of the ACT loop
+ *   2. maxUrls total URLs fetched
+ *   3. wallClockMs wall-clock milliseconds (default 120 000)
+ *
+ * No-LLM-key path: if all LLM calls fail, return collected evidence + {degraded:true}.
+ * pro model: delegates to ResearchOrchestrator.conductResearch() for richer synthesis.
+ */
+
+import { fetchAndParse } from '../tools/extract/_fetchAndParse.js';
+import { SamplingClient } from './SamplingClient.js';
+
+const DEFAULT_WALL_CLOCK_MS = 120_000;
+const DEFAULT_MAX_STEPS = 5;
+const DEFAULT_MAX_URLS = 10;
+
+// ── Helpers ───────────────────────────────────────────────────────────────────
+
+/**
+ * Naive relevance gate: does the fetched text contain any query term?
+ * Avoids an LLM call for an obviously irrelevant page.
+ */
+function isRelevant(text, query) {
+  if (!text || !query) return true; // fail-open
+  const lc = text.toLowerCase();
+  return query.toLowerCase().split(/\s+/).some(term => term.length > 3 && lc.includes(term));
+}
+
+/**
+ * Truncate text to a safe token budget (~8 000 chars ≈ ~2 000 tokens).
+ */
+function truncate(text, maxChars = 8000) {
+  if (!text || text.length <= maxChars) return text;
+  return text.slice(0, maxChars) + '\n[...truncated]';
+}
+
+// ── Orchestrator ──────────────────────────────────────────────────────────────
+
+export class AgentOrchestrator {
+  /**
+   * @param {object} options
+   * @param {object|null} options.mcpServer  - McpServer instance (for SamplingClient)
+   * @param {object}      options.searchConfig - passed to SearchWebTool constructor
+   * @param {object}      options.llmConfig    - passed to ExtractWithLlm constructor
+   */
+  constructor(options = {}) {
+    this._mcpServer = options.mcpServer || null;
+    this._searchConfig = options.searchConfig || {};
+    this._llmConfig = options.llmConfig || {};
+    this._samplingClient = null;
+    this._searchTool = null;
+    this._extractWithLlm = null;
+    this._researchOrchestrator = null;
+  }
+
+  /** Set MCP server (called by agent.js after construction). */
+  setMcpServer(mcpServer) {
+    this._mcpServer = mcpServer;
+    this._samplingClient = null; // reset so it is rebuilt with the new server
+  }
+
+  // ── Lazy accessors ──────────────────────────────────────────────────────────
+
+  _getSamplingClient() {
+    if (!this._samplingClient) {
+      this._samplingClient = new SamplingClient({ mcpServer: this._mcpServer });
+    }
+    return this._samplingClient;
+  }
+
+  async _getSearchTool() {
+    if (!this._searchTool) {
+      const { SearchWebTool } = await import('../tools/search/searchWeb.js');
+      this._searchTool = new SearchWebTool(this._searchConfig);
+    }
+    return this._searchTool;
+  }
+
+  async _getExtractWithLlm() {
+    if (!this._extractWithLlm) {
+      const { ExtractWithLlm } = await import('../tools/extract/extractWithLlm.js');
+      this._extractWithLlm = new ExtractWithLlm(this._llmConfig);
+    }
+    return this._extractWithLlm;
+  }
+
+  async _getResearchOrchestrator() {
+    if (!this._researchOrchestrator) {
+      const { ResearchOrchestrator } = await import('./ResearchOrchestrator.js');
+      this._researchOrchestrator = new ResearchOrchestrator({
+        maxUrls: 50,
+        timeLimit: DEFAULT_WALL_CLOCK_MS
+      });
+    }
+    return this._researchOrchestrator;
+  }
+
+  // ── Main entry ──────────────────────────────────────────────────────────────
+
+  /**
+   * Run the agent loop.
+   *
+   * @param {object} params
+   * @param {string}    params.prompt      - Natural-language task
+   * @param {string[]}  [params.urls]      - Seed URLs (skips search for those)
+   * @param {object}    [params.schema]    - JSON schema for structured output
+   * @param {string}    [params.model]     - 'default' | 'pro'
+   * @param {number}    [params.maxSteps]  - Max ACT iterations (≤10)
+   * @param {number}    [params.maxUrls]   - Max URLs to fetch (≤20)
+   * @param {number}    [params.wallClockMs] - Wall-clock budget in ms
+   * @returns {Promise<object>}
+   */
+  async run(params) {
+    const {
+      prompt,
+      urls: seedUrls = [],
+      schema,
+      model = 'default',
+      maxSteps = DEFAULT_MAX_STEPS,
+      maxUrls = DEFAULT_MAX_URLS,
+      wallClockMs = DEFAULT_WALL_CLOCK_MS
+    } = params;
+
+    const startTime = Date.now();
+    const deadline = () => (Date.now() - startTime) >= wallClockMs;
+
+    // Hard-cap params regardless of what caller sends
+    const capSteps = Math.min(maxSteps, 10);
+    const capUrls = Math.min(maxUrls, 20);
+
+    // pro model: delegate to ResearchOrchestrator
+    if (model === 'pro') {
+      try {
+        const orchestrator = await this._getResearchOrchestrator();
+        const result = await orchestrator.conductResearch(prompt, {
+          maxUrls: capUrls,
+          timeLimit: wallClockMs,
+          researchApproach: 'focused'
+        });
+        return { success: true, answer: result, model: 'pro', degraded: false };
+      } catch (err) {
+        // Fall through to default path on pro failure
+        return {
+          success: false,
+          degraded: true,
+          reason: `pro research failed: ${err.message}`,
+          answer: null
+        };
+      }
+    }
+
+    // ── PLAN ──────────────────────────────────────────────────────────────────
+    let searchQueries = [prompt]; // fallback: use raw prompt as query
+    try {
+      const planPrompt =
+        `Decompose this research task into 1-3 concise web search queries (one per line, no bullets):\n\n${prompt}`;
+      const { text } = await this._getSamplingClient().complete(planPrompt, { maxTokens: 200 });
+      const lines = text.split('\n').map(l => l.replace(/^[-*\d.)\s]+/, '').trim()).filter(Boolean);
+      if (lines.length > 0) searchQueries = lines.slice(0, 3);
+    } catch {
+      // Sampling unavailable — use raw prompt
+    }
+
+    // ── GATHER (search) ───────────────────────────────────────────────────────
+    const urlQueue = [...seedUrls]; // start with any user-provided seeds
+    const searchResults = [];
+
+    if (urlQueue.length < capUrls) {
+      try {
+        const searchTool = await this._getSearchTool();
+        for (const q of searchQueries) {
+          if (deadline()) break;
+          try {
+            const sr = await searchTool.execute({ query: q, limit: Math.ceil(capUrls / searchQueries.length) });
+            // SearchWebTool.execute() returns the raw results object; the MCP content-wrapped
+            // shape only appears if a caller (e.g. server.js) wraps it. Handle both.
+            const parsed = sr?.content?.[0]?.text ? JSON.parse(sr.content[0].text) : sr;
+            if (parsed?.results) {
+              for (const r of parsed.results) {
+                if (r.link && !urlQueue.includes(r.link)) urlQueue.push(r.link);
+                searchResults.push({ query: q, title: r.title || '', url: r.link || '', snippet: r.snippet || '' });
+              }
+            }
+          } catch { /* skip failed search */ }
+        }
+      } catch { /* search tool init failed */ }
+    }
+
+    // ── ACT loop ──────────────────────────────────────────────────────────────
+    const evidence = [];
+    let urlsFetched = 0;
+    let step = 0;
+
+    for (const url of urlQueue) {
+      if (step >= capSteps || urlsFetched >= capUrls || deadline()) break;
+      step++;
+      urlsFetched++;
+
+      try {
+        const { textContent, finalUrl } = await fetchAndParse(url, { timeoutMs: 10000 });
+        if (!isRelevant(textContent, prompt)) continue;
+        evidence.push({
+          url: finalUrl,
+          text: truncate(textContent),
+          step
+        });
+      } catch { /* skip unreachable URL */ }
+    }
+
+    // ── SHAPE ─────────────────────────────────────────────────────────────────
+    const combinedText = evidence.map(e => `--- Source: ${e.url} ---\n${e.text}`).join('\n\n');
+
+    if (!combinedText.trim()) {
+      return {
+        success: true,
+        degraded: true,
+        reason: 'No content could be fetched for the given prompt.',
+        search_results: searchResults,
+        evidence: [],
+        answer: null,
+        steps: step,
+        urls_fetched: urlsFetched
+      };
+    }
+
+    // Schema path: use ExtractWithLlm for structured output
+    if (schema && Object.keys(schema).length > 0) {
+      try {
+        const extractWithLlm = await this._getExtractWithLlm();
+        const result = await extractWithLlm.execute({
+          content: combinedText,
+          prompt: `From the following research sources, answer this task and extract structured data:\n${prompt}`,
+          schema,
+          provider: 'auto'
+        });
+        return {
+          success: result.success,
+          answer: result.success ? result.data : null,
+          structured: true,
+          search_results: searchResults,
+          evidence: evidence.map(e => ({ url: e.url })),
+          degraded: !result.success,
+          reason: result.success ? undefined : result.error,
+          steps: step,
+          urls_fetched: urlsFetched
+        };
+      } catch (err) {
+        // Fall through to prose synthesis
+      }
+    }
+
+    // Prose synthesis via SamplingClient
+    let answer = null;
+    let degraded = false;
+    let degradedReason;
+
+    try {
+      const synthesisPrompt =
+        `You are a research assistant. Based on the sources below, answer this task:\n\n` +
+        `Task: ${prompt}\n\n` +
+        `${truncate(combinedText, 12000)}\n\n` +
+        `Provide a clear, concise answer.`;
+
+      const { text } = await this._getSamplingClient().complete(synthesisPrompt, { maxTokens: 1024 });
+      answer = text;
+    } catch (err) {
+      degraded = true;
+      degradedReason = `LLM synthesis unavailable: ${err.message}`;
+      // Return raw evidence so the host LLM can synthesize
+      answer = null;
+    }
+
+    return {
+      success: true,
+      answer,
+      search_results: searchResults,
+      evidence: degraded ? evidence : evidence.map(e => ({ url: e.url })),
+      degraded,
+      reason: degradedReason,
+      steps: step,
+      urls_fetched: urlsFetched
+    };
+  }
+
+  async destroy() {
+    if (this._researchOrchestrator && typeof this._researchOrchestrator.destroy === 'function') {
+      await this._researchOrchestrator.destroy();
+    }
+  }
+}
+
+export default AgentOrchestrator;

package/src/core/AuthManager.js

@@ -538,7 +538,13 @@ class AuthManager {
       extract_with_llm: 5,
 
       // D3.3: Pre-built site templates (1 credit per template scrape)
-      scrape_template: 1
+      scrape_template: 1,
+
+      // Phase D (v4.6.0)
+      // scrape: base 2; projectCost() scales with format count
+      scrape: 2,
+      // agent: base 8; projectCost() scales with maxUrls
+      agent: 8
     };
 
     return costs[tool] || 1;
@@ -585,6 +591,20 @@ class AuthManager {
       case 'extract_with_llm':
         note = 'Includes external LLM API call cost (not billed in credits, billed by your LLM provider).';
         break;
+      case 'scrape': {
+        // Base 2 + 1 per format beyond the first
+        const fmtCount = Array.isArray(params?.formats) ? params.formats.length : 1;
+        projected = Math.max(base, base + Math.max(0, fmtCount - 1));
+        note = `Estimated from ${fmtCount} format(s). json format may incur external LLM cost.`;
+        break;
+      }
+      case 'agent': {
+        const agentUrls = params?.maxUrls || 10;
+        const isPro = params?.model === 'pro';
+        projected = Math.max(base, base + Math.ceil(agentUrls / 5) + (isPro ? 5 : 0));
+        note = `Lower-bound estimate. Scales with maxUrls (${agentUrls}).${isPro ? ' pro model adds deep-research cost.' : ''} External LLM billed separately.`;
+        break;
+      }
       default:
         note = 'Fixed cost per invocation.';
     }

package/src/tools/agent/agent.js

@@ -0,0 +1,71 @@
+/**
+ * agent tool — NL prompt → autonomous search/navigate/extract → answer.
+ *
+ * Wraps AgentOrchestrator for MCP registration.
+ * Mirrors the setMcpServer pattern from extractStructured.js.
+ */
+
+import { z } from 'zod';
+import { AgentOrchestrator } from '../../core/AgentOrchestrator.js';
+import { ElicitationHelper } from '../../core/ElicitationHelper.js';
+import { getToolConfig } from '../../constants/config.js';
+
+export const AgentInputSchema = z.object({
+  prompt: z.string().min(1).max(2000).describe('Natural-language task or question'),
+  urls: z.array(z.string().url()).max(20).optional().describe('Optional seed URLs to include (max 20)'),
+  schema: z.record(z.any()).optional().describe('Optional JSON schema for structured output'),
+  model: z.enum(['default', 'pro']).optional().default('default').describe('"default" = SamplingClient loop; "pro" = full ResearchOrchestrator'),
+  maxSteps: z.number().min(1).max(10).optional().default(5).describe('Max fetch iterations (hard cap: 10)'),
+  maxUrls: z.number().min(1).max(20).optional().default(10).describe('Max URLs to fetch (hard cap: 20)')
+});
+
+export class AgentTool {
+  constructor(options = {}) {
+    this._orchestrator = new AgentOrchestrator({
+      mcpServer: null,
+      searchConfig: getToolConfig('search_web') || {},
+      llmConfig: options.llmConfig || {}
+    });
+    this._elicitation = new ElicitationHelper({});
+  }
+
+  /** Wire MCP server for SamplingClient + Elicitation (called from server.js). */
+  setMcpServer(mcpServer) {
+    this._orchestrator.setMcpServer(mcpServer);
+    this._elicitation = new ElicitationHelper({ mcpServer });
+  }
+
+  async execute(params) {
+    const validated = AgentInputSchema.parse(params);
+
+    // Request confirmation before a pro run (expensive)
+    if (validated.model === 'pro') {
+      const proceed = await this._elicitation.confirm(
+        'agent tool: pro model uses ResearchOrchestrator and may incur significant costs.',
+        { model: 'pro', maxUrls: validated.maxUrls, note: 'External LLM API costs billed separately if keys are set.' }
+      );
+      if (!proceed) {
+        return {
+          success: false,
+          cancelled: true,
+          reason: 'User cancelled pro agent run.'
+        };
+      }
+    }
+
+    return this._orchestrator.run({
+      prompt: validated.prompt,
+      urls: validated.urls,
+      schema: validated.schema,
+      model: validated.model,
+      maxSteps: validated.maxSteps,
+      maxUrls: validated.maxUrls
+    });
+  }
+
+  async destroy() {
+    await this._orchestrator.destroy();
+  }
+}
+
+export default AgentTool;

package/src/tools/basic/extractText.js

@@ -25,7 +25,7 @@ const BLOCK_ELEMENTS = new Set([
  * @param {import('cheerio').CheerioAPI} $ - loaded cheerio instance
  * @returns {string}
  */
-function extractBlockText($) {
+export function extractBlockText($) {
   const parts = [];
 
   function walk(node) {
@@ -52,6 +52,27 @@ function extractBlockText($) {
   return parts.join('').replace(/\n{3,}/g, '\n\n').trim();
 }
 
+/**
+ * Convert raw HTML to GFM markdown using Readability + Turndown.
+ * Accepts the original HTML string and the final URL (needed for Readability).
+ * Returns the markdown string.
+ * @param {string} html - raw HTML
+ * @param {string} pageUrl - URL of the page (used by Readability)
+ * @returns {string}
+ */
+export function readabilityToMarkdown(html, pageUrl) {
+  let articleHtml;
+  try {
+    const dom = new JSDOM(html, { url: pageUrl });
+    const reader = new Readability(dom.window.document);
+    const article = reader.parse();
+    articleHtml = article ? article.content : html;
+  } catch {
+    articleHtml = html;
+  }
+  return htmlToMarkdown(articleHtml);
+}
+
 /**
  * @param {{ url: string, remove_scripts?: boolean, remove_styles?: boolean, output_format?: "text"|"markdown" }} params
  */
@@ -76,16 +97,7 @@ export async function extractTextHandler({ url, remove_scripts, remove_styles, o
 
     if (output_format === 'markdown') {
       // Run Readability first to get main content, then convert to GFM markdown
-      let articleHtml;
-      try {
-        const dom = new JSDOM(html, { url: response.url });
-        const reader = new Readability(dom.window.document);
-        const article = reader.parse();
-        articleHtml = article ? article.content : $.html('body');
-      } catch {
-        articleHtml = $.html('body');
-      }
-      result.markdown = htmlToMarkdown(articleHtml);
+      result.markdown = readabilityToMarkdown(html, response.url);
       result.output_format = 'markdown';
       const plainText = result.markdown.replace(/[#*`_\[\]]/g, '').replace(/\s+/g, ' ').trim();
       result.word_count = plainText.split(/\s+/).filter(w => w.length > 0).length;

package/src/tools/crawl/mapSite.js

@@ -4,6 +4,14 @@ import { DomainFilter } from '../../utils/domainFilter.js';
 import { normalizeUrl, getBaseUrl } from '../../utils/urlNormalizer.js';
 import { CacheManager } from '../../core/cache/CacheManager.js';
 import { SitemapParser } from '../../utils/sitemapParser.js';
+import { ResultRanker } from '../search/ranking/ResultRanker.js';
+
+// Lazy singleton — avoids creating a CacheManager timer per request
+let _ranker = null;
+function getRanker() {
+  if (!_ranker) _ranker = new ResultRanker({ cacheEnabled: false });
+  return _ranker;
+}
 
 const MapSiteSchema = z.object({
   url: z.string().url(),
@@ -18,7 +26,8 @@ const MapSiteSchema = z.object({
     include_patterns: z.array(z.string()).optional().default([]),
     exclude_patterns: z.array(z.string()).optional().default([])
   }).optional(),
-  import_filter_config: z.string().optional() // JSON string of exported config
+  import_filter_config: z.string().optional(), // JSON string of exported config
+  search: z.string().optional() // when set, rank URLs by relevance and emit ranked_urls
 });
 
 export class MapSiteTool {
@@ -120,6 +129,25 @@ export class MapSiteTool {
         filter_stats: domainFilter ? domainFilter.getStats() : null
       };
 
+      // Optional: rank URLs by relevance to a search string
+      if (validated.search) {
+        try {
+          const rankerInput = urlArray.map(url => {
+            let title = url;
+            try {
+              const { pathname } = new URL(url);
+              title = decodeURIComponent(pathname).replace(/[-_/]/g, ' ').trim();
+            } catch { /* keep raw url */ }
+            return { link: url, title, snippet: '' };
+          });
+          const ranked = await getRanker().rankResults(rankerInput, validated.search);
+          result.ranked_urls = ranked.map(r => ({ url: r.link, score: r.finalScore ?? 0 }));
+        } catch {
+          // ranking is best-effort; don't fail the whole call
+          result.ranked_urls = urlArray.map(u => ({ url: u, score: 0 }));
+        }
+      }
+
       // Store in cache before returning
       if (this.cache) {
         const cacheKey = this.cache.generateKey('map_site', { url: validated.url, maxUrls: validated.max_urls });

package/src/tools/scrape/unifiedScrape.js

@@ -0,0 +1,314 @@
+/**
+ * unifiedScrape — single-fetch, multi-format scraping tool.
+ *
+ * One call, one fetch.  formats[] drives what is returned.
+ * Mirrors the output shape of ScrapeWithActionsTool.generateFormats():
+ *   content.html, content.rawHtml, content.text, content.markdown,
+ *   content.links, content.metadata, content.screenshots, content.json
+ *
+ * onlyMainContent maps to Readability boilerplate removal (same as extractContent).
+ * Partial success: per-format warnings[] never fail the whole call.
+ */
+
+import { z } from 'zod';
+import { JSDOM } from 'jsdom';
+import { Readability } from '@mozilla/readability';
+import { fetchAndParse } from '../extract/_fetchAndParse.js';
+import { htmlToMarkdown } from '../../utils/htmlToMarkdown.js';
+import { extractBlockText, readabilityToMarkdown } from '../basic/extractText.js';
+
+// ── Schema ────────────────────────────────────────────────────────────────────
+
+const JsonFormatSchema = z.object({
+  type: z.literal('json'),
+  schema: z.record(z.any()).optional(),
+  prompt: z.string().optional()
+});
+
+const FormatSchema = z.union([
+  z.enum(['markdown', 'html', 'rawHtml', 'text', 'links', 'metadata', 'screenshot']),
+  JsonFormatSchema
+]);
+
+export const UnifiedScrapeSchema = z.object({
+  url: z.string().url(),
+  formats: z.array(FormatSchema).min(1).default(['markdown']),
+  onlyMainContent: z.boolean().optional().default(true),
+  // Pass-through to fetchAndParse
+  timeoutMs: z.number().min(1000).max(60000).optional().default(15000)
+});
+
+// ── Helpers ───────────────────────────────────────────────────────────────────
+
+/**
+ * Extract links from a loaded cheerio $ and the page URL.
+ */
+function extractLinksFromDom($, pageUrl) {
+  const links = [];
+  const seen = new Set();
+  let pageOrigin = '';
+  try { pageOrigin = new URL(pageUrl).origin; } catch { /* ignore */ }
+
+  $('a[href]').each((_, el) => {
+    const href = $(el).attr('href');
+    const text = $(el).text().trim();
+    if (!href) return;
+    try {
+      let absoluteUrl;
+      let isExternal = false;
+      if (href.startsWith('http://') || href.startsWith('https://')) {
+        absoluteUrl = href;
+        isExternal = new URL(href).origin !== pageOrigin;
+      } else if (href.startsWith('#') || href.startsWith('javascript:')) {
+        return;
+      } else {
+        absoluteUrl = new URL(href, pageUrl).toString();
+        isExternal = false;
+      }
+      if (!seen.has(absoluteUrl)) {
+        seen.add(absoluteUrl);
+        links.push({ href: absoluteUrl, text, is_external: isExternal, original_href: href });
+      }
+    } catch { /* skip invalid */ }
+  });
+
+  return {
+    links,
+    total_count: links.length,
+    internal_count: links.filter(l => !l.is_external).length,
+    external_count: links.filter(l => l.is_external).length
+  };
+}
+
+/**
+ * Extract metadata from a loaded cheerio $.
+ */
+function extractMetadataFromDom($, pageUrl) {
+  // JSON-LD
+  const jsonLd = [];
+  $('script[type="application/ld+json"]').each((_, el) => {
+    try { const raw = $(el).html(); if (raw) jsonLd.push(JSON.parse(raw)); } catch { /* skip */ }
+  });
+
+  // Microdata
+  const microdata = [];
+  $('[itemscope]').each((_, el) => {
+    const $el = $(el);
+    const item = { type: $el.attr('itemtype') || null, properties: {} };
+    $el.find('[itemprop]').each((_, prop) => {
+      const $prop = $(prop);
+      const name = $prop.attr('itemprop');
+      if (!name) return;
+      const tag = ($prop.get(0).tagName || '').toLowerCase();
+      let value;
+      if (tag === 'meta') value = $prop.attr('content');
+      else if (tag === 'a' || tag === 'link') value = $prop.attr('href');
+      else if (tag === 'img') value = $prop.attr('src');
+      else if (tag === 'time') value = $prop.attr('datetime') || $prop.text().trim();
+      else value = $prop.text().trim();
+      if (value) {
+        if (!item.properties[name]) item.properties[name] = [];
+        item.properties[name].push(value);
+      }
+    });
+    microdata.push(item);
+  });
+
+  const title =
+    $('meta[property="og:title"]').attr('content') ||
+    $('title').text().trim() ||
+    $('h1').first().text().trim() || '';
+
+  const ogTags = {};
+  $('meta[property^="og:"]').each((_, el) => {
+    const property = $(el).attr('property');
+    const content = $(el).attr('content');
+    if (property && content) ogTags[property.replace('og:', '')] = content;
+  });
+
+  const twitterTags = {};
+  $('meta[name^="twitter:"]').each((_, el) => {
+    const name = $(el).attr('name');
+    const content = $(el).attr('content');
+    if (name && content) twitterTags[name.replace('twitter:', '')] = content;
+  });
+
+  return {
+    title,
+    description: $('meta[name="description"]').attr('content') || $('meta[property="og:description"]').attr('content') || '',
+    keywords: ($('meta[name="keywords"]').attr('content') || '').split(',').map(k => k.trim()).filter(Boolean),
+    canonical_url: $('link[rel="canonical"]').attr('href') || '',
+    author: $('meta[name="author"]').attr('content') || '',
+    robots: $('meta[name="robots"]').attr('content') || '',
+    viewport: $('meta[name="viewport"]').attr('content') || '',
+    og_tags: ogTags,
+    twitter_tags: twitterTags,
+    json_ld: jsonLd,
+    microdata,
+    url: pageUrl
+  };
+}
+
+// ── Tool class ────────────────────────────────────────────────────────────────
+
+export class UnifiedScrapeTool {
+  constructor(options = {}) {
+    this._extractWithLlm = null;
+    this._extractWithLlmConfig = options.llmConfig || {};
+  }
+
+  /** Lazy-load ExtractWithLlm to avoid pulling in heavy deps unless needed. */
+  async _getExtractWithLlm() {
+    if (!this._extractWithLlm) {
+      const { ExtractWithLlm } = await import('../extract/extractWithLlm.js');
+      this._extractWithLlm = new ExtractWithLlm(this._extractWithLlmConfig);
+    }
+    return this._extractWithLlm;
+  }
+
+  /**
+   * Execute a unified scrape.
+   * @param {object} params - UnifiedScrapeSchema-compatible input
+   * @returns {Promise<object>}
+   */
+  async execute(params) {
+    const validated = UnifiedScrapeSchema.parse(params);
+    const { url, formats, onlyMainContent, timeoutMs } = validated;
+
+    // Single fetch
+    let html, $, finalUrl;
+    try {
+      ({ html, $, finalUrl } = await fetchAndParse(url, {
+        timeoutMs,
+        stripTags: [] // we handle boilerplate ourselves
+      }));
+    } catch (err) {
+      throw new Error(`scrape: fetch failed for ${url}: ${err.message}`);
+    }
+
+    // For onlyMainContent: extract main-content html via Readability once
+    let mainHtml = null;
+    function getMainHtml() {
+      if (mainHtml !== null) return mainHtml;
+      try {
+        const dom = new JSDOM(html, { url: finalUrl });
+        const reader = new Readability(dom.window.document);
+        const article = reader.parse();
+        mainHtml = article ? article.content : html;
+      } catch {
+        mainHtml = html;
+      }
+      return mainHtml;
+    }
+
+    const content = {};
+    const warnings = [];
+
+    for (const fmt of formats) {
+      // JSON format object
+      if (fmt && typeof fmt === 'object' && fmt.type === 'json') {
+        try {
+          const extractWithLlm = await this._getExtractWithLlm();
+          const text = onlyMainContent
+            ? htmlToMarkdown(getMainHtml())
+            : $('body').text().replace(/\s+/g, ' ').trim();
+          const result = await extractWithLlm.execute({
+            content: text,
+            prompt: fmt.prompt || 'Extract structured data from this page content.',
+            schema: fmt.schema,
+            provider: 'auto'
+          });
+          content.json = result.success ? result.data : { error: result.error };
+          if (!result.success) {
+            warnings.push(`json: extraction failed — ${result.error}`);
+          }
+        } catch (err) {
+          content.json = { error: err.message };
+          warnings.push(`json: ${err.message}`);
+        }
+        continue;
+      }
+
+      // String formats
+      switch (fmt) {
+        case 'markdown':
+          try {
+            content.markdown = onlyMainContent
+              ? readabilityToMarkdown(html, finalUrl)
+              : htmlToMarkdown($.html('body') || html);
+          } catch (err) {
+            content.markdown = '';
+            warnings.push(`markdown: ${err.message}`);
+          }
+          break;
+
+        case 'html':
+          try {
+            content.html = onlyMainContent ? getMainHtml() : $.html('body') || html;
+          } catch (err) {
+            content.html = '';
+            warnings.push(`html: ${err.message}`);
+          }
+          break;
+
+        case 'rawHtml':
+          content.rawHtml = html;
+          break;
+
+        case 'text':
+          try {
+            if (onlyMainContent) {
+              // Plain text from Readability main content via cheerio
+              const { load } = await import('cheerio');
+              const $main = load(getMainHtml());
+              $main('script, style').remove();
+              content.text = extractBlockText($main);
+            } else {
+              $('script, style').remove();
+              content.text = extractBlockText($);
+            }
+          } catch (err) {
+            content.text = '';
+            warnings.push(`text: ${err.message}`);
+          }
+          break;
+
+        case 'links':
+          try {
+            content.links = extractLinksFromDom($, finalUrl);
+          } catch (err) {
+            content.links = { links: [], total_count: 0, internal_count: 0, external_count: 0 };
+            warnings.push(`links: ${err.message}`);
+          }
+          break;
+
+        case 'metadata':
+          try {
+            content.metadata = extractMetadataFromDom($, finalUrl);
+          } catch (err) {
+            content.metadata = {};
+            warnings.push(`metadata: ${err.message}`);
+          }
+          break;
+
+        case 'screenshot':
+          // Screenshot requires a browser; not available in the basic scrape path.
+          content.screenshots = [];
+          warnings.push('screenshot: browser screenshots are not available in the scrape tool; use scrape_with_actions for screenshots');
+          break;
+
+        default:
+          warnings.push(`unknown format: ${String(fmt)}`);
+      }
+    }
+
+    return {
+      success: true,
+      url: finalUrl,
+      content,
+      warnings: warnings.length > 0 ? warnings : undefined
+    };
+  }
+}
+
+export default UnifiedScrapeTool;