crawlforge-mcp-server 5.0.4 → 5.1.0

package/CLAUDE.md

@@ -60,9 +60,9 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 ## Project Overview
 
-CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 27 web scraping, crawling, and content processing tools (5 inline + 22 advanced).
+CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 28 web scraping, crawling, and content processing tools (5 inline + 23 advanced).
 
-**Current Version:** 5.0.4
+**Current Version:** 5.1.0
 
 ## Development Commands
 
@@ -167,21 +167,24 @@ Tools are organized in subdirectories by category:
 - `crawl/` - crawlDeep, mapSite
 - `extract/` - analyzeContent, extractContent, extractStructured, extractWithLlm, listOllamaModels, processDocument, summarizeContent
 - `research/` - deepResearch
-- `search/` - searchWeb (proxied through CrawlForge.dev API), serpRank (real Google organic rank via DataForSEO; adapter in `search/adapters/dataforseoSearch.js`)
+- `search/` - searchWeb (proxied through CrawlForge.dev API), serpRank (real Google organic rank via DataForSEO; adapter in `search/adapters/dataforseoSearch.js`), redditSearch (Reddit posts/comments/threads via Arctic Shift + PullPush community archives — reddit.com blocks scrapers)
 - `templates/` - ScrapeTemplateTool (10 pre-built site templates, v4.0.0)
 - `tracking/` - trackChanges
 - `llmstxt/` - generateLLMsTxt
 
-### Available MCP Tools (27 total)
+### Available MCP Tools (28 total)
 
 **Basic Tools (server.js inline, 5):**
 fetch_url, extract_text, extract_links, extract_metadata, scrape_structured
 
-**Advanced Tools (22):**
-search_web, serp_rank, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template, scrape, agent
+**Advanced Tools (23):**
+search_web, serp_rank, reddit_search, crawl_deep, map_site, extract_content, process_document, summarize_content, analyze_content, extract_structured, extract_with_llm, list_ollama_models, batch_scrape, scrape_with_actions, deep_research, track_changes, generate_llms_txt, stealth_mode, localization, scrape_template, scrape, agent
 
 **serp_rank (DataForSEO):**
-- `serp_rank` — reports where a target domain ranks in Google's REAL organic results for a keyword (the SERP position Google Custom Search / `search_web` cannot give). Backed by the DataForSEO Google Organic SERP API (Live Advanced, `POST /v3/serp/google/organic/live/advanced`, HTTP Basic auth). Credentials via `DATAFORSEO_LOGIN` / `DATAFORSEO_PASSWORD`, billed to the user's own DataForSEO account (~US$0.002/call), separate from CrawlForge credits. When unconfigured it returns `{ configured:false }` and charges **0** credits; when configured, **Cost: 5**. Never fabricates a rank.
+- `serp_rank` — reports where a target domain ranks in Google's REAL organic results for a keyword (the SERP position Google Custom Search / `search_web` cannot give). Backed by the DataForSEO Google Organic SERP API (Live Advanced, `POST /v3/serp/google/organic/live/advanced`, HTTP Basic auth). Credentials via `DATAFORSEO_LOGIN` / `DATAFORSEO_PASSWORD`, billed to the user's own DataForSEO account (~US$0.002 per 10 results of `depth`, so $0.004 at the default `depth:20` and $0.02 at `depth:100`), separate from CrawlForge credits. When unconfigured it returns `{ configured:false }` and charges **0** credits; when configured, **Cost: 5**. Never fabricates a rank.
+
+**reddit_search (community archives, v5.1.0):**
+- `reddit_search` — searches Reddit posts/comments or reads a full comment thread. reddit.com 403-blocks ALL direct access (fetch, browser-UA, stealth — IP/TLS-reputation based), so this never touches reddit.com; it queries the Arctic Shift archive (`arctic-shift.photon-reddit.com`, near-real-time, comment trees) and PullPush (`api.pullpush.io`, Pushshift-compatible). Routing: Arctic Shift's keyword search REQUIRES a subreddit/author scope (verified live: HTTP 400 without one), so unscoped full-text search goes to PullPush only; scoped searches use Arctic Shift with PullPush as error-only fallback. Modes: `posts` (default), `comments`, `thread` (post + nested comment tree by `link_id`). Free, no credentials; PullPush rate-limits aggressively (~15 req/min). Cost: 2. `REDDIT_SEARCH_TIMEOUT_MS` overrides the 30s default.
 
 **v4.6.0 additions (Phase D):**
 - `scrape` — single fetch + one cheerio load dispatching a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) + `onlyMainContent`; partial-success via per-format `warnings[]`. Cost: 2.

package/README.md

@@ -3,7 +3,7 @@
 </p>
 
 <p align="center">
-  <b>27 web scraping, crawling, deep-research &amp; autonomous-extraction tools for Claude, Cursor &amp; any MCP client.</b><br>
+  <b>28 web scraping, crawling, deep-research &amp; autonomous-extraction tools for Claude, Cursor &amp; any MCP client.</b><br>
   Clean Markdown &amp; structured JSON from any site. Get started with <b>1,000 free credits</b> — no credit card required.
 </p>
 
@@ -35,7 +35,7 @@
 
 ## 🎯 Why CrawlForge?
 
-- **27 MCP-native tools** — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, and more, callable directly from your AI assistant.
+- **28 MCP-native tools** — scraping, crawling, search, real Google SERP rank tracking, deep research, an autonomous `agent`, a unified multi-format `scrape`, document processing, stealth browsing, and more, callable directly from your AI assistant.
 - **Generous free tier** — 1,000 credits to start instantly, no credit card. Credits never expire and roll over month-to-month.
 - **Local-LLM by default** — `extract_with_llm` runs against a local **Ollama** model out of the box: no LLM API key, no per-token cost, and your data never leaves your machine. Cloud (OpenAI/Anthropic) is opt-in.
 - **LLM-ready output** — clean Markdown, structured JSON (schema-driven), screenshots, links, and metadata from a single fetch.
@@ -47,7 +47,7 @@
 
 | | **CrawlForge MCP** | Firecrawl | Raw scraping API |
 |---|:---:|:---:|:---:|
-| Native MCP server | ✅ 27 tools | ✅ | ❌ |
+| Native MCP server | ✅ 28 tools | ✅ | ❌ |
 | Free tier | ✅ 1,000 credits, rollover | Limited | Varies |
 | Self-hosted / local LLM extraction (Ollama) | ✅ default, $0/token | ❌ | ❌ |
 | Autonomous agent (no URLs needed) | ✅ `agent` | ✅ | ❌ |
@@ -146,6 +146,21 @@ The setup wizard automatically configures Cursor by adding to `~/.cursor/mcp.jso
 Restart Cursor to activate.
 </details>
 
+<details>
+<summary>🔁 For n8n (workflow automation)</summary>
+
+n8n's built-in **MCP Client Tool** node connects over Streamable HTTP (works on n8n Cloud and self-hosted). Run the server in HTTP mode:
+
+```bash
+export CRAWLFORGE_API_KEY=your_api_key
+npm run start:http    # Streamable HTTP endpoint at http://localhost:10000/mcp
+```
+
+Then point the MCP Client Tool node at `http://<host>:10000/mcp` with transport **HTTP Streamable** and a **Bearer** credential set to the same API key. On self-hosted n8n you can instead use the community `n8n-nodes-mcp` node over STDIO (`npx -y crawlforge-mcp-server`).
+
+Full guide: [docs/n8n-integration.md](docs/n8n-integration.md)
+</details>
+
 > **Which launch command?** `npx -y crawlforge-mcp-server` needs no global install and always runs the published version (recommended for Claude Desktop). For a global install (`npm i -g crawlforge-mcp-server`), use the dedicated `crawlforge-mcp` bin — it resolves on your `PATH`, so it survives Node/nvm version switches. The bare `crawlforge` command still launches the server when an MCP client spawns it over stdio (backward compatibility for configs created before v4.2.5); interactively it's the CLI — run `crawlforge mcp` to start the server by hand.
 
 ## 📊 Available Tools
@@ -169,6 +184,7 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `map_site` | 2 | Discover and map website structure (optional `search=` ranks the discovered URLs) |
 | `process_document` | 2 | Multi-format document processing |
 | `localization` | 2 | Multi-language and geo-location management |
+| `reddit_search` | 2 | Search Reddit posts/comments or read a full thread — reddit.com blocks direct scraping, so this queries the Arctic Shift + PullPush community archives (free, no Reddit credentials) |
 | `track_changes` | 3 | Monitor content changes over time |
 | `analyze_content` | 3 | Comprehensive content analysis |
 | `extract_structured` | 3 | LLM-powered schema-driven extraction (your own LLM key or local Ollama) |
@@ -200,7 +216,7 @@ For the full canonical capabilities reference (all tools, CLI commands, stealth
 | **Business** ($399) | 250,000 | Large scale operations |
 
 **All plans include:**
-- Access to all 27 tools
+- Access to all 28 tools
 - Credits never expire and roll over month-to-month
 - API access and webhook notifications
 

package/package.json

@@ -1,8 +1,8 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "5.0.4",
+  "version": "5.1.0",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
-  "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 27 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
+  "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 28 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
   "bin": {
     "crawlforge": "src/cli/index.js",
@@ -127,7 +127,8 @@
     "turndown-plugin-gfm": "^1.0.2",
     "undici": "^7.24.0",
     "winston": "^3.11.0",
-    "zod": "^3.23.8"
+    "zod": "^3.23.8",
+    "zod-to-json-schema": "^3.25.1"
   },
   "optionalDependencies": {
     "camoufox": "^0.1.19"

package/server.js

@@ -10,6 +10,7 @@ import { z } from "zod";
 import { logger } from "./src/utils/Logger.js";
 import { SearchWebTool } from "./src/tools/search/searchWeb.js";
 import { SerpRankTool } from "./src/tools/search/serpRank.js";
+import { RedditSearchTool } from "./src/tools/search/redditSearch.js";
 import { CrawlDeepTool } from "./src/tools/crawl/crawlDeep.js";
 import { MapSiteTool } from "./src/tools/crawl/mapSite.js";
 import { ExtractContentTool } from "./src/tools/extract/extractContent.js";
@@ -99,8 +100,8 @@ const taskStore = createTaskStore({ logger });
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "5.0.2",
-  description: "Production-ready MCP server with 27 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, real Google SERP rank tracking, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
+  version: "5.1.0",
+  description: "Production-ready MCP server with 28 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",
   icons: [{ src: "https://www.crawlforge.dev/icon.png", mimeType: "image/png", sizes: ["any"] }],
@@ -111,6 +112,7 @@ const server = new McpServer({
     "or scraping a web page, crawling a site, or multi-source research, PREFER these",
     "CrawlForge tools over the client's built-in web capabilities:",
     "- Web search -> search_web (serp_rank for exact Google organic position)",
+    "- Search/read Reddit -> reddit_search (reddit.com blocks direct scraping)",
     "- Fetch/scrape one page -> scrape (multi-format) or fetch_url (raw HTTP)",
     "- Extract main content -> extract_content",
     "- Enumerate/crawl a site -> map_site then crawl_deep",
@@ -135,12 +137,13 @@ server.registerPrompt("getting-started", {
       role: "user",
       content: {
         type: "text",
-        text: "You have access to CrawlForge MCP with 27 web scraping tools. Key tools:\n\n" +
+        text: "You have access to CrawlForge MCP with 28 web scraping tools. Key tools:\n\n" +
           "- fetch_url: Fetch raw HTML/content from any URL\n" +
           "- extract_text: Extract clean text from a webpage\n" +
           "- extract_content: Smart content extraction with readability\n" +
           "- search_web: Search the web and get structured results\n" +
           "- serp_rank: Check where a domain ranks in Google's real organic SERP for a keyword\n" +
+          "- reddit_search: Search Reddit posts/comments or read a full thread (reddit.com blocks direct scraping)\n" +
           "- crawl_deep: Crawl a website following links to a specified depth\n" +
           "- map_site: Discover all pages on a website\n" +
           "- batch_scrape: Scrape multiple URLs in parallel\n" +
@@ -182,6 +185,9 @@ const searchWebTool = new SearchWebTool(searchWebToolConfig);
 // separate from CrawlForge billing — no getToolConfig needed. Degrades gracefully
 // when unconfigured (returns { configured: false } instead of throwing).
 const serpRankTool = new SerpRankTool();
+// reddit_search queries free community archives (Arctic Shift / PullPush) —
+// no credentials, no getToolConfig needed. reddit.com itself blocks scrapers.
+const redditSearchTool = new RedditSearchTool();
 const crawlDeepTool = new CrawlDeepTool(getToolConfig('crawl_deep'));
 const mapSiteTool = new MapSiteTool(getToolConfig('map_site'));
 const extractContentTool = new ExtractContentTool();
@@ -427,7 +433,7 @@ registerToolIfEnabled("serp_rank", {
     location_code: z.number().optional().describe("Numeric DataForSEO location code (overrides location_name)"),
     language_code: z.string().optional().describe("Language code (e.g. 'en')"),
     device: z.enum(["desktop", "mobile"]).optional().describe("Device to emulate"),
-    depth: z.number().min(10).max(200).optional().describe("How many results to scan, 10-200 (100 = 1 page of cost)")
+    depth: z.number().min(10).max(200).optional().describe("How many results to scan, 10-200 (default 20; DataForSEO bills ~$0.002 per 10 and gets slower the deeper it goes)")
   },
   outputSchema: OUTPUT_SCHEMAS.serp_rank
 }, withAuth("serp_rank", async ({ keyword, target, location_name, location_code, language_code, device, depth }) => {
@@ -442,6 +448,32 @@ registerToolIfEnabled("serp_rank", {
   }
 }));
 
+// Tool: reddit_search — search Reddit posts/comments or read a full thread (via community archives)
+registerToolIfEnabled("reddit_search", {
+  description: "Use this to search Reddit posts or comments, or read a full comment thread — reddit.com blocks direct scraping, so this queries the Arctic Shift and PullPush community archives instead (free, no Reddit credentials). Modes: 'posts' (default) and 'comments' search; 'thread' returns a post plus its nested comment tree by link_id. Keyword search across ALL of Reddit routes to PullPush; subreddit/author-scoped searches use Arctic Shift (near-real-time) with PullPush fallback. Example: reddit_search({query: \"best mechanical keyboard\", subreddit: \"MechanicalKeyboards\", limit: 10})",
+  annotations: { title: "Reddit Search", readOnlyHint: true, destructiveHint: false, idempotentHint: true, openWorldHint: true },
+  inputSchema: {
+    query: z.string().optional().describe("Keyword search. Posts: matches title+selftext; comments: matches body. Supports \"quoted phrases\", OR, -exclusion"),
+    subreddit: z.string().optional().describe("Limit to one subreddit (with or without the r/ prefix)"),
+    author: z.string().optional().describe("Limit to one author (with or without the u/ prefix)"),
+    mode: z.enum(["posts", "comments", "thread"]).optional().describe("What to search: posts (default), comments, or thread (full comment tree — requires link_id)"),
+    link_id: z.string().optional().describe("Post ID (e.g. '1twm1zh' or 't3_1twm1zh') — required for thread mode, optional filter for comments mode"),
+    after: z.string().optional().describe("Only content posted after this date — ISO 8601, epoch seconds, or an offset like '7d'"),
+    before: z.string().optional().describe("Only content posted before this date — same formats as after"),
+    limit: z.number().min(1).max(100).optional().describe("Max results (default 25; thread mode: max comments returned)"),
+    sort: z.enum(["asc", "desc"]).optional().describe("Sort by post date (default desc = newest first)"),
+    source: z.enum(["auto", "arctic_shift", "pullpush"]).optional().describe("Force a specific archive backend (default auto routes + falls back)")
+  },
+  outputSchema: OUTPUT_SCHEMAS.reddit_search
+}, withAuth("reddit_search", async ({ query, subreddit, author, mode, link_id, after, before, limit, sort, source }) => {
+  try {
+    const result = await redditSearchTool.execute({ query, subreddit, author, mode, link_id, after, before, limit, sort, source });
+    return dualOutput(result);
+  } catch (error) {
+    return { content: [{ type: "text", text: `Reddit search failed: ${error.message}` }], isError: true };
+  }
+}));
+
 // Tool: crawl_deep (async task pattern — Phase 6; taskSupport:'optional' keeps sync callers working)
 if (toolFilter.isEnabled("crawl_deep")) {
   server.experimental.tasks.registerToolTask("crawl_deep", {
@@ -1441,7 +1473,7 @@ async function runServer() {
 
   const allTools = [
     "fetch_url", "extract_text", "extract_links", "extract_metadata", "scrape_structured",
-    "search_web", "serp_rank", "crawl_deep", "map_site",
+    "search_web", "serp_rank", "reddit_search", "crawl_deep", "map_site",
     "extract_content", "process_document", "summarize_content", "analyze_content",
     "batch_scrape", "get_batch_results", "scrape_with_actions",
     "deep_research", "track_changes", "generate_llms_txt",

package/src/core/AuthManager.js

@@ -560,6 +560,7 @@ class AuthManager {
       process_document: 2,
       localization: 2,
       scrape: 2,
+      reddit_search: 2, // free community archives (Arctic Shift / PullPush), no external billing
 
       // 3 credits
       track_changes: 3,
@@ -644,7 +645,7 @@ class AuthManager {
       case 'serp_rank':
         note = projected === 0
           ? 'DataForSEO not configured — no-op, no credits charged. Set DATAFORSEO_LOGIN/PASSWORD to enable.'
-          : 'DataForSEO SERP API (~US$0.002/call) billed to your own DataForSEO account, separate from the credit cost.';
+          : 'DataForSEO SERP API (~US$0.002 per 10 results of depth — $0.004 at the default depth 20) billed to your own DataForSEO account, separate from the credit cost.';
         break;
       case 'scrape': {
         projected = base;

package/src/schemas/toolOutputSchemas.js

@@ -148,6 +148,55 @@ const serpRankShape = {
   _cost: costShape
 };
 
+// ── reddit_search ───────────────────────────────────────────────────────────
+
+const redditPostShape = z.object({
+  id: z.string().nullable().optional(),
+  title: z.string().nullable().optional(),
+  author: z.string().nullable().optional(),
+  subreddit: z.string().nullable().optional(),
+  created_utc: z.number().nullable().optional(),
+  created_iso: z.string().nullable().optional(),
+  score: z.number().nullable().optional(),
+  num_comments: z.number().nullable().optional(),
+  selftext: z.string().nullable().optional(),
+  selftext_truncated: z.boolean().optional(),
+  url: z.string().nullable().optional(),
+  permalink: z.string().nullable().optional().describe('Full reddit.com URL of the post')
+}).passthrough();
+
+const redditCommentShape = z.object({
+  id: z.string().nullable().optional(),
+  author: z.string().nullable().optional(),
+  subreddit: z.string().nullable().optional(),
+  created_utc: z.number().nullable().optional(),
+  created_iso: z.string().nullable().optional(),
+  score: z.number().nullable().optional(),
+  body: z.string().nullable().optional(),
+  body_truncated: z.boolean().optional(),
+  link_id: z.string().nullable().optional(),
+  parent_id: z.string().nullable().optional(),
+  permalink: z.string().nullable().optional()
+}).passthrough();
+
+const redditSearchShape = {
+  source: z.enum(['arctic_shift', 'pullpush']).optional().describe('Which community archive served this result'),
+  mode: z.string().optional(),
+  query: z.string().nullable().optional(),
+  subreddit: z.string().nullable().optional(),
+  author: z.string().nullable().optional(),
+  link_id: z.string().optional().describe('Present in thread mode'),
+  count: z.number().optional(),
+  results: z.array(z.union([redditPostShape, redditCommentShape])).optional().describe('posts/comments modes'),
+  post: redditPostShape.nullable().optional().describe('thread mode: the post itself'),
+  comments: z.array(z.unknown()).optional().describe('thread mode: nested comment tree ({...comment, replies:[...]}); collapsed branches appear as {more_count, more_ids}'),
+  comment_count: z.number().optional(),
+  fallback_used: z.string().optional().describe('Present when the primary archive failed and the fallback served the result'),
+  notes: z.array(z.string()).optional().describe('Data-provenance caveats (archive freshness, coverage gaps)'),
+  checkedAt: z.string().optional(),
+  _cost: costShape
+};
+
 // ── search_web ──────────────────────────────────────────────────────────────
 
 const searchWebResultShape = z.object({
@@ -261,6 +310,7 @@ export const OUTPUT_SCHEMAS = {
   scrape: scrapeShape,
   map_site: mapSiteShape,
   serp_rank: serpRankShape,
+  reddit_search: redditSearchShape,
   search_web: searchWebShape,
   extract_structured: extractStructuredShape,
   crawl_deep: crawlDeepShape

package/src/server/transports/streamableHttp.js

@@ -30,10 +30,50 @@ import { McpServer } from '@modelcontextprotocol/sdk/server/mcp.js';
 import { createServer } from 'node:http';
 import { randomUUID } from 'node:crypto';
 import { readFileSync } from 'node:fs';
+import { z } from 'zod';
+import { zodToJsonSchema } from 'zod-to-json-schema';
 
 const pkg = JSON.parse(readFileSync(new URL('../../../package.json', import.meta.url), 'utf8'));
 const SERVER_VERSION = pkg.version;
 
+/**
+ * Build the `tools` array for the Smithery static server card, straight from
+ * the live tool registry.
+ *
+ * Smithery scans a published server to populate its listing, but our /mcp
+ * endpoint 401s without a key, so the scan cannot enumerate anything. Their
+ * documented fallback is a static server card carrying the metadata — and a
+ * card with no `tools` leaves the listing showing whatever was typed in by
+ * hand at publish time, which is how it goes stale.
+ *
+ * Deriving it here means the card tracks the registry on every release instead
+ * of drifting. Tool `inputSchema`s are registered as ZodRawShapes, so wrap
+ * before converting; a tool whose schema will not convert is still listed,
+ * with an open object schema, rather than dropped.
+ */
+function buildToolCards(server) {
+  const registered = server?._registeredTools ?? {};
+  return Object.entries(registered)
+    .filter(([, tool]) => tool?.enabled !== false)
+    .map(([name, tool]) => {
+      let inputSchema = { type: 'object', properties: {} };
+      try {
+        const shape = tool?.inputSchema;
+        if (shape) {
+          const zodObject = typeof shape?.safeParse === 'function' ? shape : z.object(shape);
+          const converted = zodToJsonSchema(zodObject, { $refStrategy: 'none' });
+          delete converted.$schema;
+          inputSchema = converted;
+        }
+      } catch {
+        // Keep the tool visible with an open schema rather than hiding it.
+      }
+      const card = { name, description: tool?.description ?? tool?.annotations?.title ?? '', inputSchema };
+      if (tool?.annotations) card.annotations = tool.annotations;
+      return card;
+    });
+}
+
 /**
  * The MCP SDK's Protocol.connect() allows at most one active transport per
  * Server/McpServer instance (it throws 'Already connected to a transport'
@@ -163,6 +203,10 @@ export async function connectStreamableHttp(server, authManager, logger, options
           icon: 'https://www.crawlforge.dev/icon.png'
         },
         transport: { type: 'streamable-http', url: '/mcp' },
+        authentication: { required: true, schemes: ['apiKey'] },
+        tools: buildToolCards(server),
+        resources: [],
+        prompts: [],
         configSchema: {
           type: 'object',
           properties: {

package/src/tools/search/adapters/dataforseoSearch.js

@@ -11,9 +11,13 @@
  * Endpoint (Live Advanced, synchronous — one request, one response):
  *   POST https://api.dataforseo.com/v3/serp/google/organic/live/advanced
  *
- * Cost: ~US$0.002 per 100 results (depth) on Live Advanced. For high-volume
- * scheduled tracking, DataForSEO's task-based "Standard" queue (task_post →
- * tasks_ready → task_get) is cheaper; swap the endpoint + poll if cost matters.
+ * Cost: US$0.002 per 10 results of `depth` on Live Advanced — measured live,
+ * not estimated: depth 10 bills $0.002, the depth 20 default bills $0.004, and
+ * depth 100 bills $0.02. Deeper scans are also slower (see the timeout note
+ * below), so raise `depth` only when a rank below the default is worth paying
+ * for. For high-volume scheduled tracking, DataForSEO's task-based "Standard"
+ * queue (task_post → tasks_ready → task_get) is cheaper; swap the endpoint +
+ * poll if cost matters.
  */
 
 export class DataForSEOSearchAdapter {
@@ -25,9 +29,16 @@ export class DataForSEOSearchAdapter {
     this.login = login;
     this.password = password;
     this.apiBaseUrl = options.apiBaseUrl || 'https://api.dataforseo.com';
-    // Live Advanced is synchronous and usually answers in a few seconds; cap it
-    // so a hung connection can't wedge the tool. Overridable for tests/self-host.
-    this.timeoutMs = options.timeoutMs ?? 30000;
+    // Live Advanced is synchronous and runs a real-time Google scrape, so its
+    // latency swings widely with their capacity AND with `depth`. Measured on
+    // one account in a single session: ~13-15s at depth 10, and 30s / 45s /
+    // over 60s (twice) for the SAME depth-100 request. A killed request is
+    // still billed — DataForSEO has already run the scrape — so the cap exists
+    // only to stop a hung connection wedging the tool, and sits well above the
+    // slow end rather than through the middle of it.
+    // Overridable via DATAFORSEO_TIMEOUT_MS, or directly for tests/self-host.
+    this.timeoutMs =
+      options.timeoutMs ?? (Number(process.env.DATAFORSEO_TIMEOUT_MS) || 120000);
     // HTTP Basic auth header, computed once.
     this.authHeader = 'Basic ' + Buffer.from(`${login}:${password}`).toString('base64');
   }
@@ -40,7 +51,7 @@ export class DataForSEOSearchAdapter {
    * @param {number} [params.locationCode] - Numeric DataForSEO location code (overrides locationName)
    * @param {string} [params.languageCode='en'] - Language code
    * @param {('desktop'|'mobile')} [params.device='desktop'] - Device to emulate
-   * @param {number} [params.depth=100] - How many results to scan (100 = one page of cost)
+   * @param {number} [params.depth=20] - How many results to scan (billed per 10)
    * @returns {Promise<{items: Array<Object>, meta: Object}>} Normalized organic results + metadata
    */
   async search(params) {
@@ -50,7 +61,7 @@ export class DataForSEOSearchAdapter {
       locationCode,
       languageCode = 'en',
       device = 'desktop',
-      depth = 100,
+      depth = 20,
     } = params;
 
     if (!keyword) {

package/src/tools/search/redditSearch.js

@@ -0,0 +1,350 @@
+/**
+ * reddit_search tool
+ *
+ * Searches Reddit posts and comments, or reads a full comment thread.
+ *
+ * reddit.com hard-blocks non-browser clients (403 on fetch, stealth browsers
+ * included — the block is IP/TLS-reputation based), so this tool never touches
+ * reddit.com. It queries the two community-run Reddit archives instead:
+ *
+ * - Arctic Shift (https://arctic-shift.photon-reddit.com) — near-real-time
+ *   ingestion, comment trees, richer endpoints. Constraint from its API docs:
+ *   keyword search (`query`/`body`) only works when scoped to a subreddit,
+ *   author, or post — NOT across all of Reddit.
+ * - PullPush (https://api.pullpush.io) — Pushshift-compatible, supports
+ *   cross-subreddit full-text search, but has known post-2023 archive gaps
+ *   and recurring outages.
+ *
+ * Routing: scoped searches go to Arctic Shift (fresher) with PullPush as an
+ * error-only fallback; unscoped keyword searches can only go to PullPush.
+ * Both services are free and need no credentials.
+ */
+
+import { z } from 'zod';
+
+const ARCTIC_SHIFT_BASE = 'https://arctic-shift.photon-reddit.com';
+const PULLPUSH_BASE = 'https://api.pullpush.io';
+
+/**
+ * Identify ourselves. Verified live: Arctic Shift throttles UA-less clients
+ * into a shared bucket (422 "Timeout. Maybe slow down a bit" while curl got
+ * 200 for the same URL); with a descriptive UA it answers instantly.
+ */
+const USER_AGENT = 'CrawlForge-MCP/5.1.0 (+https://www.crawlforge.dev)';
+
+/** Cap selftext/body length so a 100-result payload stays LLM-friendly. */
+const TEXT_MAX = 2000;
+
+const RedditSearchSchema = z.object({
+  query: z.string().min(1).optional(),
+  subreddit: z.string().min(1).optional(),
+  author: z.string().min(1).optional(),
+  mode: z.enum(['posts', 'comments', 'thread']).optional().default('posts'),
+  link_id: z.string().min(1).optional(), // post ID — required for thread mode
+  after: z.string().min(1).optional(),
+  before: z.string().min(1).optional(),
+  limit: z.number().int().min(1).max(100).optional().default(25),
+  sort: z.enum(['asc', 'desc']).optional().default('desc'),
+  source: z.enum(['auto', 'arctic_shift', 'pullpush']).optional().default('auto'),
+});
+
+/** "t3_abc123" / "t1_abc123" → "abc123" (both archives accept bare IDs). */
+function stripIdPrefix(id) {
+  return String(id).replace(/^t[13]_/, '');
+}
+
+/** "r/Foo" → "Foo", "u/bar" → "bar" (Arctic Shift ignores prefixes; PullPush doesn't). */
+function stripNamePrefix(name) {
+  return name == null ? name : String(name).replace(/^[ru]\//, '');
+}
+
+/**
+ * PullPush (Pushshift schema) wants epoch seconds for after/before. Pass
+ * through epoch and offset forms ("7d"); convert ISO dates. Arctic Shift
+ * accepts all of these natively, so this is only used on the PullPush path.
+ */
+function toEpochSeconds(value) {
+  if (/^\d+$/.test(value) || /^\d+[a-z]+$/i.test(value)) return value;
+  const ms = Date.parse(value);
+  if (Number.isNaN(ms)) {
+    throw new Error(`Unparseable date "${value}" — use ISO 8601, epoch seconds, or an offset like "7d"`);
+  }
+  return String(Math.floor(ms / 1000));
+}
+
+function truncate(text) {
+  if (typeof text !== 'string' || text.length <= TEXT_MAX) {
+    return { text: text ?? null, truncated: false };
+  }
+  return { text: text.slice(0, TEXT_MAX), truncated: true };
+}
+
+function toIso(epochSeconds) {
+  return typeof epochSeconds === 'number'
+    ? new Date(epochSeconds * 1000).toISOString()
+    : null;
+}
+
+/** Both archives store raw Reddit post objects — reduce to the fields that matter. */
+function normalizePost(raw) {
+  const { text: selftext, truncated } = truncate(raw.selftext);
+  return {
+    id: raw.id ?? null,
+    title: raw.title ?? null,
+    author: raw.author ?? null,
+    subreddit: raw.subreddit ?? null,
+    created_utc: raw.created_utc ?? null,
+    created_iso: toIso(raw.created_utc),
+    score: raw.score ?? null,
+    num_comments: raw.num_comments ?? null,
+    upvote_ratio: raw.upvote_ratio ?? null,
+    flair: raw.link_flair_text ?? null,
+    over_18: raw.over_18 ?? null,
+    selftext,
+    selftext_truncated: truncated,
+    url: raw.url ?? null,
+    permalink: raw.permalink ? `https://www.reddit.com${raw.permalink}` : null,
+  };
+}
+
+function normalizeComment(raw) {
+  const { text: body, truncated } = truncate(raw.body);
+  return {
+    id: raw.id ?? null,
+    author: raw.author ?? null,
+    subreddit: raw.subreddit ?? null,
+    created_utc: raw.created_utc ?? null,
+    created_iso: toIso(raw.created_utc),
+    score: raw.score ?? null,
+    body,
+    body_truncated: truncated,
+    link_id: raw.link_id ? stripIdPrefix(raw.link_id) : null,
+    parent_id: raw.parent_id ?? null,
+    permalink: raw.permalink ? `https://www.reddit.com${raw.permalink}` : null,
+  };
+}
+
+/**
+ * Arctic Shift's /api/comments/tree returns Reddit-API-style nodes:
+ * {kind:"t1", data:{...comment, replies:{kind:"Listing", data:{children:[...]}}}}
+ * and {kind:"more", data:{count, children:[ids]}} for collapsed branches.
+ * Flatten to a nested {..comment, replies:[...]} shape.
+ */
+function normalizeTreeNodes(nodes) {
+  if (!Array.isArray(nodes)) return [];
+  return nodes.map((node) => {
+    if (node?.kind === 'more') {
+      return { more_count: node.data?.count ?? null, more_ids: node.data?.children ?? [] };
+    }
+    const data = node?.data ?? {};
+    const children = data.replies?.data?.children;
+    return { ...normalizeComment(data), replies: normalizeTreeNodes(children) };
+  });
+}
+
+export class RedditSearchTool {
+  constructor(options = {}) {
+    // Overridable for tests / self-hosted mirrors.
+    this.arcticBaseUrl = options.arcticBaseUrl || ARCTIC_SHIFT_BASE;
+    this.pullpushBaseUrl = options.pullpushBaseUrl || PULLPUSH_BASE;
+    // Community services with no SLA — generous but bounded.
+    this.timeoutMs = options.timeoutMs ?? (Number(process.env.REDDIT_SEARCH_TIMEOUT_MS) || 30000);
+    // Pause before the single retry of a transient throttle response.
+    this.retryDelayMs = options.retryDelayMs ?? 3000;
+  }
+
+  async execute(params) {
+    const v = RedditSearchSchema.parse(params);
+
+    // Cross-field validation zod can't express per-mode.
+    if (v.mode === 'thread') {
+      if (!v.link_id) throw new Error('thread mode requires link_id (the post ID, e.g. "1twm1zh" or "t3_1twm1zh")');
+    } else if (!v.query && !v.subreddit && !v.author && !(v.mode === 'comments' && v.link_id)) {
+      throw new Error(`${v.mode} mode requires at least one of: query, subreddit, author${v.mode === 'comments' ? ', link_id' : ''}`);
+    }
+
+    const subreddit = stripNamePrefix(v.subreddit);
+    const author = stripNamePrefix(v.author);
+    // Arctic Shift keyword search must be scoped (its documented constraint).
+    const scoped = Boolean(subreddit || author || (v.mode === 'comments' && v.link_id));
+    const arcticPossible = v.mode === 'thread' || !v.query || scoped;
+
+    let order; // backends to try, in order
+    if (v.source === 'arctic_shift') {
+      if (!arcticPossible) {
+        throw new Error('Arctic Shift cannot keyword-search across all of Reddit — add a subreddit or author scope, or use source:"pullpush"');
+      }
+      order = ['arctic_shift'];
+    } else if (v.source === 'pullpush') {
+      if (v.mode === 'thread') throw new Error('thread mode requires Arctic Shift (source:"pullpush" only supports posts/comments search)');
+      order = ['pullpush'];
+    } else {
+      order = v.mode === 'thread' ? ['arctic_shift']
+        : arcticPossible ? ['arctic_shift', 'pullpush']
+        : ['pullpush'];
+    }
+
+    const errors = [];
+    for (const source of order) {
+      try {
+        const result = source === 'arctic_shift'
+          ? await this.#searchArcticShift(v, { subreddit, author })
+          : await this.#searchPullPush(v, { subreddit, author });
+        if (errors.length > 0) result.fallback_used = `primary source failed (${errors[0]}), fell back to ${source}`;
+        return result;
+      } catch (error) {
+        errors.push(`${source}: ${error.message}`);
+      }
+    }
+    // Unscoped keyword searches have no Arctic Shift fallback (it requires a
+    // scope — verified live: HTTP 400 without one), so point at the fix.
+    const hint = order.length === 1 && order[0] === 'pullpush' && v.source === 'auto'
+      ? ' Tip: add a subreddit or author filter to route to the more reliable Arctic Shift archive.'
+      : '';
+    throw new Error(`All Reddit archive sources failed — ${errors.join('; ')}.${hint}`);
+  }
+
+  async #searchArcticShift(v, { subreddit, author }) {
+    const notes = [
+      'Data from the Arctic Shift community archive (arctic-shift.photon-reddit.com), not reddit.com (which blocks scrapers).',
+      'Scores and comment counts of content less than ~36h old may read 0/1 — the archive captures content the moment it is posted.',
+    ];
+
+    if (v.mode === 'thread') {
+      const id = stripIdPrefix(v.link_id);
+      const postData = await this.#get(`${this.arcticBaseUrl}/api/posts/ids`, { ids: id });
+      const post = postData.data?.[0] ? normalizePost(postData.data[0]) : null;
+      const treeData = await this.#get(`${this.arcticBaseUrl}/api/comments/tree`, {
+        link_id: `t3_${id}`,
+        limit: String(v.limit),
+      });
+      const comments = normalizeTreeNodes(treeData.data);
+      return {
+        source: 'arctic_shift', mode: 'thread', link_id: id,
+        post, comments, comment_count: comments.length,
+        notes, checkedAt: new Date().toISOString(),
+      };
+    }
+
+    const query = {
+      limit: String(v.limit),
+      sort: v.sort,
+      ...(subreddit && { subreddit }),
+      ...(author && { author }),
+      ...(v.after && { after: v.after }),
+      ...(v.before && { before: v.before }),
+    };
+    let path;
+    if (v.mode === 'posts') {
+      path = '/api/posts/search';
+      if (v.query) query.query = v.query; // searches title + selftext
+    } else {
+      path = '/api/comments/search';
+      if (v.query) query.body = v.query;
+      if (v.link_id) query.link_id = stripIdPrefix(v.link_id);
+    }
+    const data = await this.#get(`${this.arcticBaseUrl}${path}`, query);
+    const rows = Array.isArray(data.data) ? data.data : [];
+    const results = v.mode === 'posts' ? rows.map(normalizePost) : rows.map(normalizeComment);
+    return {
+      source: 'arctic_shift', mode: v.mode,
+      query: v.query ?? null, subreddit: subreddit ?? null, author: author ?? null,
+      count: results.length, results,
+      notes, checkedAt: new Date().toISOString(),
+    };
+  }
+
+  async #searchPullPush(v, { subreddit, author }) {
+    if (v.mode === 'thread') throw new Error('thread mode is Arctic Shift only');
+    const notes = [
+      'Data from the PullPush community archive (api.pullpush.io), not reddit.com (which blocks scrapers).',
+      'PullPush has known gaps in its post-2023 archive — an empty result does not prove the content does not exist.',
+    ];
+    const query = {
+      size: String(v.limit),
+      sort: v.sort,
+      sort_type: 'created_utc',
+      ...(v.query && { q: v.query }),
+      ...(subreddit && { subreddit }),
+      ...(author && { author }),
+      ...(v.after && { after: toEpochSeconds(v.after) }),
+      ...(v.before && { before: toEpochSeconds(v.before) }),
+    };
+    if (v.mode === 'comments' && v.link_id) query.link_id = stripIdPrefix(v.link_id);
+    const path = v.mode === 'posts' ? '/reddit/search/submission/' : '/reddit/search/comment/';
+    const data = await this.#get(`${this.pullpushBaseUrl}${path}`, query);
+    if (data.error) throw new Error(`PullPush error: ${data.error}`);
+    const rows = Array.isArray(data.data) ? data.data : [];
+    const results = v.mode === 'posts' ? rows.map(normalizePost) : rows.map(normalizeComment);
+    return {
+      source: 'pullpush', mode: v.mode,
+      query: v.query ?? null, subreddit: subreddit ?? null, author: author ?? null,
+      count: results.length, results,
+      notes, checkedAt: new Date().toISOString(),
+    };
+  }
+
+  /**
+   * GET with one bounded retry: both archives shed load transiently (PullPush
+   * 429s at ~15 req/min; Arctic Shift answers 422 "Timeout. Maybe slow down a
+   * bit" under per-IP pressure — observed live) and usually recover in seconds.
+   */
+  async #get(base, queryParams) {
+    const url = `${base}?${new URLSearchParams(queryParams)}`;
+    let lastError;
+    for (let attempt = 0; attempt < 2; attempt++) {
+      if (attempt > 0) await new Promise((resolve) => setTimeout(resolve, this.retryDelayMs));
+      try {
+        return await this.#getOnce(url);
+      } catch (error) {
+        lastError = error;
+        if (!error.retryable) throw error;
+      }
+    }
+    throw lastError;
+  }
+
+  async #getOnce(url) {
+    let response;
+    try {
+      response = await fetch(url, {
+        headers: { Accept: 'application/json', 'User-Agent': USER_AGENT },
+        signal: AbortSignal.timeout(this.timeoutMs),
+      });
+    } catch (error) {
+      if (error.name === 'TimeoutError' || error.name === 'AbortError') {
+        throw new Error(`request timed out after ${this.timeoutMs}ms`);
+      }
+      throw new Error(`network error: ${error.message}`);
+    }
+    if (response.status === 429) {
+      const reset = response.headers?.get?.('x-ratelimit-reset');
+      // PullPush's 429 body states its actual policy ("does not provide free
+      // scraping resources for agents...") — pass that through verbatim.
+      let detail = '';
+      try { detail = (await response.json())?.error ?? ''; } catch { /* no body */ }
+      throw Object.assign(
+        new Error(`rate limited (429)${reset ? `, retry in ${reset}s` : ''}${detail ? ` — ${detail}` : ''}`),
+        { retryable: true },
+      );
+    }
+    if (!response.ok) {
+      // Both archives put the real reason in the body (e.g. Arctic Shift's
+      // throttle/parameter complaints) — surface it, bounded.
+      let detail = '';
+      try {
+        const body = await response.text();
+        let msg = body.slice(0, 200);
+        try { msg = JSON.parse(body)?.error || msg; } catch { /* non-JSON body — use it raw */ }
+        detail = msg ? `: ${msg}` : '';
+      } catch { /* unreadable body — status alone will have to do */ }
+      throw Object.assign(
+        new Error(`HTTP ${response.status} ${response.statusText}${detail}`),
+        { retryable: response.status === 422 && /timeout|slow down/i.test(detail) },
+      );
+    }
+    return response.json();
+  }
+}
+
+export default RedditSearchTool;

package/src/tools/search/serpRank.js

@@ -24,7 +24,10 @@ const SerpRankSchema = z.object({
   location_code: z.number().int().optional(),
   language_code: z.string().optional().default('en'),
   device: z.enum(['desktop', 'mobile']).optional().default('desktop'),
-  depth: z.number().int().min(10).max(200).optional().default(100), // DataForSEO caps depth at 200
+  // DataForSEO bills per 10 results scanned and gets slower the deeper it goes,
+  // so the default stays shallow: 20 covers Google's first two pages for $0.004.
+  // Raise it (max 200, their cap) when a deeper position is worth the spend.
+  depth: z.number().int().min(10).max(200).optional().default(20),
 });
 
 /** Reduce a domain or URL to a bare, comparable host: "https://www.Example.com/x" → "example.com". */