crawlforge-mcp-server 5.2.3 → 5.2.4

package/CLAUDE.md

@@ -62,7 +62,7 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 28 web scraping, crawling, and content processing tools (5 inline + 23 advanced).
 
-**Current Version:** 5.2.3
+**Current Version:** 5.2.4
 
 ## Development Commands
 
@@ -184,7 +184,7 @@ search_web, serp_rank, reddit_search, crawl_deep, map_site, extract_content, pro
 - `serp_rank` — reports where a target domain ranks in Google's REAL organic results for a keyword (the SERP position Google Custom Search / `search_web` cannot give). Backed by the DataForSEO Google Organic SERP API (Live Advanced, `POST /v3/serp/google/organic/live/advanced`, HTTP Basic auth). Credentials via `DATAFORSEO_LOGIN` / `DATAFORSEO_PASSWORD`, billed to the user's own DataForSEO account (~US$0.002 per 10 results of `depth`, so $0.004 at the default `depth:20` and $0.02 at `depth:100`), separate from CrawlForge credits. When unconfigured it returns `{ configured:false }` and charges **0** credits; when configured, **Cost: 5**. Never fabricates a rank.
 
 **reddit_search (community archives, v5.1.0):**
-- `reddit_search` — searches Reddit posts/comments or reads a full comment thread. reddit.com 403-blocks ALL direct access (fetch, browser-UA, stealth — IP/TLS-reputation based), so this never touches reddit.com; it queries the Arctic Shift archive (`arctic-shift.photon-reddit.com`, near-real-time, comment trees) and PullPush (`api.pullpush.io`, Pushshift-compatible). Routing: Arctic Shift's keyword search REQUIRES a subreddit/author scope (verified live: HTTP 400 without one), so unscoped full-text search goes to PullPush only; scoped searches use Arctic Shift with PullPush as error-only fallback. Modes: `posts` (default), `comments`, `thread` (post + nested comment tree by `link_id`). Free, no credentials; PullPush rate-limits aggressively (~15 req/min). Cost: 2. `REDDIT_SEARCH_TIMEOUT_MS` overrides the 30s default.
+- `reddit_search` — searches Reddit posts/comments or reads a full comment thread. reddit.com 403-blocks ALL direct access (fetch, browser-UA, stealth — IP/TLS-reputation based), so this never touches reddit.com; it queries the Arctic Shift archive (`arctic-shift.photon-reddit.com`, near-real-time, comment trees) and PullPush (`api.pullpush.io`, Pushshift-compatible). Routing: Arctic Shift's keyword search REQUIRES a subreddit/author scope (verified live: HTTP 400 without one), so a scoped search queries it directly, while an unscoped keyword search for posts discovers them with a site-restricted web search and then reads those posts from the archive by ID (`source:"web_discovery"`). PullPush stopped serving automated clients in August 2026 (429 "does not provide free scraping resources for agents" on every UA; Cloudflare 403 from some IPs) and is no longer tried automatically anywhere — `source:"pullpush"` still reaches it. An unscoped COMMENT search has no backend and returns a scope-required error. Modes: `posts` (default), `comments`, `thread` (post + nested comment tree by `link_id`). Free, no Reddit credentials. Cost: 5 — a Reddit-wide search spends the same web search `search_web` does. `REDDIT_SEARCH_TIMEOUT_MS` overrides the 30s default.
 
 **v4.6.0 additions (Phase D):**
 - `scrape` — single fetch + one cheerio load dispatching a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) + `onlyMainContent`; partial-success via per-format `warnings[]`. Cost: 2.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "5.2.3",
+  "version": "5.2.4",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 28 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",

package/server.js

@@ -100,7 +100,7 @@ const taskStore = createTaskStore({ logger });
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "5.2.3",
+  version: "5.2.4",
   description: "Production-ready MCP server with 28 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",

package/src/core/AuthManager.js

@@ -560,7 +560,7 @@ class AuthManager {
       process_document: 2,
       localization: 2,
       scrape: 2,
-      reddit_search: 2, // free community archives (Arctic Shift / PullPush), no external billing
+      reddit_search: 5, // a Reddit-wide search spends a web search to discover posts, same as search_web
 
       // 3 credits
       track_changes: 3,

package/src/skills/agent-skills/crawlforge-deep-research/SKILL.md

@@ -51,7 +51,7 @@ Returns titles, URLs, snippets. Supports `lang`, `site` (domain filter),
 `enable_ranking`, and `enable_deduplication`. CLI:
 `crawlforge search "MCP server tutorial" --limit 5`.
 
-## reddit_search (cost: 2)
+## reddit_search (cost: 5)
 
 reddit.com 403-blocks direct scraping, so this queries the Arctic Shift and
 PullPush community archives instead (free, no Reddit credentials).

package/src/skills/agent-skills/crawlforge-getting-started/references/credits.md

@@ -25,7 +25,6 @@ metered; there is no free tier. Tools marked "scales" cost more as work grows.
 | `map_site` | URL discovery / sitemap. |
 | `process_document` | PDF / DOCX / TXT parsing. |
 | `localization` | Locale / geo emulation. |
-| `reddit_search` | Reddit posts/comments/threads via community archives. |
 
 ## 3 credits
 
@@ -51,6 +50,7 @@ metered; there is no free tier. Tools marked "scales" cost more as work grows.
 | `scrape_with_actions` | Browser automation then scrape. |
 | `batch_scrape` | Many URLs; projection scales with URL count. |
 | `search_web` | Web search. |
+| `reddit_search` | Reddit posts/comments/threads; a Reddit-wide search spends a web search to discover posts. |
 | `generate_llms_txt` | AI-compliance file. |
 
 ## 8 credits