crawlforge-mcp-server 5.2.3 → 5.2.5

package/CLAUDE.md

@@ -62,7 +62,7 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 28 web scraping, crawling, and content processing tools (5 inline + 23 advanced).
 
-**Current Version:** 5.2.3
+**Current Version:** 5.2.5
 
 ## Development Commands
 
@@ -184,7 +184,7 @@ search_web, serp_rank, reddit_search, crawl_deep, map_site, extract_content, pro
 - `serp_rank` — reports where a target domain ranks in Google's REAL organic results for a keyword (the SERP position Google Custom Search / `search_web` cannot give). Backed by the DataForSEO Google Organic SERP API (Live Advanced, `POST /v3/serp/google/organic/live/advanced`, HTTP Basic auth). Credentials via `DATAFORSEO_LOGIN` / `DATAFORSEO_PASSWORD`, billed to the user's own DataForSEO account (~US$0.002 per 10 results of `depth`, so $0.004 at the default `depth:20` and $0.02 at `depth:100`), separate from CrawlForge credits. When unconfigured it returns `{ configured:false }` and charges **0** credits; when configured, **Cost: 5**. Never fabricates a rank.
 
 **reddit_search (community archives, v5.1.0):**
-- `reddit_search` — searches Reddit posts/comments or reads a full comment thread. reddit.com 403-blocks ALL direct access (fetch, browser-UA, stealth — IP/TLS-reputation based), so this never touches reddit.com; it queries the Arctic Shift archive (`arctic-shift.photon-reddit.com`, near-real-time, comment trees) and PullPush (`api.pullpush.io`, Pushshift-compatible). Routing: Arctic Shift's keyword search REQUIRES a subreddit/author scope (verified live: HTTP 400 without one), so unscoped full-text search goes to PullPush only; scoped searches use Arctic Shift with PullPush as error-only fallback. Modes: `posts` (default), `comments`, `thread` (post + nested comment tree by `link_id`). Free, no credentials; PullPush rate-limits aggressively (~15 req/min). Cost: 2. `REDDIT_SEARCH_TIMEOUT_MS` overrides the 30s default.
+- `reddit_search` — searches Reddit posts/comments or reads a full comment thread. reddit.com 403-blocks ALL direct access (fetch, browser-UA, stealth — IP/TLS-reputation based), so this never touches reddit.com; it queries the Arctic Shift archive (`arctic-shift.photon-reddit.com`, near-real-time, comment trees) and PullPush (`api.pullpush.io`, Pushshift-compatible). Routing: Arctic Shift's keyword search REQUIRES a subreddit/author scope (verified live: HTTP 400 without one), so a scoped search queries it directly, while an unscoped keyword search for posts discovers them with a site-restricted web search and then reads those posts from the archive by ID (`source:"web_discovery"`). PullPush stopped serving automated clients in August 2026 (429 "does not provide free scraping resources for agents" on every UA; Cloudflare 403 from some IPs) and is no longer tried automatically anywhere — `source:"pullpush"` still reaches it. An unscoped COMMENT search has no backend and returns a scope-required error. Modes: `posts` (default), `comments`, `thread` (post + nested comment tree by `link_id`). Free, no Reddit credentials. Cost: 5 — a Reddit-wide search spends the same web search `search_web` does. `REDDIT_SEARCH_TIMEOUT_MS` overrides the 30s default.
 
 **v4.6.0 additions (Phase D):**
 - `scrape` — single fetch + one cheerio load dispatching a `formats` array (markdown/html/rawHtml/text/links/metadata/screenshot/json-schema) + `onlyMainContent`; partial-success via per-format `warnings[]`. Cost: 2.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "5.2.3",
+  "version": "5.2.5",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 28 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
@@ -113,7 +113,7 @@
     "cheerio": "^1.1.2",
     "commander": "^14.0.3",
     "compromise": "^14.14.4",
-    "crawlforge-extractors": "^1.2.0",
+    "crawlforge-extractors": "^1.2.1",
     "diff": "^9.0.0",
     "dotenv": "^17.2.1",
     "franc": "^6.2.0",

package/server.js

@@ -100,7 +100,7 @@ const taskStore = createTaskStore({ logger });
 // Create the server
 const server = new McpServer({
   name: "crawlforge",
-  version: "5.2.3",
+  version: "5.2.5",
   description: "Production-ready MCP server with 28 web scraping, crawling, and content processing tools. Features MCP Resources (crawlforge://), Prompts, Sampling fallback, Elicitation, stealth browsing, deep research, structured extraction, real Google SERP rank tracking, Reddit search via community archives, change tracking, local-LLM extraction via Ollama, unified multi-format scrape, and autonomous agent tool.",
   homepage: "https://www.crawlforge.dev",
   icon: "https://www.crawlforge.dev/icon.png",

package/src/core/AuthManager.js

@@ -560,7 +560,7 @@ class AuthManager {
       process_document: 2,
       localization: 2,
       scrape: 2,
-      reddit_search: 2, // free community archives (Arctic Shift / PullPush), no external billing
+      reddit_search: 5, // a Reddit-wide search spends a web search to discover posts, same as search_web
 
       // 3 credits
       track_changes: 3,

package/src/core/ChangeTracker.js

@@ -312,14 +312,22 @@ export class ChangeTracker extends EventEmitter {
       
       this.emit('changeDetected', changeRecord);
       
+      const ignoredOptions = this.findIgnoredCompareOptions(options, baseline.options);
+
       return {
         hasChanges: significance !== 'none',
         significance,
         changeType: changeRecord.changeType,
-        summary: this.generateChangeSummary(changeAnalysis),
+        summary: this.generateChangeSummary(changeAnalysis, significance),
         details: changeAnalysis,
         metrics: changeRecord.metrics,
-        recommendations: this.generateChangeRecommendations(changeRecord)
+        recommendations: this.generateChangeRecommendations(changeRecord),
+        ...(ignoredOptions.length ? {
+          warnings: [
+            `${ignoredOptions.join(', ')} passed to this compare ${ignoredOptions.length === 1 ? 'was' : 'were'} ignored — ` +
+            `the baseline's options are applied to both sides of the diff. Recreate the baseline to change them.`
+          ]
+        } : {})
       };
       
     } catch (error) {
@@ -328,6 +336,27 @@ export class ChangeTracker extends EventEmitter {
     }
   }
   
+  /**
+   * Report analysis options supplied at compare time that differ from the
+   * baseline's and were therefore not applied.
+   *
+   * Ignoring them is deliberate — both sides of a diff have to be analyzed
+   * identically, and once customSelectors scoped the baseline it no longer
+   * holds the full document to re-scope. But doing it silently let a caller
+   * scope a compare and read the resulting whole-page churn as real change:
+   * the result is byte-identical to an unscoped run, with nothing saying so.
+   *
+   * @param {Object} callerOptions - trackingOptions passed to this compare
+   * @param {Object} baselineOptions - options stored with the baseline
+   * @returns {string[]} - names of the ignored options
+   */
+  findIgnoredCompareOptions(callerOptions = {}, baselineOptions = {}) {
+    return ['granularity', 'customSelectors', 'excludeSelectors'].filter(key => {
+      if (callerOptions[key] === undefined) return false;
+      return JSON.stringify(callerOptions[key]) !== JSON.stringify(baselineOptions[key]);
+    });
+  }
+  
   /**
    * Analyze content structure and create hierarchical hashes
    * @param {string} content - Content to analyze
@@ -701,6 +730,29 @@ export class ChangeTracker extends EventEmitter {
         }
       });
     });
+
+    // Scoping to a tag outside that list (address, td, li, tr, dd) otherwise
+    // indexes ZERO elements: the scoped document is hashed, but nothing in it
+    // matches the allowlist, so every compare sees an empty element map and
+    // can never report an element-level change. Hash the scoped elements
+    // themselves for any tag the loop above does not already cover.
+    if (options.customSelectors?.length) {
+      const alreadyHashed = new Set(importantElements);
+      options.customSelectors.forEach((selector, selectorIndex) => {
+        $(selector).each((index, element) => {
+          const tag = element.tagName?.toLowerCase();
+          if (!tag || alreadyHashed.has(tag)) return;
+
+          const elementKey = `custom_${selectorIndex}_${index}`;
+          analysis.hashes.elements[elementKey] = this.hashContent($(element).html() || '');
+
+          if (options.trackAttributes) {
+            const attributes = element.attribs || {};
+            analysis.hashes.elements[`${elementKey}_attr`] = this.hashContent(JSON.stringify(attributes));
+          }
+        });
+      });
+    }
   }
   
   async analyzeTextLevel($, analysis, options) {
@@ -1160,7 +1212,7 @@ export class ChangeTracker extends EventEmitter {
     return 'text_change';
   }
   
-  generateChangeSummary(changeAnalysis) {
+  generateChangeSummary(changeAnalysis, significance) {
     const { addedElements, removedElements, modifiedElements, similarity } = changeAnalysis;
     
     const total = addedElements.length + removedElements.length + modifiedElements.length;
@@ -1171,7 +1223,13 @@ export class ChangeTracker extends EventEmitter {
       added: addedElements.length,
       removed: removedElements.length,
       modified: modifiedElements.length,
-      changeDescription: this.generateChangeDescription(changeAnalysis)
+      // Sub-threshold text noise (a rotating session token, a base64 timestamp)
+      // still lands in textChanges, so the description read "Text content
+      // changed" on a compare that reported hasChanges:false and
+      // totalChanges:0. Defer to the verdict the caller is given.
+      changeDescription: significance === 'none'
+        ? 'No significant changes detected'
+        : this.generateChangeDescription(changeAnalysis)
     };
   }
   

package/src/skills/agent-skills/crawlforge-deep-research/SKILL.md

@@ -51,7 +51,7 @@ Returns titles, URLs, snippets. Supports `lang`, `site` (domain filter),
 `enable_ranking`, and `enable_deduplication`. CLI:
 `crawlforge search "MCP server tutorial" --limit 5`.
 
-## reddit_search (cost: 2)
+## reddit_search (cost: 5)
 
 reddit.com 403-blocks direct scraping, so this queries the Arctic Shift and
 PullPush community archives instead (free, no Reddit credentials).

package/src/skills/agent-skills/crawlforge-getting-started/references/credits.md

@@ -25,7 +25,6 @@ metered; there is no free tier. Tools marked "scales" cost more as work grows.
 | `map_site` | URL discovery / sitemap. |
 | `process_document` | PDF / DOCX / TXT parsing. |
 | `localization` | Locale / geo emulation. |
-| `reddit_search` | Reddit posts/comments/threads via community archives. |
 
 ## 3 credits
 
@@ -51,6 +50,7 @@ metered; there is no free tier. Tools marked "scales" cost more as work grows.
 | `scrape_with_actions` | Browser automation then scrape. |
 | `batch_scrape` | Many URLs; projection scales with URL count. |
 | `search_web` | Web search. |
+| `reddit_search` | Reddit posts/comments/threads; a Reddit-wide search spends a web search to discover posts. |
 | `generate_llms_txt` | AI-compliance file. |
 
 ## 8 credits

package/src/tools/tracking/trackChanges/index.js

@@ -281,6 +281,7 @@ export class TrackChangesTool extends EventEmitter {
       details: comparisonResult.details,
       metrics: comparisonResult.metrics,
       recommendations: comparisonResult.recommendations,
+      ...(comparisonResult.warnings ? { warnings: comparisonResult.warnings } : {}),
       snapshot: snapshotInfo, timestamp: Date.now()
     };
   }