crawlforge-mcp-server 3.0.3 → 3.0.4

package/CLAUDE.md

@@ -4,7 +4,10 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co
 
 ## Project Overview
 
-CrawlForge MCP Server - A professional MCP (Model Context Protocol) server implementation providing 19 comprehensive web scraping, crawling, and content processing tools. Version 3.0 includes advanced content extraction, document processing, summarization, and analysis capabilities. Wave 2 adds asynchronous batch processing and browser automation features. Wave 3 introduces deep research orchestration, stealth scraping, localization, and change tracking.
+CrawlForge MCP Server - A professional MCP (Model Context Protocol) server implementation providing 19 comprehensive web scraping, crawling, and content processing tools. Version 3.0.3 includes advanced content extraction, document processing, summarization, and analysis capabilities. Wave 2 adds asynchronous batch processing and browser automation features. Wave 3 introduces deep research orchestration, stealth scraping, localization, and change tracking.
+
+**Current Version:** 3.0.3
+**Security Status:** Secure (authentication bypass vulnerability fixed in v3.0.3)
 
 ## Development Commands
 
@@ -12,11 +15,16 @@ CrawlForge MCP Server - A professional MCP (Model Context Protocol) server imple
 # Install dependencies
 npm install
 
-# Setup (required for first run)
+# Setup (required for first run - users only)
 npm run setup
 # Or provide API key via environment:
 export CRAWLFORGE_API_KEY="your_api_key_here"
 
+# Creator Mode (for package maintainer only)
+# Set your creator secret in .env file:
+# CRAWLFORGE_CREATOR_SECRET=your-secret-uuid
+# This enables unlimited access for development/testing
+
 # Run the server (production)
 npm start
 
@@ -104,33 +112,48 @@ Tools are organized in subdirectories by category:
 
 The main server implementation is in `server.js` which:
 
-1. **Authentication Flow**: Uses AuthManager for API key validation and credit tracking
+1. **Secure Creator Mode** (server.js lines 3-25):
+   - Loads `.env` file early to check for `CRAWLFORGE_CREATOR_SECRET`
+   - Validates secret using SHA256 hash comparison
+   - Only creator with valid secret UUID can enable unlimited access
+   - Hash stored in code is safe to commit (one-way cryptographic hash)
+
+2. **Authentication Flow**: Uses AuthManager for API key validation and credit tracking
    - Checks for authentication on startup
    - Auto-setup if CRAWLFORGE_API_KEY environment variable is present
-2. **Tool Registration**: All tools registered via `server.registerTool()` pattern
+   - Creator mode bypasses credit checks for development/testing
+
+3. **Tool Registration**: All tools registered via `server.registerTool()` pattern
    - Wrapped with `withAuth()` function for credit tracking and authentication
    - Each tool has inline Zod schema for parameter validation
    - Response format uses `content` array with text objects
-3. **Transport**: Uses stdio transport for MCP protocol communication
-4. **Graceful Shutdown**: Cleans up browser instances, job managers, and other resources
+
+4. **Transport**: Uses stdio transport for MCP protocol communication
+
+5. **Graceful Shutdown**: Cleans up browser instances, job managers, and other resources
 
 ### Tool Credit System
 
 Each tool wrapped with `withAuth(toolName, handler)`:
 
-- Checks credits before execution
+- Checks credits before execution (skipped in creator mode)
 - Reports usage with credit deduction on success
 - Charges half credits on error
 - Returns credit error if insufficient balance
+- Creator mode: Unlimited access for package maintainer
 
 ### Key Configuration
 
 Critical environment variables defined in `src/constants/config.js`:
 
 ```bash
-# Authentication (required)
+# Authentication (required for users)
 CRAWLFORGE_API_KEY=your_api_key_here
 
+# Creator Mode (maintainer only - KEEP SECRET!)
+# CRAWLFORGE_CREATOR_SECRET=your-uuid-secret
+# Enables unlimited access for development/testing
+
 # Search Provider (auto, google, duckduckgo)
 SEARCH_PROVIDER=auto
 

package/README.md

@@ -9,7 +9,7 @@ Professional web scraping and content extraction server implementing the Model C
 
 ## 🎯 Features
 
-- **19 Professional Tools**: Web scraping, deep research, stealth browsing, content analysis
+- **18 Professional Tools**: Web scraping, deep research, stealth browsing, content analysis
 - **Free Tier**: 1,000 credits to get started instantly
 - **MCP Compatible**: Works with Claude, Cursor, and other MCP-enabled AI tools
 - **Enterprise Ready**: Scale up with paid plans for production use
@@ -113,7 +113,7 @@ Or use the MCP plugin in Cursor settings.
 | **Enterprise** | 250,000 | Large scale operations |
 
 **All plans include:**
-- Access to all 19 tools
+- Access to all 18 tools
 - Credits never expire and roll over month-to-month
 - API access and webhook notifications
 
@@ -125,7 +125,7 @@ Or use the MCP plugin in Cursor settings.
 
 ```bash
 # Optional: Set API key via environment
-export CRAWLFORGE_API_KEY="sk_live_your_api_key_here"
+export CRAWLFORGE_API_KEY="cf_live_your_api_key_here"
 
 # Optional: Custom API endpoint (for enterprise)
 export CRAWLFORGE_API_URL="https://api.crawlforge.dev"
@@ -137,7 +137,7 @@ Your configuration is stored at `~/.crawlforge/config.json`:
 
 ```json
 {
-  "apiKey": "sk_live_...",
+  "apiKey": "cf_live_...",
   "userId": "user_...",
   "email": "you@example.com"
 }
@@ -157,11 +157,16 @@ Once configured, use these tools in your AI assistant:
 
 ## 🔒 Security & Privacy
 
-- API keys are stored locally and encrypted
-- All connections use HTTPS
-- No data is stored on our servers beyond usage logs
-- Compliant with robots.txt and rate limits
-- GDPR compliant
+- **Secure Authentication**: API keys required for all operations (no bypass methods)
+- **Local Storage**: API keys stored securely at `~/.crawlforge/config.json`
+- **HTTPS Only**: All connections use encrypted HTTPS
+- **No Data Retention**: We don't store scraped data, only usage logs
+- **Rate Limiting**: Built-in protection against abuse
+- **Compliance**: Respects robots.txt and GDPR requirements
+
+### Security Updates
+
+**v3.0.3 (2025-10-01)**: Removed authentication bypass vulnerability. All users must authenticate with valid API keys.
 
 ## 🆘 Support
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "3.0.3",
+  "version": "3.0.4",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 19 comprehensive web scraping, crawling, and content processing tools.",
   "main": "server.js",
   "bin": {
@@ -95,12 +95,12 @@
     "compromise": "^14.14.4",
     "diff": "^8.0.2",
     "dotenv": "^17.2.1",
+    "duck-duck-scrape": "^2.2.7",
     "franc": "^6.2.0",
     "isomorphic-dompurify": "^2.26.0",
     "jsdom": "^26.1.0",
     "lru-cache": "^11.1.0",
     "node-cron": "^3.0.3",
-    "node-fetch": "^3.3.2",
     "node-summarizer": "^1.0.7",
     "p-queue": "^8.1.0",
     "pdf-parse": "^1.1.1",

package/server.js

@@ -97,7 +97,7 @@ if (configErrors.length > 0 && config.server.nodeEnv === 'production') {
 }
 
 // Create the server
-const server = new McpServer({ name: "crawlforge", version: "3.0.1" });
+const server = new McpServer({ name: "crawlforge", version: "3.0.4" });
 
 // Helper function to wrap tool handlers with authentication and credit tracking
 function withAuth(toolName, handler) {

package/src/core/AlertNotificationSystem.js

@@ -4,7 +4,7 @@
  */
 
 import { EventEmitter } from 'events';
-import fetch from 'node-fetch';
+// Using native fetch (Node.js 18+)
 import crypto from 'crypto';
 
 export class AlertNotificationSystem extends EventEmitter {

package/src/core/AuthManager.js

@@ -3,7 +3,7 @@
  * Handles API key validation, credit tracking, and usage reporting
  */
 
-import fetch from 'node-fetch';
+// Using native fetch (Node.js 18+)
 import fs from 'fs/promises';
 import path from 'path';
 
@@ -221,7 +221,7 @@ class AuthManager {
         responseStatus,
         processingTime,
         timestamp: new Date().toISOString(),
-        version: '3.0.0'
+        version: '3.0.3'
       };
 
       await fetch(`${this.apiEndpoint}/api/v1/usage`, {
@@ -268,12 +268,13 @@ class AuthManager {
       deep_research: 10,
       stealth_mode: 10,
       
-      // Heavy processing (10+ credits)
+      // Heavy processing (3-5 credits)
       process_document: 3,
       extract_content: 3,
       scrape_with_actions: 5,
       generate_llms_txt: 3,
-      localization: 5
+      localization: 5,
+      track_changes: 3
     };
 
     return costs[tool] || 1;

package/src/core/connections/ConnectionPool.js

@@ -218,7 +218,7 @@ export class ConnectionPool extends EventEmitter {
    * @returns {Promise<Object>} - Request result
    */
   async executeRequest(options, requestId) {
-    const { fetch } = await import('node-fetch');
+    // Using native fetch (Node.js 18+)
     
     const {
       url,

package/src/core/integrations/PerformanceIntegration.js

@@ -109,8 +109,7 @@ export async function enhancedFetch(url, options = {}) {
     const requestOptions = typeof url === 'string' ? { url, ...options } : url;
     return await connectionPoolInstance.request(requestOptions);
   } else {
-    // Fallback to regular fetch
-    const { default: fetch } = await import('node-fetch');
+    // Fallback to native fetch (Node.js 18+)
     return await fetch(url, options);
   }
 }
@@ -182,8 +181,7 @@ export async function enhancedConcurrentRequests(requests, options = {}) {
   if (connectionPoolInstance) {
     return await connectionPoolInstance.requestBatch(requests, options);
   } else {
-    // Fallback to Promise.all with regular fetch
-    const { default: fetch } = await import('node-fetch');
+    // Fallback to Promise.all with native fetch (Node.js 18+)
     const promises = requests.map(request => fetch(request.url || request, request));
     return await Promise.all(promises);
   }

package/src/tools/search/adapters/duckduckgoSearch.js

@@ -1,11 +1,12 @@
 import * as cheerio from 'cheerio';
+import { search as ddgSearch, SafeSearchType, SearchTimeType } from 'duck-duck-scrape';
 
 export class DuckDuckGoSearchAdapter {
   constructor(options = {}) {
     this.timeout = options.timeout || 30000;
     this.maxRetries = options.maxRetries || 3;
-    this.userAgent = options.userAgent || 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36';
-    this.retryDelay = options.retryDelay || 1000;
+    this.userAgent = options.userAgent || 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36';
+    this.retryDelay = options.retryDelay || 2000; // Increased base delay
     this.baseUrl = 'https://html.duckduckgo.com/html/';
   }
 
@@ -19,19 +20,36 @@ export class DuckDuckGoSearchAdapter {
       dateRestrict
     } = params;
 
-    // Calculate pagination offset for DuckDuckGo
+    // Try duck-duck-scrape library first (more reliable API access)
+    try {
+      const results = await this.searchWithLibrary(query, num, safe, dateRestrict);
+      if (results.items && results.items.length > 0) {
+        return results;
+      }
+    } catch (libraryError) {
+      console.warn('DuckDuckGo library search failed:', libraryError.message);
+      // Check if it's a CAPTCHA/anomaly error
+      if (libraryError.message.includes('anomaly') || libraryError.message.includes('too quickly')) {
+        throw new Error(
+          'DuckDuckGo is blocking automated requests. ' +
+          'To use web search reliably, please configure Google Custom Search API by setting ' +
+          'GOOGLE_API_KEY and GOOGLE_SEARCH_ENGINE_ID environment variables. ' +
+          'See: https://developers.google.com/custom-search/v1/introduction'
+        );
+      }
+    }
+
+    // Fallback to HTML scraping (legacy method)
     const offset = (start - 1) * num;
 
-    // Build form data for POST request to DuckDuckGo HTML endpoint
     const formData = new URLSearchParams({
       q: query,
-      b: offset.toString(), // DuckDuckGo uses 'b' for pagination offset
-      kl: 'us-en',         // Default language
-      df: '',              // Date filter
-      safe: 'moderate'     // Safe search setting
+      b: offset.toString(),
+      kl: 'us-en',
+      df: '',
+      safe: 'moderate'
     });
 
-    // Update safe search parameter
     if (safe === 'active') {
       formData.set('safe', 'strict');
     } else if (safe === 'off') {
@@ -40,13 +58,11 @@ export class DuckDuckGoSearchAdapter {
       formData.set('safe', 'moderate');
     }
 
-    // Add language if specified
     if (lr && lr.startsWith('lang_')) {
       const lang = lr.replace('lang_', '');
       formData.set('kl', this.mapLanguageCode(lang));
     }
 
-    // Add date filter if specified
     if (dateRestrict) {
       const timeFilter = this.mapDateRestrict(dateRestrict);
       if (timeFilter) {
@@ -57,15 +73,20 @@ export class DuckDuckGoSearchAdapter {
     let lastError;
     for (let attempt = 1; attempt <= this.maxRetries; attempt++) {
       try {
+        // Add delay between attempts to avoid rate limiting
+        if (attempt > 1) {
+          await new Promise(resolve =>
+            setTimeout(resolve, this.retryDelay * Math.pow(2, attempt - 1))
+          );
+        }
+
         const htmlResponse = await this.makeRequest(formData);
         return this.parseHtmlResponse(htmlResponse, query, num, start);
       } catch (error) {
         lastError = error;
-        if (attempt < this.maxRetries) {
-          // Exponential backoff
-          await new Promise(resolve => 
-            setTimeout(resolve, this.retryDelay * Math.pow(2, attempt - 1))
-          );
+        // If it's a CAPTCHA error, don't retry - it won't help
+        if (error.message.includes('CAPTCHA') || error.message.includes('automated requests')) {
+          throw error;
         }
       }
     }
@@ -73,6 +94,67 @@ export class DuckDuckGoSearchAdapter {
     throw new Error(`DuckDuckGo search failed after ${this.maxRetries} attempts: ${lastError.message}`);
   }
 
+  async searchWithLibrary(query, num, safe, dateRestrict) {
+    // Map safe search settings
+    let safeSearch = SafeSearchType.MODERATE;
+    if (safe === 'active' || safe === 'strict') {
+      safeSearch = SafeSearchType.STRICT;
+    } else if (safe === 'off') {
+      safeSearch = SafeSearchType.OFF;
+    }
+
+    // Map time filter
+    let time = undefined;
+    if (dateRestrict) {
+      const timeMap = {
+        'd1': SearchTimeType.DAY,
+        'w1': SearchTimeType.WEEK,
+        'm1': SearchTimeType.MONTH,
+        'y1': SearchTimeType.YEAR
+      };
+      time = timeMap[dateRestrict];
+    }
+
+    const searchResults = await ddgSearch(query, {
+      safeSearch,
+      time,
+      locale: 'en-us'
+    });
+
+    // Transform results to match expected format
+    const items = (searchResults.results || []).slice(0, num).map(result => ({
+      title: result.title || '',
+      link: result.url || '',
+      snippet: result.description || '',
+      displayLink: this.extractDomain(result.url),
+      formattedUrl: result.url || '',
+      htmlSnippet: result.description || '',
+      pagemap: {
+        metatags: {
+          title: result.title || '',
+          description: result.description || ''
+        }
+      },
+      metadata: {
+        source: 'duckduckgo_api',
+        type: 'web_result',
+        hostname: result.hostname || '',
+        icon: result.icon || ''
+      }
+    }));
+
+    return {
+      kind: 'duckduckgo#search',
+      searchInformation: {
+        searchTime: 0.1,
+        formattedSearchTime: '0.10',
+        totalResults: items.length.toString(),
+        formattedTotalResults: items.length.toLocaleString()
+      },
+      items: items
+    };
+  }
+
   async makeRequest(formData) {
     const controller = new AbortController();
     const timeoutId = setTimeout(() => controller.abort(), this.timeout);
@@ -121,6 +203,26 @@ export class DuckDuckGoSearchAdapter {
       const $ = cheerio.load(html);
       const items = [];
 
+      // Check for CAPTCHA challenge (DuckDuckGo bot protection)
+      const captchaIndicators = [
+        'anomaly-modal',
+        'Unfortunately, bots use DuckDuckGo too',
+        'Select all squares containing a duck',
+        'confirm this search was made by a human',
+        'challenge-form'
+      ];
+
+      for (const indicator of captchaIndicators) {
+        if (html.includes(indicator)) {
+          throw new Error(
+            'DuckDuckGo CAPTCHA detected - automated requests are being blocked. ' +
+            'To use web search reliably, please configure Google Custom Search API by setting ' +
+            'GOOGLE_API_KEY and GOOGLE_SEARCH_ENGINE_ID environment variables. ' +
+            'See: https://developers.google.com/custom-search/v1/introduction'
+          );
+        }
+      }
+
       // Look for search result containers - DuckDuckGo uses various selectors
       const resultSelectors = [
         '.result',           // Primary result class