crawlforge-mcp-server 3.0.0

package/CLAUDE.md

@@ -0,0 +1,315 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+CrawlForge MCP Server - A professional MCP (Model Context Protocol) server implementation providing 16 comprehensive web scraping, crawling, and content processing tools. Version 3.0 includes advanced content extraction, document processing, summarization, and analysis capabilities. Wave 2 adds asynchronous batch processing and browser automation features. Wave 3 introduces deep research orchestration, stealth scraping, localization, and change tracking.
+
+## Development Commands
+
+```bash
+# Install dependencies
+npm install
+
+# Copy and configure environment
+cp .env.example .env
+# Edit .env to add Google API credentials if using Google search
+
+# Run the server
+npm start
+
+# Test MCP protocol compliance
+# NOTE: test-server.js doesn't exist, use integration tests instead
+npm run test:integration      # Integration tests including MCP compliance
+
+# Lint checks (no linter configured yet, placeholder)
+npm run lint
+
+# Performance tests
+npm run test:performance       # Full performance test suite
+npm run test:performance:quick # Quick performance tests
+npm run test:load             # Load testing
+npm run test:memory           # Memory usage tests
+npm run test:benchmark        # Component benchmarks
+npm run test:integration      # Integration tests
+npm run test:security         # Security test suite
+npm run test:all             # Run all tests
+
+# Wave 2 Validation Tests
+node tests/validation/test-wave2-runner.js  # Test Wave 2 features
+node tests/validation/test-batch-scrape.js  # Test batch scraping
+node tests/validation/test-scrape-with-actions.js  # Test action scraping
+node tests/integration/master-test-runner.js # Run master test suite
+
+# Wave 3 Tests
+npm run test:wave3             # Full Wave 3 validation
+npm run test:wave3:quick       # Quick Wave 3 tests
+npm run test:wave3:verbose     # Verbose Wave 3 output
+npm run test:unit:wave3        # Wave 3 unit tests (Jest)
+npm run test:integration:wave3 # Wave 3 integration tests
+
+# Docker commands
+npm run docker:build         # Build Docker image
+npm run docker:dev          # Run development container
+npm run docker:prod         # Run production container
+npm run docker:test         # Run test container
+npm run docker:perf         # Run performance test container
+
+# Security Testing (CI/CD Integration)
+npm run test:security       # Run comprehensive security test suite
+npm audit                   # Check for dependency vulnerabilities
+npm audit fix               # Automatically fix vulnerabilities
+npm outdated                # Check for outdated packages
+
+# Release management
+npm run release:patch       # Patch version bump
+npm run release:minor       # Minor version bump
+npm run release:major       # Major version bump
+
+# Cleanup
+npm run clean              # Remove cache, logs, test results
+
+# Running specific test files
+node tests/unit/linkAnalyzer.test.js          # Unit test for link analyzer
+node tests/validation/wave3-validation.js     # Wave 3 validation suite
+node tests/security/security-test-suite.js    # Security test suite
+```
+
+## High-Level Architecture
+
+### Core Infrastructure (`src/core/`)
+- **PerformanceManager**: Centralized performance monitoring and optimization
+- **JobManager**: Asynchronous job tracking and management for batch operations
+- **WebhookDispatcher**: Event notification system for job completion callbacks
+- **ActionExecutor**: Browser automation engine for complex interactions
+- **ResearchOrchestrator**: Coordinates multi-stage research with query expansion and synthesis
+- **StealthBrowserManager**: Manages stealth mode scraping with anti-detection features
+- **LocalizationManager**: Handles multi-language content and localization
+- **ChangeTracker**: Tracks and compares content changes over time
+- **SnapshotManager**: Manages website snapshots and version history
+
+### Tool Layer (`src/tools/`)
+Tools are organized in subdirectories by category:
+- `advanced/` - BatchScrapeTool, ScrapeWithActionsTool
+- `crawl/` - crawlDeep, mapSite
+- `extract/` - analyzeContent, extractContent, processDocument, summarizeContent
+- `research/` - deepResearch
+- `search/` - searchWeb and provider adapters (Google, DuckDuckGo)
+- `tracking/` - trackChanges
+
+### MCP Server Entry Point
+The main server implementation is in `server.js` which:
+1. Uses stdio transport for MCP protocol communication
+2. Registers all 16 tools using `server.registerTool()` pattern
+3. Each tool has inline Zod schema for parameter validation
+4. Parameter extraction from `request.params?.arguments` structure
+5. Response format uses `content` array with text objects
+
+### Key Configuration
+
+Critical environment variables:
+
+```bash
+# Search Provider (auto, google, duckduckgo)
+SEARCH_PROVIDER=auto
+
+# Google API (optional, only if using Google)
+GOOGLE_API_KEY=your_key
+GOOGLE_SEARCH_ENGINE_ID=your_id
+
+# Performance Settings
+MAX_WORKERS=10
+QUEUE_CONCURRENCY=10
+CACHE_TTL=3600000
+RATE_LIMIT_REQUESTS_PER_SECOND=10
+
+# Crawling Limits
+MAX_CRAWL_DEPTH=5
+MAX_PAGES_PER_CRAWL=100
+RESPECT_ROBOTS_TXT=true
+```
+
+## Common Development Tasks
+
+### Running a Single Test
+```bash
+# Run a specific test file
+node tests/unit/linkAnalyzer.test.js
+
+# Run a specific Wave test
+node tests/validation/test-batch-scrape.js
+
+# Run Wave 3 tests with verbose output
+npm run test:wave3:verbose
+```
+
+### Testing Tool Integration
+```bash
+# Test MCP protocol compliance
+npm test
+
+# Test specific tool functionality
+node tests/validation/test-batch-scrape.js
+node tests/validation/test-scrape-with-actions.js
+
+# Test research features
+node tests/validation/wave3-validation.js
+```
+
+### Debugging Tips
+- Server logs are written to console via Winston logger
+- Set `NODE_ENV=development` for verbose logging
+- Use `--expose-gc` flag for memory profiling tests
+- Check `cache/` directory for cached responses
+- Review `logs/` directory for application logs
+
+## CI/CD Security Integration
+
+### Automated Security Testing Pipeline
+
+The project includes comprehensive security testing integrated into the CI/CD pipeline:
+
+#### Main CI Pipeline (`.github/workflows/ci.yml`)
+The CI pipeline runs on every PR and push to main/develop branches and includes:
+
+**Security Test Suite:**
+- SSRF Protection validation
+- Input validation (XSS, SQL injection, command injection)
+- Rate limiting functionality
+- DoS protection measures
+- Regex DoS vulnerability detection
+
+**Dependency Security:**
+- npm audit with JSON output and summary generation
+- Vulnerability severity analysis (critical/high/moderate/low)
+- License compliance checking
+- Outdated package detection
+
+**Static Code Analysis:**
+- CodeQL security analysis with extended queries
+- ESLint security rules for dangerous patterns
+- Hardcoded secret detection
+- Security file scanning
+
+**Reporting & Artifacts:**
+- Comprehensive security reports generated
+- PR comments with security summaries
+- Artifact upload for detailed analysis
+- Build failure on critical vulnerabilities
+
+#### Dedicated Security Workflow (`.github/workflows/security.yml`)
+Daily scheduled comprehensive security scanning:
+
+**Dependency Security Scan:**
+- Full vulnerability audit with configurable severity levels
+- License compliance verification
+- Detailed vulnerability reporting
+
+**Static Code Analysis:**
+- Extended CodeQL analysis with security-focused queries
+- ESLint security plugin integration
+- Pattern-based secret detection
+
+**Container Security:**
+- Trivy vulnerability scanning
+- SARIF report generation
+- Container base image analysis
+
+**Automated Issue Creation:**
+- GitHub issues created for critical vulnerabilities
+- Detailed security reports with remediation steps
+- Configurable severity thresholds
+
+### Security Thresholds and Policies
+
+**Build Failure Conditions:**
+- Any critical severity vulnerabilities
+- More than 3 high severity vulnerabilities
+- Security test suite failures
+
+**Automated Actions:**
+- Daily security scans at 2 AM UTC
+- PR blocking for security failures
+- Automatic security issue creation
+- Comprehensive artifact collection
+
+### Running Security Tests Locally
+
+```bash
+# Run the complete security test suite
+npm run test:security
+
+# Check for dependency vulnerabilities
+npm audit --audit-level moderate
+
+# Fix automatically resolvable vulnerabilities
+npm audit fix
+
+# Generate security report manually
+mkdir security-results
+npm audit --json > security-results/audit.json
+
+# Run specific security validation
+node tests/security/security-test-suite.js
+```
+
+### Security Artifacts and Reports
+
+**Generated Reports:**
+- `SECURITY-REPORT.md`: Comprehensive security assessment
+- `npm-audit.json`: Detailed vulnerability data
+- `security-tests.log`: Test execution logs
+- `dependency-analysis.md`: Package security analysis
+- `license-check.md`: License compliance report
+
+**Artifact Retention:**
+- CI security results: 30 days
+- Comprehensive security reports: 90 days
+- Critical vulnerability reports: Indefinite
+
+### Manual Security Scan Triggers
+
+The security workflow can be manually triggered with custom parameters:
+
+```bash
+# Via GitHub CLI
+gh workflow run security.yml \
+  --field scan_type=all \
+  --field severity_threshold=moderate
+
+# Via GitHub UI
+# Go to Actions > Security Scanning > Run workflow
+```
+
+**Available Options:**
+- `scan_type`: all, dependencies, code-analysis, container-scan
+- `severity_threshold`: low, moderate, high, critical
+
+### Security Integration Best Practices
+
+**For Contributors:**
+1. Always run `npm run test:security` before submitting PRs
+2. Address any security warnings in your code
+3. Keep dependencies updated with `npm audit fix`
+4. Review security artifacts when CI fails
+
+**For Maintainers:**
+1. Review security reports weekly
+2. Respond to automated security issues promptly
+3. Keep security thresholds updated
+4. Monitor trending vulnerabilities in dependencies
+
+### Security Documentation
+
+Comprehensive security documentation is available in:
+- `.github/SECURITY.md` - Complete security policy and procedures
+- Security workflow logs and artifacts
+- Generated security reports in CI runs
+
+The security integration ensures that:
+- No critical vulnerabilities reach production
+- Security issues are detected early in development
+- Comprehensive audit trails are maintained
+- Automated remediation guidance is provided
+

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 CrawlForge
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,181 @@
+# CrawlForge MCP Server
+
+Professional web scraping and content extraction server implementing the Model Context Protocol (MCP). Get started with **1,000 free credits** - no credit card required!
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Node.js Version](https://img.shields.io/badge/node-%3E%3D18.0.0-brightgreen)](https://nodejs.org/)
+[![MCP Protocol](https://img.shields.io/badge/MCP-Compatible-blue)](https://modelcontextprotocol.io/)
+[![npm version](https://img.shields.io/npm/v/crawlforge-mcp-server.svg)](https://www.npmjs.com/package/crawlforge-mcp-server)
+
+## 🎯 Features
+
+- **18+ Advanced Tools**: Web scraping, deep research, stealth browsing, content analysis
+- **Free Tier**: 1,000 credits to get started instantly
+- **MCP Compatible**: Works with Claude, Cursor, and other MCP-enabled AI tools
+- **Enterprise Ready**: Scale up with paid plans for production use
+- **Credit-Based**: Pay only for what you use
+
+## 🚀 Quick Start (2 Minutes)
+
+### 1. Install from NPM
+
+```bash
+npm install -g crawlforge-mcp-server
+```
+
+### 2. Setup Your API Key
+
+```bash
+npx crawlforge-setup
+```
+
+This will:
+- Guide you through getting your free API key
+- Configure your credentials securely
+- Verify your setup is working
+
+**Don't have an API key?** Get one free at [https://crawlforge.com/signup](https://crawlforge.com/signup)
+
+### 3. Configure Your IDE
+
+<details>
+<summary>🤖 For Claude Desktop</summary>
+
+Add to `claude_desktop_config.json`:
+```json
+{
+  "mcpServers": {
+    "crawlforge": {
+      "command": "npx",
+      "args": ["crawlforge-mcp-server"]
+    }
+  }
+}
+```
+
+**Location:**
+- macOS: `~/Library/Application Support/Claude/claude_desktop_config.json`
+- Windows: `%APPDATA%/Claude/claude_desktop_config.json`
+- Linux: `~/.config/Claude/claude_desktop_config.json`
+
+Restart Claude Desktop to activate.
+</details>
+
+<details>
+<summary>💻 For Cursor IDE</summary>
+
+Add to `.cursorrules` in your project:
+```bash
+mcp_servers:
+  crawlforge:
+    command: npx
+    args: ["crawlforge-mcp-server"]
+```
+
+Or use the MCP plugin in Cursor settings.
+</details>
+
+## 📊 Available Tools
+
+### Basic Tools (1 credit each)
+- `fetch_url` - Fetch content from any URL
+- `extract_text` - Extract clean text from web pages
+- `extract_links` - Get all links from a page
+- `extract_metadata` - Extract page metadata
+
+### Advanced Tools (2-3 credits)
+- `scrape_structured` - Extract structured data with CSS selectors
+- `search_web` - Search the web with Google/DuckDuckGo
+- `summarize_content` - Generate intelligent summaries
+- `analyze_content` - Comprehensive content analysis
+
+### Premium Tools (5-10 credits)
+- `crawl_deep` - Deep crawl entire websites
+- `map_site` - Discover and map website structure
+- `batch_scrape` - Process multiple URLs simultaneously
+- `deep_research` - Multi-stage research with source verification
+- `stealth_mode` - Anti-detection browser management
+
+### Heavy Processing (3-10 credits)
+- `process_document` - Multi-format document processing
+- `extract_content` - Enhanced content extraction
+- `scrape_with_actions` - Browser automation chains
+- `generate_llms_txt` - Generate AI interaction guidelines
+- `localization` - Multi-language and geo-location management
+
+## 💳 Pricing
+
+| Plan | Credits/Month | Price | Best For |
+|------|---------------|-------|----------|
+| **Free** | 1,000 | $0 | Testing & personal projects |
+| **Hobby** | 10,000 | $19 | Small projects |
+| **Pro** | 50,000 | $49 | Professional use |
+| **Business** | 200,000 | $149 | Teams & automation |
+| **Enterprise** | Unlimited | Custom | Large scale operations |
+
+[View full pricing](https://crawlforge.com/pricing)
+
+## 🔧 Advanced Configuration
+
+### Environment Variables
+
+```bash
+# Optional: Set API key via environment
+export CRAWLFORGE_API_KEY="sk_live_your_api_key_here"
+
+# Optional: Custom API endpoint (for enterprise)
+export CRAWLFORGE_API_URL="https://api.crawlforge.com"
+```
+
+### Manual Configuration
+
+Your configuration is stored at `~/.crawlforge/config.json`:
+
+```json
+{
+  "apiKey": "sk_live_...",
+  "userId": "user_...",
+  "email": "you@example.com"
+}
+```
+
+## 📖 Usage Examples
+
+Once configured, use these tools in your AI assistant:
+
+```
+"Search for the latest AI news"
+"Extract all links from example.com"
+"Crawl the documentation site and summarize it"
+"Monitor this page for changes"
+"Extract product prices from this e-commerce site"
+```
+
+## 🔒 Security & Privacy
+
+- API keys are stored locally and encrypted
+- All connections use HTTPS
+- No data is stored on our servers beyond usage logs
+- Compliant with robots.txt and rate limits
+- GDPR compliant
+
+## 🆘 Support
+
+- **Documentation**: [https://crawlforge.com/docs](https://crawlforge.com/docs)
+- **Issues**: [GitHub Issues](https://github.com/crawlforge/mcp-server/issues)
+- **Email**: support@crawlforge.com
+- **Discord**: [Join our community](https://discord.gg/crawlforge)
+
+## 📄 License
+
+MIT License - see [LICENSE](LICENSE) file for details.
+
+## 🤝 Contributing
+
+Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) first.
+
+---
+
+**Built with ❤️ by the CrawlForge team**
+
+[Website](https://crawlforge.com) | [Documentation](https://crawlforge.com/docs) | [API Reference](https://crawlforge.com/api)

package/package.json

@@ -0,0 +1,115 @@
+{
+  "name": "crawlforge-mcp-server",
+  "version": "3.0.0",
+  "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 16+ comprehensive web scraping, crawling, and content processing tools.",
+  "main": "server.js",
+  "bin": {
+    "crawlforge": "server.js",
+    "crawlforge-setup": "setup.js"
+  },
+  "scripts": {
+    "start": "node server.js",
+    "setup": "node setup.js",
+    "dev": "cross-env NODE_ENV=development node server.js",
+    "test": "node tests/integration/mcp-protocol-compliance.test.js",
+    "postinstall": "echo '\n🎉 CrawlForge MCP Server installed!\n\nRun \"npx crawlforge-setup\" to configure your API key and get started.\n'",
+    "docker:build": "docker build -t crawlforge .",
+    "docker:dev": "docker-compose up crawlforge-dev",
+    "docker:prod": "docker-compose up crawlforge-prod"
+  },
+  "keywords": [
+    "mcp",
+    "model-context-protocol",
+    "web-scraping",
+    "scraper",
+    "html-parser",
+    "metadata-extraction",
+    "link-extraction",
+    "content-processing",
+    "document-analysis",
+    "web-crawler",
+    "search-engine",
+    "text-summarization",
+    "content-analysis",
+    "nlp",
+    "ai-tools",
+    "automation",
+    "data-extraction",
+    "pdf-processing",
+    "sitemap-parser",
+    "performance-optimized",
+    "llms-txt",
+    "llms-txt-generator",
+    "ai-compliance",
+    "website-analysis"
+  ],
+  "author": {
+    "name": "Simon Lacey",
+    "email": "your-email@example.com"
+  },
+  "license": "MIT",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/crawlforge/mcp-server.git"
+  },
+  "bugs": {
+    "url": "https://github.com/crawlforge/mcp-server/issues"
+  },
+  "homepage": "https://crawlforge.com",
+  "type": "module",
+  "engines": {
+    "node": ">=18.0.0",
+    "npm": ">=8.0.0"
+  },
+  "os": [
+    "linux",
+    "darwin",
+    "win32"
+  ],
+  "cpu": [
+    "x64",
+    "arm64"
+  ],
+  "publishConfig": {
+    "access": "public",
+    "registry": "https://registry.npmjs.org/",
+    "tag": "latest"
+  },
+  "files": [
+    "server.js",
+    "setup.js",
+    "src/",
+    "README.md",
+    "LICENSE",
+    "CLAUDE.md",
+    "package.json"
+  ],
+  "dependencies": {
+    "@googleapis/customsearch": "^5.0.1",
+    "@modelcontextprotocol/sdk": "^1.17.3",
+    "@mozilla/readability": "^0.6.0",
+    "cheerio": "^1.1.2",
+    "compromise": "^14.14.4",
+    "diff": "^8.0.2",
+    "dotenv": "^17.2.1",
+    "franc": "^6.2.0",
+    "isomorphic-dompurify": "^2.26.0",
+    "jsdom": "^26.1.0",
+    "lru-cache": "^11.1.0",
+    "node-cron": "^3.0.3",
+    "node-fetch": "^3.3.2",
+    "node-summarizer": "^1.0.7",
+    "p-queue": "^8.1.0",
+    "pdf-parse": "^1.1.1",
+    "playwright": "^1.54.2",
+    "robots-parser": "^3.0.1",
+    "winston": "^3.11.0",
+    "zod": "^3.23.8"
+  },
+  "devDependencies": {
+    "@jest/globals": "^30.0.5",
+    "cross-env": "^10.0.0",
+    "jest": "^30.0.5",
+    "shx": "^0.4.0"
+  }
+}