crawlforge-mcp-server 4.9.0 → 5.0.0

package/CLAUDE.md

@@ -62,7 +62,7 @@ These guidelines are working if: fewer unnecessary changes in diffs, fewer rewri
 
 CrawlForge MCP Server - A professional MCP (Model Context Protocol) server providing 27 web scraping, crawling, and content processing tools (5 inline + 22 advanced).
 
-**Current Version:** 4.9.0
+**Current Version:** 5.0.0
 
 ## Development Commands
 
@@ -96,9 +96,10 @@ npm test
 npm run test:unit
 # Phase D regressions live in tests/unit/phaseD-regressions.test.js (agent hard stops, unified scrape, map_site ranking)
 # Run a single test file:  node --test tests/unit/phaseD-regressions.test.js
-# Note: add --test-force-exit if the run appears to hang at the end — importing
-# StealthBrowserManager (d2-reliability.test.js) leaves a Playwright handle that
-# otherwise delays process exit ~100s. Tests themselves pass either way.
+# Note: the script includes --test-force-exit — importing StealthBrowserManager
+# (d2-reliability.test.js) leaves a Playwright handle that otherwise hangs the
+# runner at exit after all tests pass. When running a single file manually, put
+# --test-force-exit BEFORE the file path (after it, the flag is silently ignored).
 
 # Integration tests
 npm run test:integration
@@ -260,7 +261,7 @@ Security testing and CI/CD pipeline details are in:
 - `docs/security-audit-report.md` — Full security audit
 - `.github/workflows/ci.yml` — CI pipeline with security checks
 - `.github/workflows/security.yml` — Daily scheduled security scanning
-- `.github/SECURITY.md` — Security policy and procedures
+- `docs/SECURITY.md` — Security policy and procedures
 
 Run `npm audit` locally to check dependencies.
 

package/README.md

@@ -184,7 +184,7 @@ CrawlForge requires a CrawlForge API key — **every tool is metered and consume
 | `agent` | 8 | **Autonomous research/extraction from a natural-language prompt — no URLs required.** Plans, gathers, and shapes an answer under hard safety stops (max steps/URLs/wall-clock enforced by the orchestrator, never the LLM) |
 | `deep_research` | 10 | Multi-stage research with source verification |
 
-For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see [SKILL.md](SKILL.md).
+For the full canonical capabilities reference (all tools, CLI commands, stealth engines, research workflow), see [SKILL.md](docs/SKILL.md).
 
 <p align="right"><a href="#table-of-contents">↑ Back to top</a></p>
 
@@ -226,12 +226,28 @@ export OLLAMA_DEFAULT_MODEL="llama3.2"             # default; any locally-pulled
 export OPENAI_API_KEY="sk-..."
 export ANTHROPIC_API_KEY="sk-ant-..."
 
+# Optional: limit which tools this client sees — by name, by group, or both (comma-separated)
+export CRAWLFORGE_TOOLS="scrape,search_web,extract_content"
+export CRAWLFORGE_TOOL_GROUPS="basic,search,scrape"   # unset = all tools; unknown names/groups are ignored with a warning
+
 # Optional: deep_research stealth extraction fallback (v4.6.6) — see below
 export RESEARCH_STEALTH_ENGINE="auto"      # auto (default) | camoufox | chromium
 export RESEARCH_STEALTH_FALLBACK="true"    # set to "false" to disable entirely
 export RESEARCH_MAX_STEALTH_RETRIES="8"    # cap on stealth retries per research run
 ```
 
+### MCP Spec Features
+
+CrawlForge tracks the current MCP spec (2025-06-18) plus select experimental extensions:
+
+- **Structured output** — `scrape`, `map_site`, `serp_rank`, `search_web`, `extract_structured`, and `crawl_deep` return machine-parseable `structuredContent` alongside the usual text, validated against a published `outputSchema`; legacy clients keep working off the text.
+- **Self-correctable errors** — invalid tool input now comes back as an `isError: true` result the calling model can read and retry from, instead of a raw JSON-RPC protocol error.
+- **JSON Schema 2020-12** tool schemas, deterministic `tools/list` ordering (client prompt-cache friendly), and cacheable-result hints on read-only tools.
+- **Icons** on the server, its tools, and its prompts.
+- **Async tasks** (experimental) on the four long-running tools — `crawl_deep`, `batch_scrape`, `deep_research`, `agent` — for clients that support polling; synchronous results are still returned for clients that don't.
+
+See [docs/mcp-spec-adoption.md](docs/mcp-spec-adoption.md) for wire-level examples and client-compatibility notes.
+
 ### Local-LLM quickstart (`extract_with_llm` with Ollama)
 
 `extract_with_llm` defaults to a local Ollama model — no LLM-provider key, no per-token LLM costs, and no data leaving your machine (the CrawlForge credit cost still applies).
@@ -321,7 +337,7 @@ See [docs/sandboxing-and-approvals.md](docs/sandboxing-and-approvals.md) for the
 
 **v3.0.3 (2025-10-01)**: Removed authentication bypass vulnerability. All users must authenticate with valid API keys.
 
-For the full security policy and how to report a vulnerability, see [SECURITY.md](SECURITY.md).
+For the full security policy and how to report a vulnerability, see [SECURITY.md](docs/SECURITY.md).
 
 <p align="right"><a href="#table-of-contents">↑ Back to top</a></p>
 
@@ -338,7 +354,7 @@ MIT License - see [LICENSE](LICENSE) file for details.
 
 ## 🤝 Contributing
 
-Contributions are welcome! Please read our [Contributing Guide](CONTRIBUTING.md) first.
+Contributions are welcome! Please read our [Contributing Guide](docs/CONTRIBUTING.md) first.
 
 ---
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "crawlforge-mcp-server",
-  "version": "4.9.0",
+  "version": "5.0.0",
   "mcpName": "io.github.mysleekdesigns/crawlforge-mcp-server",
   "description": "CrawlForge MCP Server - Professional Model Context Protocol server with 27 web scraping, crawling, deep-research, and autonomous-extraction tools. Returns clean Markdown and structured JSON for Claude, Cursor, and any MCP client. Defaults to local Ollama for LLM extraction (no API key needed); OpenAI/Anthropic available as opt-in. Includes a unified multi-format scrape tool, an autonomous agent, pre-built site templates, and Camoufox stealth browsing.",
   "main": "server.js",
@@ -16,7 +16,7 @@
     "setup": "node setup.js",
     "dev": "cross-env NODE_ENV=development node server.js",
     "test": "node tests/integration/mcp-protocol-compliance.test.js",
-    "test:unit": "CRAWLFORGE_CREATOR_SECRET= node --test 'tests/unit/*.test.js'",
+    "test:unit": "CRAWLFORGE_CREATOR_SECRET= node --test --test-force-exit 'tests/unit/**/*.test.js'",
     "test:integration": "CRAWLFORGE_CREATOR_SECRET= node --test 'tests/integration/tools/*.test.js'",
     "test:coverage": "CRAWLFORGE_CREATOR_SECRET= c8 --reporter=text --reporter=lcov --include='src/**/*.js' --exclude='src/**/_*.js' --lines=60 --statements=60 --functions=55 --branches=45 node --test --test-force-exit 'tests/unit/*.test.js' 'tests/integration/tools/*.test.js'",
     "test:tools": "node test-tools.js",
@@ -81,7 +81,7 @@
   "homepage": "https://crawlforge.dev",
   "type": "module",
   "engines": {
-    "node": ">=18.0.0",
+    "node": ">=20.16.0",
     "npm": ">=8.0.0"
   },
   "os": [
@@ -108,22 +108,19 @@
     "package.json"
   ],
   "dependencies": {
-    "@googleapis/customsearch": "^5.0.1",
     "@modelcontextprotocol/sdk": "^1.29.0",
     "@mozilla/readability": "^0.6.0",
     "cheerio": "^1.1.2",
-    "commander": "^12.1.0",
+    "commander": "^14.0.3",
     "compromise": "^14.14.4",
-    "diff": "^8.0.2",
+    "diff": "^9.0.0",
     "dotenv": "^17.2.1",
     "franc": "^6.2.0",
     "isomorphic-dompurify": "^3.9.0",
     "jsdom": "^29.0.2",
     "lru-cache": "^11.1.0",
-    "node-cron": "^3.0.3",
-    "node-summarizer": "^1.0.7",
-    "p-queue": "^8.1.0",
-    "pdf-parse": "^1.1.1",
+    "p-queue": "^9.3.3",
+    "pdf-parse": "2.4.5",
     "playwright": "^1.54.2",
     "robots-parser": "^3.0.1",
     "turndown": "^7.2.4",
@@ -147,8 +144,9 @@
     "underscore": "^1.13.8",
     "qs": "^6.14.2",
     "path-to-regexp": "^8.4.2",
-    "@hono/node-server": "^1.19.13",
+    "@hono/node-server": "2.0.12",
     "hono": "^4.12.4",
-    "dompurify": "^3.4.0"
+    "dompurify": "^3.4.0",
+    "adm-zip": "^0.6.0"
   }
 }