scrape-do-mcp 0.1.2 → 0.1.4

package/README-ZH.md

@@ -0,0 +1,146 @@
+# scrape-do-mcp
+
+Scrape.do 网页抓取和 Google 搜索 MCP 服务器 - 支持反机器人保护
+
+## 功能特点
+
+- **scrape_url**: 抓取任意网页并返回 Markdown 格式内容。自动绕过 Cloudflare、WAF、CAPTCHA 和反爬虫保护。支持 JavaScript 渲染页面。
+- **google_search**: 搜索 Google 并返回结构化的 SERP 结果 JSON。包含自然搜索结果、知识图谱、本地商家、新闻、相关问题（People Also Ask）等。
+
+## 安装
+
+### 快速安装（推荐）
+
+在终端中运行以下命令：
+
+```bash
+claude mcp add-json scrape-do --scope user '{
+  "type": "stdio",
+  "command": "npx",
+  "args": ["-y", "scrape-do-mcp"],
+  "env": {
+    "SCRAPE_DO_TOKEN": "你的Token"
+  }
+}'
+```
+
+将 `你的Token` 替换为你在 https://app.scrape.do 获取的 API Token。
+
+### Claude Desktop
+
+添加到 `~/.claude.json`：
+
+```json
+{
+  "mcpServers": {
+    "scrape-do": {
+      "command": "npx",
+      "args": ["-y", "scrape-do-mcp"],
+      "env": {
+        "SCRAPE_DO_TOKEN": "你的Token"
+      }
+    }
+  }
+}
+```
+
+获取免费 API Token：https://app.scrape.do
+
+## 使用方法
+
+### scrape_url
+
+抓取任意网页并获取 Markdown 内容。
+
+```typescript
+// 参数
+{
+  url: string,           // 要抓取的网址
+  render_js?: boolean,  // 渲染 JavaScript（默认 false）
+  super_proxy?: boolean, // 使用住宅代理（消耗 10 积分，默认 false）
+  output?: "markdown" | "raw"  // 输出格式（默认 markdown）
+}
+```
+
+### google_search
+
+搜索 Google 并获取结构化结果。
+
+```typescript
+// 参数
+{
+  query: string,                      // 搜索关键词
+  country?: string,                   // 国家代码（默认 "us"）
+  language?: string,                  // 界面语言（默认 "en"）
+  page?: number,                      // 页码（默认 1）
+  time_period?: "" | "last_hour" | "last_day" | "last_week" | "last_month" | "last_year",
+  device?: "desktop" | "mobile"       // 设备类型（默认 desktop）
+}
+```
+
+## 使用示例
+
+### 抓取网页
+```
+请抓取 https://github.com 并给我主要内容（Markdown 格式）。
+```
+
+### Google 搜索
+```
+搜索 "2026 年最佳 Python Web 框架"，返回前 5 个结果。
+```
+
+### 带筛选条件的搜索
+```
+用中文搜索 "AI 新闻"，限定为中国，过去一周的内容。
+```
+
+### JavaScript 渲染
+```
+抓取这个 React 单页应用：https://example-spa.com
+使用 render_js=true 获取完整渲染内容。
+```
+
+### 获取原始 HTML
+```
+抓取 https://example.com 并返回原始 HTML 而不是 markdown。
+```
+
+## 与其他工具对比
+
+| 功能 | scrape-do-mcp | Firecrawl | Browserbase |
+|------|--------------|-----------|-------------|
+| Google 搜索 | ✅ | ❌ | ❌ |
+| 免费积分 | 1,000 | 500 | 无 |
+| 价格 | 按量付费 | $19+/月 | $15+/月 |
+| MCP 原生 | ✅ | ✅ | ❌ |
+| 配置难度 | 无需配置 | 需要 API key | 需要 API key + 浏览器 |
+
+### 为什么选择 scrape-do-mcp？
+
+- **零配置**：获取 Token 后即可立即使用
+- **一体化**：网页抓取和 Google 搜索集于一个 MCP
+- **反爬虫绕过**：自动处理 Cloudflare、WAF、CAPTCHA
+- **成本效益**：按需付费，免费额度可用
+
+## 积分消耗
+
+| 工具 | 积分消耗 |
+|------|---------|
+| scrape_url（普通） | 1 积分/次 |
+| scrape_url（super_proxy） | 10 积分/次 |
+| google_search | 1 积分/次 |
+
+注册即送 **1,000 积分**：https://app.scrape.do
+
+## 开发
+
+```bash
+npm install
+npm run build
+npm run dev  # 开发模式运行
+```
+
+## 许可证
+
+MIT

package/README.md

@@ -9,7 +9,24 @@ MCP Server for Scrape.do - Web Scraping & Google Search with anti-bot bypass
 
 ## Installation
 
-### Claude Code / Claude Desktop
+### Quick Install (Recommended)
+
+Run this command in your terminal:
+
+```bash
+claude mcp add-json scrape-do --scope user '{
+  "type": "stdio",
+  "command": "npx",
+  "args": ["-y", "scrape-do-mcp"],
+  "env": {
+    "SCRAPE_DO_TOKEN": "YOUR_TOKEN_HERE"
+  }
+}'
+```
+
+Replace `YOUR_TOKEN_HERE` with your Scrape.do API token from https://app.scrape.do
+
+### Claude Desktop
 
 Add to your `~/.claude.json`:
 
@@ -29,29 +46,12 @@ Add to your `~/.claude.json`:
 
 Get your free API token at: https://app.scrape.do
 
-### Smithery.ai
-
-Published on [Smithery.ai](https://smithery.ai) - Search for "scrape-do" to install.
-
-### HTTP Server Mode
-
-The server supports both STDIO and HTTP modes:
-
-- **STDIO mode** (default): For local Claude Code / Claude Desktop usage
-- **HTTP mode**: For Smithery托管或 custom HTTP deployment
-
-```bash
-# HTTP mode
-TRANSPORT=http PORT=3000 SCRAPE_DO_TOKEN=your_token npm start
-
-# Health check
-curl http://localhost:3000/health
-```
-
 ## Usage
 
 ### scrape_url
 
+Scrape any webpage and get content as Markdown.
+
 ```typescript
 // Parameters
 {
@@ -64,6 +64,8 @@ curl http://localhost:3000/health
 
 ### google_search
 
+Search Google and get structured results.
+
 ```typescript
 // Parameters
 {
@@ -76,6 +78,53 @@ curl http://localhost:3000/health
 }
 ```
 
+## Example Prompts
+
+Here are some prompts you can use to invoke the tools:
+
+### Scrape a Website
+```
+Please scrape https://github.com and give me the main content as markdown.
+```
+
+### Search Google
+```
+Search Google for "best Python web frameworks 2026" and return the top 5 results.
+```
+
+### Search with Filters
+```
+Search for "AI news" in Chinese, from China, last week.
+```
+
+### JavaScript Rendering
+```
+Scrape this React Single Page Application: https://example-spa.com
+Use render_js=true to get the fully rendered content.
+```
+
+### Get Raw HTML
+```
+Scrape https://example.com and return raw HTML instead of markdown.
+```
+
+## Comparison with Alternatives
+
+| Feature | scrape-do-mcp | Firecrawl | Browserbase |
+|---------|--------------|-----------|-------------|
+| Google Search | ✅ | ❌ | ❌ |
+| Free Credits | 1,000 | 500 | None |
+| Pricing | Pay per use | $19+/mo | $15+/mo |
+| MCP Native | ✅ | ✅ | ❌ |
+| Setup Required | None | API key | API key + browser |
+
+### Why scrape-do-mcp?
+
+- **Zero setup**: Just get a token and use immediately
+- **All-in-one**: Both web scraping AND Google search in one MCP
+- **Anti-bot bypass**: Automatically handles Cloudflare, WAFs, CAPTCHAs
+- **Cost-effective**: Pay only for what you use, free tier available
+
 ## Credit Usage
 
 | Tool | Credit Cost |

package/dist/index.js

@@ -6,16 +6,13 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
 Object.defineProperty(exports, "__esModule", { value: true });
 const mcp_js_1 = require("@modelcontextprotocol/sdk/server/mcp.js");
 const stdio_js_1 = require("@modelcontextprotocol/sdk/server/stdio.js");
-const streamableHttp_js_1 = require("@modelcontextprotocol/sdk/server/streamableHttp.js");
 const zod_1 = require("zod");
 const axios_1 = __importDefault(require("axios"));
-const http_1 = __importDefault(require("http"));
 const SCRAPE_DO_TOKEN = process.env.SCRAPE_DO_TOKEN || "";
 const SCRAPE_API_BASE = "https://api.scrape.do";
-const HTTP_PORT = process.env.PORT || process.env.HTTP_PORT || 3000;
 const server = new mcp_js_1.McpServer({
     name: "scrape-do-mcp",
-    version: "0.1.1",
+    version: "0.1.3",
 });
 // ─── Tool 1: scrape_url ──────────────────────────────────────────────────────
 server.tool("scrape_url", "Scrape any webpage and return its content as Markdown. Automatically bypasses Cloudflare, WAFs, CAPTCHAs, and anti-bot protection. Supports JavaScript-rendered pages.", {
@@ -97,47 +94,7 @@ server.tool("google_search", "Search Google and return structured SERP results a
 });
 // ─── Start Server ────────────────────────────────────────────────────────────
 async function main() {
-    const transportMode = process.env.TRANSPORT || "stdio";
-    if (transportMode === "http" || transportMode === "streamable-http") {
-        console.error(`Starting Streamable HTTP server on port ${HTTP_PORT}...`);
-        const transport = new streamableHttp_js_1.StreamableHTTPServerTransport({
-            sessionIdGenerator: () => Math.random().toString(36).substring(2, 15),
-        });
-        await server.connect(transport);
-        const serverInstance = http_1.default.createServer();
-        serverInstance.on("request", async (req, res) => {
-            // Handle CORS
-            res.setHeader("Access-Control-Allow-Origin", "*");
-            res.setHeader("Access-Control-Allow-Methods", "GET, POST, OPTIONS");
-            res.setHeader("Access-Control-Allow-Headers", "Content-Type");
-            if (req.method === "OPTIONS") {
-                res.writeHead(204);
-                res.end();
-                return;
-            }
-            // Health check
-            if (req.url === "/health") {
-                res.writeHead(200, { "Content-Type": "application/json" });
-                res.end(JSON.stringify({ status: "ok", name: "scrape-do-mcp", version: "0.1.1" }));
-                return;
-            }
-            // MCP endpoint
-            if (req.url === "/" || req.url?.startsWith("/mcp")) {
-                await transport.handleRequest(req, res);
-                return;
-            }
-            res.writeHead(404);
-            res.end("Not found");
-        });
-        serverInstance.listen(parseInt(String(HTTP_PORT), 10), () => {
-            console.error(`MCP server running on http://localhost:${HTTP_PORT}`);
-        });
-    }
-    else {
-        // Default to stdio mode
-        console.error("Starting STDIO server...");
-        const transport = new stdio_js_1.StdioServerTransport();
-        await server.connect(transport);
-    }
+    const transport = new stdio_js_1.StdioServerTransport();
+    await server.connect(transport);
 }
 main().catch(console.error);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "scrape-do-mcp",
-  "version": "0.1.2",
+  "version": "0.1.4",
   "description": "MCP Server for Scrape.do - Web Scraping & Google Search with anti-bot bypass",
   "main": "dist/index.js",
   "bin": {
@@ -25,7 +25,8 @@
   },
   "files": [
     "dist",
-    "README.md"
+    "README.md",
+    "README-ZH.md"
   ],
   "engines": {
     "node": ">=18.0.0"