mcp-scraper 0.2.18 → 0.2.20

package/README.md

@@ -10,19 +10,25 @@ MCP Scraper ships three local stdio entrypoints plus human-facing helper CLIs:
 
 - `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook ads and organic video transcripts, Maps, directory, rank tracker blueprint, and credit tools.
 - `browser-agent` — an agent-controlled live cloud browser with screenshots, clicks, typing, scrolling, live watch URLs, replay links, and MP4 replay download.
-- `mcp-scraper-combined` — one local MCP server that exposes both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
-- `mcp-scraper-install` — a human-facing terminal installer card with the branded ASCII intro and copyable install commands. This command is safe to print because it is not an MCP stdio server.
+- `mcp-scraper-combined` — one context-aware command. In a human terminal it prints the branded ASCII install card; in an MCP client it runs the combined stdio server with both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
+- `mcp-scraper-install` — explicit alias for the human-facing terminal installer card with the branded ASCII intro and copyable install commands.
 - `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, local SEO workflow runs, and local HTML reports. This command is safe to print because it is not an MCP stdio server.
 
 ### Terminal installer
 
-Run this when you want the designed terminal install experience:
+Run the combined command when you want the designed terminal install experience:
 
 ```bash
-npx -y -p mcp-scraper@latest mcp-scraper-install
+npx -y -p mcp-scraper@latest mcp-scraper-combined
 ```
 
-It prints the MCP Scraper banner, loaded tool groups, Desktop Extension download, Claude Code command, and Codex config. This is the command to use when you want the ASCII terminal card. The MCP server commands, including `mcp-scraper-combined`, stay silent so stdout remains valid JSON-RPC for MCP clients.
+In a human terminal, it prints the MCP Scraper banner, loaded tool groups, Desktop Extension download, Claude Code command, and Codex config. When launched by an MCP client, the same command detects non-interactive stdio and writes only valid JSON-RPC to stdout.
+
+The explicit installer alias still works:
+
+```bash
+npx -y -p mcp-scraper@latest mcp-scraper-install
+```
 
 ### Human CLI
 
@@ -81,7 +87,7 @@ Build the branded one-click bundle:
 npm run build:mcpb
 ```
 
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.18`, SHA-256 `3b94a39bf23712005a1a04accdefcf3ee24978995033b92e4488390d47350ea0`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
+The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.19`, SHA-256 `b324b7bb2a0dca06978d436da0f69a16ad8b40e91cdc76e19547de2b078b66d3`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
 
 The MCPB install exposes the same web-intelligence tools as `mcp-scraper` plus all `browser_*` tools from `browser-agent` through one server.
 
@@ -189,15 +195,15 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 For accurate annotated videos, do not guess annotation times from a script. Start the replay, navigate until each target is visible and stable, call `browser_replay_mark` for each callout, then stop the replay and pass the returned annotations to `browser_replay_annotate` with the returned `source_width` and `source_height`.
 
-For US local SERP tools (`harvest_paa` and `search_serp`), keep `proxyMode` at the default `location` unless you are debugging. Location mode uses fresh residential proxy IDs across retries. When Google shows a CAPTCHA/challenge, browser-service sessions briefly wait for automatic challenge solving first, then rotate to a new proxy/session if the challenge does not clear. Proxy tunnel failure and wrong-location evidence are also retryable before returning.
+For general SERP tools (`harvest_paa`, `search_serp`, and hosted SERP capture), omit `proxyMode` for normal use. The default is `configured`, which uses the configured browser-service proxy without city/ZIP targeting for the highest general success rate. Use `proxyMode: "location"` only when the user explicitly needs city/ZIP-targeted residential proxy evidence. When Google shows a CAPTCHA/challenge, browser-service sessions briefly wait for automatic challenge solving first, then rotate to a new proxy/session if the challenge does not clear. Proxy tunnel failure and wrong-location evidence are also retryable before returning.
 
 For Google Maps tools (`maps_search` and `directory_workflow`), keep `proxyMode` at the default `location` for US city/state work. Retryable Maps failures such as CAPTCHA, timeout, proxy tunnel failure, proxy unavailability, or browser-session death rotate to a new residential proxy and a new browser session for up to 5 attempts. Successful structured responses include sanitized attempt telemetry so callers can verify the proxy source, observed city/region, and attempt count without exposing full proxy or browser IDs.
 
 The MCPB bundle and `mcp-scraper-combined` expose both sections through one local MCP server. The split `mcp-scraper` entrypoint exposes only the web-intelligence tools, and the split `browser-agent` entrypoint exposes only the browser-agent tools.
 
-Chaining and planning tools (`maps_search`, `directory_workflow`, `workflow_list`, `workflow_suggest`, `workflow_run`, `workflow_status`, `workflow_artifact_read`, `rank_tracker_blueprint`, `map_site_urls`, `youtube_harvest`, `facebook_ad_search`, `facebook_page_intel`, `facebook_video_transcribe`) return `structuredContent` with the IDs, URLs, CSV paths, transcripts, artifacts, recipe fields, or blueprint fields needed by the next step. All tools carry MCP annotations (`readOnlyHint: true`; live-web tools use `openWorldHint: true`, while local planning and credit tools are closed-world/idempotent).
+All local and hosted MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their local file side effects.
 
-The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 21 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (23 total). Browser-agent tools are local stdio tools backed by the REST API under `https://mcpscraper.dev/agent/*`.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split local `mcp-scraper` stdio server exposes 22 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 17 browser tools. The combined local stdio server exposes 39 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 22 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (24 total). Across all public local and hosted surfaces there are 41 unique tool names.
 
 ## Resources
 
@@ -223,7 +229,7 @@ Recommended config for update-friendly installs:
 npx -y -p mcp-scraper@latest mcp-scraper-combined
 ```
 
-This is an MCP stdio server command, not a human-facing terminal UI. If you run it directly, it appears to hang because it is waiting for JSON-RPC input from the MCP client. Use `npx -y -p mcp-scraper@latest mcp-scraper-install` when you want the visible installer and ASCII card.
+This is context-aware: in a normal terminal it prints the visible installer and ASCII card; in an MCP client it runs as the silent stdio server. Use `--stdio` or `MCP_SCRAPER_FORCE_STDIO=1` if you need to force server mode from a terminal.
 
 Split-server config:
 
@@ -243,7 +249,7 @@ Users who do not update can keep using the tools their local package already adv
 
 ## Branded One-Click Installs
 
-Raw `npx` MCP server installs are command/config based. They do not provide a reliable user-facing install card, logo, or setup screen inside MCP clients. Use `mcp-scraper-install` for terminal onboarding text and the ASCII card. Do not print marketing text to stdout from an MCP server; stdout is reserved for JSON-RPC protocol messages.
+Raw `npx` MCP server installs are command/config based. `mcp-scraper-combined` is context-aware: terminal TTY prints onboarding text and the ASCII card; MCP-client stdio pipes stay protocol-clean. Do not print marketing text to stdout from an active MCP stdio session; stdout is reserved for JSON-RPC protocol messages.
 
 For a branded Claude Desktop install, package MCP Scraper as an MCPB Desktop Extension. The repository now builds one combined MCPB bundle with a generated icon, `manifest.json`, bundled runtime dependencies, and `user_config` fields for API-key setup, API URL, and output folder.