mcp-scraper 0.1.9 → 0.2.1

package/README.md

@@ -4,14 +4,26 @@ MCP Scraper is an MCP server for live web intelligence tools backed by `https://
 
 ## Install
 
-Use the npm package from any MCP client that can run a local command:
+Use the npm package from any MCP client that can run local stdio commands. MCP Scraper ships two separate local MCP servers:
+
+- `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook, Maps, directory, and credit tools.
+- `browser-agent` — an agent-controlled live cloud browser with screenshots, clicks, typing, scrolling, live watch URLs, replay links, and MP4 replay download.
+
+Claude Desktop:
 
 ```json
 {
   "mcpServers": {
     "mcp-scraper": {
       "command": "npx",
-      "args": ["-y", "mcp-scraper"],
+        "args": ["-y", "mcp-scraper@latest"],
+      "env": {
+        "MCP_SCRAPER_API_KEY": "sk_live_your_key"
+      }
+    },
+    "browser-agent": {
+      "command": "npx",
+        "args": ["-y", "-p", "mcp-scraper@latest", "browser-agent"],
       "env": {
         "MCP_SCRAPER_API_KEY": "sk_live_your_key"
       }
@@ -20,12 +32,13 @@ Use the npm package from any MCP client that can run a local command:
 }
 ```
 
-Existing MCP configs that use `npx -y mcp-scraper` do not need a command change. After a new npm version is published, restart the MCP client so it starts a fresh `npx` process.
+Existing MCP configs that use only `npx -y mcp-scraper` still work for the web intelligence server, but they do not automatically add the `browser-agent` server. Add the second config entry if you want browser tools. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
 
 Claude Code:
 
 ```bash
-claude mcp add mcp-scraper --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y mcp-scraper
+claude mcp add mcp-scraper --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y mcp-scraper@latest
+claude mcp add browser-agent --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y -p mcp-scraper@latest browser-agent
 ```
 
 Codex config:
@@ -33,12 +46,19 @@ Codex config:
 ```toml
 [mcp_servers.mcp-scraper]
 command = "npx"
-args = ["-y", "mcp-scraper"]
+args = ["-y", "mcp-scraper@latest"]
+env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
+
+[mcp_servers.browser-agent]
+command = "npx"
+args = ["-y", "-p", "mcp-scraper@latest", "browser-agent"]
 env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 ```
 
 ## Tools
 
+### `mcp-scraper` stdio tools
+
 - `harvest_paa`
 - `search_serp`
 - `extract_url`
@@ -51,15 +71,35 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 - `facebook_ad_transcribe`
 - `maps_search` — search Google Maps for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. `maxResults` defaults to 10 and is capped at 50.
 - `maps_place_intel` — hydrate one known/named Google Maps business with profile details and optional reviews. Use after `maps_search` when a selected candidate needs full details.
+- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus Google Maps searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups. It captures Maps star ratings from list cards, not profile review counts.
 - `credits_info`
 
+### `browser-agent` stdio tools
+
+- `browser_open` — open a live cloud browser session. Returns a `session_id`, a human `watch_url`, and the raw `live_view_url` when available.
+- `browser_screenshot` — capture a screenshot plus visible text and clickable element coordinates.
+- `browser_read` — read the current page text and elements without an image.
+- `browser_goto`
+- `browser_click`
+- `browser_type`
+- `browser_scroll`
+- `browser_press`
+- `browser_replay_start` — start an MP4 replay. Returns `replay_id`, `view_url`, and `download_url` when available.
+- `browser_replay_stop` — stop a replay. Returns the final `view_url` and `download_url`.
+- `browser_list_replays` — list replay videos for a session.
+- `browser_replay_download` — download and save the replay MP4 locally under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
+- `browser_close`
+- `browser_list_sessions`
+
+For US local SERP tools (`harvest_paa` and `search_serp`), keep `proxyMode` at the default `location` unless you are debugging. Location mode uses fresh residential proxy IDs across retries and treats CAPTCHA, proxy tunnel failure, and wrong-location evidence as retryable before returning.
+
 Chaining tools (`maps_search`, `map_site_urls`, `youtube_harvest`, `facebook_ad_search`, `facebook_page_intel`) advertise an `outputSchema` and return `structuredContent` with the IDs and URLs needed by the next tool. All tools carry MCP annotations (`readOnlyHint: true`, `openWorldHint: true` for live-web tools).
 
-The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes these 13 tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (15 total).
+The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 14 `mcp-scraper` tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (16 total). The `browser-agent` server is currently a separate local stdio server; its REST backing API lives under `https://mcpscraper.dev/agent/*`.
 
 ## Resources
 
-The NPX stdio server also exposes saved reports as MCP resources: `resources/list` returns the most recent Markdown reports from your output directory as `report://` URIs, and `resources/read` returns their content — so an MCP client can pull prior research into context without re-scraping or spending credits. The hosted endpoint does not expose resources (it saves no files).
+The `mcp-scraper` NPX stdio server also exposes saved reports as MCP resources: `resources/list` returns the most recent Markdown reports from your output directory as `report://` URIs, and `resources/read` returns their content — so an MCP client can pull prior research into context without re-scraping or spending credits. The hosted endpoint does not expose resources (it saves no files).
 
 ## Environment
 
@@ -69,7 +109,33 @@ The NPX stdio server also exposes saved reports as MCP resources: `resources/lis
 - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
 - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
 
-Every tool call made through the NPX stdio server saves a full Markdown report locally by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files.
+Every web intelligence tool call made through the `mcp-scraper` NPX stdio server saves a full Markdown report locally by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
+
+## Updating Existing Installs
+
+Hosted API and website changes deploy immediately to `https://mcpscraper.dev`. Local stdio MCP changes require publishing a new npm package version and restarting the MCP client. Running MCP server processes do not hot-update, and tool names/descriptions are loaded when the local server process starts.
+
+Recommended config for update-friendly installs:
+
+```bash
+npx -y mcp-scraper@latest
+npx -y -p mcp-scraper@latest browser-agent
+```
+
+If a user configured `mcp-scraper@0.2.0`, installed globally with `npm install -g mcp-scraper`, or installed it as a project dependency, they will stay on that version until they update the config or reinstall:
+
+```bash
+npm update -g mcp-scraper
+npm install mcp-scraper@latest
+```
+
+Users who do not update can keep using the tools their local package already advertises, but they will not see newly added local stdio tools, schemas, or AI-facing descriptions. For example, a client running an older local package cannot call `directory_workflow` through stdio even if the hosted API already supports it. Users who configured only `mcp-scraper` must add `browser-agent` separately; MCP clients do not auto-create a second server entry from an existing config.
+
+## Branded One-Click Installs
+
+Raw `npx` MCP installs are command/config based. They do not provide a reliable user-facing install card, logo, or setup screen inside MCP clients. Do not print marketing text to stdout from an MCP server; stdout is reserved for JSON-RPC protocol messages.
+
+For a branded Claude Desktop install, package MCP Scraper as an MCPB Desktop Extension. An MCPB bundle can include a `manifest.json`, bundled server files/dependencies, `user_config` fields for API-key setup, and an optional `icon.png`. That is the right path for a designed install experience with a logo and guided configuration.
 
 ## Development