mcp-scraper 0.3.5 → 0.3.7

package/README.md

@@ -4,15 +4,15 @@ MCP Scraper is an MCP server for live web intelligence tools backed by `https://
 
 ## Install
 
-Use the MCPB Desktop Extension for the branded Claude Desktop install, or use the npm package from any MCP client that can run local stdio commands.
+Use the MCPB Desktop Extension for the branded Claude Desktop install, or use the npm package from any MCP client that can run stdio commands.
 
-MCP Scraper ships three local stdio entrypoints plus human-facing helper CLIs:
+MCP Scraper ships three stdio entrypoints plus human-facing helper CLIs:
 
 - `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook ads and organic video transcripts, Maps, directory, rank tracker blueprint, and credit tools.
 - `browser-agent` — agent-controlled direct/no-proxy hosted browser sessions with screenshots, clicks, typing, scrolling, watch URLs, replay links, MP4 replay download, and saved profile setup for authenticated AI visibility workflows.
 - `mcp-scraper-combined` — one context-aware command. In a human terminal it prints the branded ASCII install card; in an MCP client it runs the combined stdio server with both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
 - `mcp-scraper-install` — explicit alias for the human-facing terminal installer card with the branded ASCII intro and copyable install commands.
-- `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, local SEO workflow runs, and local HTML reports. This command is safe to print because it is not an MCP stdio server.
+- `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, SEO workflow runs, and HTML reports. This command is safe to print because it is not an MCP stdio server.
 
 ### Terminal installer
 
@@ -52,7 +52,7 @@ MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cl
 
 Each account has 1 base concurrent operation. Extra concurrency slots are `$5/month` per slot. If an MCP tool hits `concurrency_limit_exceeded`, the error includes `active`, `limit`, `upgrade_url`, and the `mcp-scraper-cli billing concurrency checkout` command so the AI can explain the next step without guessing.
 
-Run local workflow reports:
+Run workflow reports:
 
 ```bash
 npx -y -p mcp-scraper@latest mcp-scraper-cli workflow list
@@ -115,13 +115,13 @@ Claude Desktop:
 
 Existing MCP configs that use only `npx -y mcp-scraper` still work for the web intelligence server, but they do not automatically add browser tools. Switch to `mcp-scraper-combined` or add the second `browser-agent` config entry if you want browser tools. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
 
-For customer AI visibility setup, use the hosted profile connection flow. Open the setup browser at Google, give the user the returned `watch_url`, have them click Sign in and complete login, then call `browser_close` to save cookies and local storage into the saved MCP Scraper profile. If the profile later disconnects or shows login controls, run the same setup flow again.
+For customer AI visibility setup, use the hosted profile connection flow. Open the setup browser at Google, give the user the returned `watch_url`, have them click Sign in and complete login, then call `browser_close` to save cookies and storage into the saved MCP Scraper profile. If the profile later disconnects or shows login controls, run the same setup flow again.
 
 Browser-agent hosted sessions are direct/no-proxy by default. They do not inherit the configured SERP/Maps proxy. Use proxies for SERP/Maps tools when those tools need location or anti-bot handling, not for the interactive login browser.
 
-`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions. For first-time setup, use `browser_profile_onboard`, open the returned `watch_url`, complete login in the hosted browser, then call `browser_close` to persist cookies/local storage into the saved profile.
+`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions. For first-time setup, use `browser_profile_onboard`, open the returned `watch_url`, complete login in the hosted browser, then call `browser_close` to persist cookies and storage into the saved profile.
 
-Inside MCP clients, use `browser_profile_onboard` for customer setup and reconnects. The customer path does not import the user's local Chrome profile: it uses the hosted `watch_url` so the user controls the login session directly.
+Inside MCP clients, use `browser_profile_onboard` for customer setup and reconnects. The customer signs in through the hosted `watch_url` so they control the login session directly.
 
 Claude Code one-command setup:
 
@@ -186,8 +186,8 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 - `facebook_page_intel`
 - `facebook_ad_transcribe` — transcribe a direct Facebook ad video URL returned by `facebook_page_intel`.
 - `facebook_video_transcribe` — transcribe an organic Facebook reel, video, watch, post, or share URL, including `fb.watch` links. The tool renders the page, extracts the best matching public Facebook CDN MP4 URL, then returns transcript text, timestamped chunks, selected quality, video metadata, and the extracted MP4 URL for follow-up download.
-- `instagram_profile_content` — discover Instagram profile grid content links for a handle or profile URL. Returns collected post/reel/tv URLs, profile counts, type counts, shortcodes, browser mode, pagination attempts, stop reason, and limitations. Pass `browserProfile` from `browser_profile_import/sync` plus higher `maxItems`/`maxScrolls` for authenticated pagination.
-- `instagram_media_download` — extract and download one Instagram post/reel/tv URL. Returns text/caption, image URL/downloads, selected video/audio MP4 tracks, optional muxed MP4 when `ffmpeg` is available, optional transcript, and browser mode. Pass `browserProfile` for login-gated media.
+- `instagram_profile_content` — discover Instagram profile grid content links for a handle or profile URL. Returns collected post/reel/tv URLs, profile counts, type counts, shortcodes, browser details, pagination attempts, stop reason, and limitations.
+- `instagram_media_download` — extract and download one Instagram post/reel/tv URL. Returns text/caption, image URL/downloads, selected video/audio MP4 tracks, optional muxed MP4 when `ffmpeg` is available, optional transcript, and browser details.
 - `maps_search` — search Google Maps for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. In default `proxyMode: "location"`, retryable failures rotate to a new residential proxy and new browser session for up to 5 attempts. `maxResults` defaults to 10 and is capped at 50.
 - `maps_place_intel` — hydrate one known/named Google Maps business with profile details and optional reviews. Use after `maps_search` when a selected candidate needs full details.
 - `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus Google Maps searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." In default `proxyMode: "location"`, each city search rotates retryable failures to a new residential proxy and new browser session for up to 5 attempts. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `review_count`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups.
@@ -196,16 +196,13 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 - `workflow_run` — run hosted workflows such as `agent-packet`, `local-competitive-audit`, `map-comparison`, `serp-comparison`, `paa-expansion-brief`, and `ai-overview-language`; returns run metadata, summary, and artifact IDs.
 - `workflow_status` — reopen a workflow run and list its current status and artifacts.
 - `workflow_artifact_read` — pull generated workflow artifacts such as `evidence.json`, CSVs, Markdown briefs, and reports back into MCP context.
-- `rank_tracker_blueprint` — generate a database schema, cron/heartbeat plan, ingestion workflow, metrics list, and implementation prompt for building rank trackers. It has modes for Maps rankings via `directory_workflow`/`maps_search`, organic rankings via `search_serp`, AI Overview citation tracking, and PAA source presence tracking. This is a local planning tool and does not spend credits.
+- `rank_tracker_blueprint` — generate a database schema, cron/heartbeat plan, ingestion workflow, metrics list, and implementation prompt for building rank trackers. It has modes for Maps rankings via `directory_workflow`/`maps_search`, organic rankings via `search_serp`, AI Overview citation tracking, and PAA source presence tracking. This planning tool does not spend credits.
 - `credits_info`
 
 ### Browser-agent tools
 
-- `browser_open` — open a direct/no-proxy hosted browser session. Pass a saved profile name to load an authenticated session. Hosted mode returns a human `watch_url`.
-- `browser_profile_onboard` — create or load a saved hosted browser profile and open a direct/no-proxy setup browser at Google with profile saving enabled. The user clicks Sign in through the `watch_url`, then `browser_close` persists cookies/local storage into the saved profile. Use this again whenever the profile disconnects.
-- `browser_profile_list` — developer-only local Chrome profile metadata helper. Do not use for customer auth setup.
-- `browser_profile_import` — developer-only local profile clone helper. Do not use for customer auth setup.
-- `browser_profile_sync` — developer-only local profile refresh helper. Do not use for customer auth setup.
+- `browser_open` — open a direct/no-proxy hosted browser session. Pass a saved profile name to load an authenticated session. Returns a human `watch_url`.
+- `browser_profile_onboard` — create or load a saved hosted browser profile and open a direct/no-proxy setup browser at Google with profile saving enabled. The user clicks Sign in through the `watch_url`, then `browser_close` persists cookies and storage into the saved profile. Use this again whenever the profile disconnects.
 - `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds.
 - `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image.
 - `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds.
@@ -229,11 +226,11 @@ For general SERP tools (`harvest_paa`, `search_serp`, and hosted SERP capture),
 
 For Google Maps tools (`maps_search` and `directory_workflow`), keep `proxyMode` at the default `location` for US city/state work. Retryable Maps failures such as CAPTCHA, timeout, proxy tunnel failure, proxy unavailability, or browser-session death rotate to a new residential proxy and a new browser session for up to 5 attempts. Successful structured responses include sanitized attempt telemetry so callers can verify the proxy source, observed city/region, and attempt count without exposing full proxy or browser IDs.
 
-The MCPB bundle and `mcp-scraper-combined` expose both sections through one local MCP server. The split `mcp-scraper` entrypoint exposes only the web-intelligence tools, and the split `browser-agent` entrypoint exposes only the browser-agent tools.
+The MCPB bundle and `mcp-scraper-combined` expose both sections through one MCP server. The split `mcp-scraper` entrypoint exposes only the web-intelligence tools, and the split `browser-agent` entrypoint exposes only the browser-agent tools.
 
-All local and hosted MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their local file side effects.
+All MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split local `mcp-scraper` stdio server exposes 22 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 21 browser tools. The combined local stdio server exposes 43 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 22 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (24 total). Across all public local and hosted surfaces there are 45 unique tool names.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split `mcp-scraper` stdio server exposes 24 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 19 browser tools. The combined stdio server exposes 43 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 24 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (26 total). Across all public surfaces there are 45 unique tool names.
 
 ## Resources
 
@@ -246,19 +243,14 @@ The `mcp-scraper` and `mcp-scraper-combined` NPX stdio servers also expose saved
 - `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`.
 - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
 - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
-- `MCP_SCRAPER_BROWSER_MODE=local` is a developer-only override that makes `browser_open` use local Google Chrome instead of the hosted browser service. Do not set this for customer AI visibility setup.
-- `MCP_SCRAPER_BROWSER_PROFILE` is a developer-only local browser profile override. Hosted customer setup uses `BROWSER_AGENT_PROFILE_NAME` and `browser_profile_onboard`.
-- `MCP_SCRAPER_BROWSER_EXECUTABLE` optionally points developer-only local browser mode at a Chrome executable.
-- `MCP_SCRAPER_BROWSER_PROFILE_DIR` optionally points developer-only local browser mode at a direct Chrome-compatible user data directory.
-- Instagram tools accept `browserMode` (`auto`, `hosted`, `local`) and `browserProfile`; local mode is developer-only.
 - `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `browser-agent` and `mcp-scraper-combined` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
-- `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup mode. It persists cookies and local storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
+- `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
 
 Every web intelligence tool call made through `mcp-scraper` or `mcp-scraper-combined` saves a full Markdown report locally by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
 
 ## Updating Existing Installs
 
-Hosted API and website changes deploy immediately to `https://mcpscraper.dev`. Local stdio MCP changes require publishing a new npm package version and restarting the MCP client. Running MCP server processes do not hot-update, and tool names/descriptions are loaded when the local server process starts.
+Hosted API and website changes deploy immediately to `https://mcpscraper.dev`. Stdio MCP changes require publishing a new npm package version and restarting the MCP client. Running MCP server processes do not hot-update, and tool names/descriptions are loaded when the server process starts.
 
 Recommended config for update-friendly installs:
 
@@ -282,7 +274,7 @@ npm update -g mcp-scraper
 npm install mcp-scraper@latest
 ```
 
-Users who do not update can keep using the tools their local package already advertises, but they will not see newly added local stdio tools, schemas, or AI-facing descriptions. For example, a client running an older local package cannot call `rank_tracker_blueprint`, `directory_workflow`, or browser tools through stdio even if the hosted API already supports adjacent endpoints. Users who configured only `mcp-scraper` must switch to `mcp-scraper-combined` or add `browser-agent` separately; MCP clients do not auto-create a second server entry from an existing config.
+Users who do not update can keep using the tools their installed package already advertises, but they will not see newly added stdio tools, schemas, or AI-facing descriptions. For example, a client running an older package cannot call `rank_tracker_blueprint`, `directory_workflow`, or browser tools through stdio even if the hosted API already supports adjacent endpoints. Users who configured only `mcp-scraper` must switch to `mcp-scraper-combined` or add `browser-agent` separately; MCP clients do not auto-create a second server entry from an existing config.
 
 ## Branded One-Click Installs