mcp-scraper 0.34.1 → 0.36.0

package/README.md

@@ -39,7 +39,9 @@ npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
 ```
 
-`agent install claude --apply` upserts the Claude Code user-scope `mcp-scraper` entry to `npx -y -p mcp-scraper@latest mcp-scraper`. Fully exit Claude Code and open a new Claude terminal after applying; MCP servers are attached when Claude starts.
+`agent install claude --apply` upserts the Claude Code user-scope `mcp-scraper` entry to `npx -y --package mcp-scraper@latest mcp-scraper`. Fully exit Claude Code and open a new Claude terminal after applying; MCP servers are attached when Claude starts.
+
+The registered command uses the long `--package` flag deliberately. Claude Code's `mcp add` leaks short flags that appear after `--` back into its own option parsing, so a registered `-p` makes it reject its own `--scope`/`-s` argument with a misleading `unknown option` error. If the registration ever fails, the previous entry is captured beforehand and restored automatically.
 
 Check usage and upgrade concurrency from a normal terminal:
 
@@ -152,9 +154,10 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 - `harvest_paa`
 - `search_serp`
-- `extract_url`
+- `extract_url` — extract normal or Wayback-replayed page copy; Wayback results omit playback chrome and can include a timestamp-matched featured image.
 - `map_site_urls`
-- `extract_site`
+- `map_wayback_snapshots` — count and inventory Wayback captures across an inclusive date range without downloading page bodies. Supports exact pages, prefixes, hosts, domains, or selected URLs; reports exact versus lower-bound counts, unique URLs/content digests, monthly coverage, missing months, and optional timestamp rows.
+- `extract_site` — crawl a live site, batch one archived site snapshot from a Wayback replay URL, or pass a `wayback` plan for whole-site, single-page, or selected-page timelines across explicit months or a `from`/`to` range. Timeline ZIPs include month folders and a capture matrix.
 - `youtube_harvest`
 - `youtube_transcribe`
 - `facebook_ad_search`
@@ -165,7 +168,7 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 - `instagram_media_download` — extract and download one Instagram post/reel/tv URL, optionally through a saved hosted browser `profile` for authenticated access. Returns text/caption, image URL/downloads, selected video/audio MP4 tracks, optional muxed MP4 when `ffmpeg` is available, optional transcript, and browser details.
 - `maps_search` — search Google's localized local-results list for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. It opens the rendered business card, reads the profile dialog, then closes it before continuing to the next ranked card. Set `includeServices: true` to return services and areas served without collecting review cards. `maxResults` defaults to 10 and is capped at 50.
 - `maps_place_intel` — hydrate one known/named Google Maps business with profile details and optional reviews. Use after `maps_search` when a selected candidate needs full details.
-- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus localized Google business searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." The default direct route uses city-in-query plus UULE, `gl`, and `hl`; residential location proxying remains available only as an explicit override. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `review_count`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups.
+- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus localized Google business searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." Supply the business category, state, and market limits; MCP Scraper manages search transport and retry behavior internally. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `review_count`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups.
 - `workflow_list` — list higher-level workflow IDs plus AI-facing recipes for market analysis, ICP research, forum/review acquisition, brand design briefings, CRO audits, positioning briefs, content gaps, and AI search visibility audits.
 - `workflow_suggest` — route a high-level business goal to the right workflow/tool chain before spending credits.
 - `workflow_run` — run hosted workflows such as `agent-packet`, `local-competitive-audit`, `map-comparison`, `serp-comparison`, `paa-expansion-brief`, and `ai-overview-language`; returns run metadata, summary, and artifact IDs.
@@ -178,7 +181,7 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 - `list_service_connections` — list this caller's tenant-owned Nango OAuth and official remote MCP connections, including verified provider-side account email/name when exposed, exact live reads, gated actions, permanently blocked administrative tools, credential transport, and schema-discovery metadata. Provider identity is distinct from the MCP Scraper login, and connections are never shared between customers.
 - `describe_service_connection_tool` — fetch the sanitized live MCP Tool definition for one tool listed on one tenant-owned connection, including its current callability, input schema, optional output schema, safe annotations, and schema hash. Use this before constructing provider-native arguments; provider functions stay behind the generic bridges instead of becoming dozens of permanent top-level tools.
-- `export_connected_service_data` — fetch a fresh bounded Gmail, Google Calendar, Google Search Console, Zoom, Resend, or Meta time range in one MCP call. Search Console's `search_console_performance` dataset walks accessible properties and bounded live Search Analytics pages with signed continuation. Small exports return inline; larger exports become private JSONL retained for seven days with a 15-minute signed URL. For relationship work, gather source evidence first: inspect existing People records, resolve the exact provider account, use RFC3339 `from`/`to` for Gmail ranges longer than 90 days, preserve provider provenance when writing a linked Communication, and never treat an export as permission to mutate the source account.
+- `export_connected_service_data` — fetch a fresh bounded Gmail, Google Calendar, Google Search Console, Zoom, Resend, or Meta time range in one MCP call. Zoom's `zoom_transcripts` dataset resolves VTT files from recording metadata and downloads them through the authenticated connection without looping the separately rate-limited `get-meeting-transcript` function. Search Console's `search_console_performance` dataset walks accessible properties and bounded live Search Analytics pages with signed continuation. Small exports return inline; larger exports become private JSONL retained for seven days with a 15-minute signed URL. For relationship work, gather source evidence first: inspect existing People records, resolve the exact provider account, use RFC3339 `from`/`to` for Gmail ranges longer than 90 days, preserve provider provenance when writing a linked Communication, and never treat an export as permission to mutate the source account.
 - `export_search_console_table_data` — filter up to 50,000 Search Console rows already persisted by a scheduled `connection_sync` and create a private renewable JSONL artifact without calling Google again. Get the typed `gsc_performance_*` table name from `list_service_connections`, inspect it with `table-describe`, and use the same filters with `table-query` for interactive analysis.
 - `renew_connected_data_download` — issue a fresh 15-minute signed URL for an unexpired private export artifact without pulling the provider again.
 - `read_service_connection` — run one small live read by exact allowlisted name across Nango OAuth or official remote MCP connections, including bounded Google Drive inventory, change, Doc, Sheet, and text-file tools. Do not loop it over a time range when `export_connected_service_data` supports that provider's collection.
@@ -219,15 +222,13 @@ Google Search Console exposes eight bounded reads and eight gated property and s
 
 For accurate annotated videos, do not guess annotation times from a script. Start the replay, navigate until each target is visible and stable, call `browser_replay_mark` for each callout, then stop the replay and pass the returned annotations to `browser_replay_annotate` with the returned `source_width` and `source_height`.
 
-For general SERP tools (`harvest_paa`, `search_serp`, and hosted SERP capture), omit `proxyMode` for normal use. The default is `configured`, which uses the configured browser-service proxy without city/ZIP targeting for the highest general success rate. Use `proxyMode: "location"` only when the user explicitly needs city/ZIP-targeted residential proxy evidence. When Google shows a CAPTCHA/challenge, browser-service sessions briefly wait for automatic challenge solving first, then rotate to a new proxy/session if the challenge does not clear. Proxy tunnel failure and wrong-location evidence are also retryable before returning.
-
-For Google Maps tools (`maps_search` and `directory_workflow`), leave `proxyMode` unset for the default direct Google route. Localization is carried by the city in the query, UULE, `gl`, and `hl`; do not treat proxy geolocation as the source of truth. Use `proxyMode: "location"` only when an explicit residential-proxy experiment or compatibility path needs it. Retryable failures open a new browser session; location mode can additionally rotate its disposable proxy. Successful structured responses include sanitized attempt telemetry without exposing full proxy or browser IDs.
+For Google SERP and Maps tools, callers provide the query, two-letter country code (`gl`), language (`hl`), device when relevant, and an optional city or region. MCP Scraper owns transport selection, anti-bot handling, and bounded retries internally; those implementation controls and receipts are not part of the public tool contract.
 
 The `mcp-scraper` server (and the MCPB bundle, which runs it) exposes both sections through one MCP server.
 
 All MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. Both the `mcp-scraper` stdio server and the hosted endpoint at `https://mcpscraper.dev/mcp` expose the same 167 tools: 78 scraper, browser, workflow, billing, and connected-service tools plus 89 durable-memory tools. Release verification compares the exact local and remote tool-name sets, not only the count.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The current local unified server exposes 170 tools: 81 scraper, browser, workflow, billing, and connected-service tools plus 89 durable-memory tools. Release verification compares the exact local and hosted tool-name sets, not only the count.
 
 For contract parity, stdio and MCPB memory calls invoke the matching public tool on the hosted MCP Scraper `/mcp` endpoint. The hosted aggregate runtime owns MCP Scraper-specific billing, scheduling, credential, and in-process cutover policy; its internal `/memory/mcp-call` bridge is a fallback to the standalone memory service, not the public stdio execution path. Direct `mcp-memory` OAuth and stdio clients continue to use `memory.mcpscraper.dev` and must be verified as a separate dependent release surface.