mcp-scraper 0.2.7 → 0.2.9

package/README.md

@@ -6,12 +6,13 @@ MCP Scraper is an MCP server for live web intelligence tools backed by `https://
 
 Use the MCPB Desktop Extension for the branded Claude Desktop install, or use the npm package from any MCP client that can run local stdio commands.
 
-MCP Scraper ships three local stdio entrypoints:
+MCP Scraper ships three local stdio entrypoints plus human-facing helper CLIs:
 
 - `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook, Maps, directory, rank tracker blueprint, and credit tools.
 - `browser-agent` — an agent-controlled live cloud browser with screenshots, clicks, typing, scrolling, live watch URLs, replay links, and MP4 replay download.
 - `mcp-scraper-combined` — one local MCP server that exposes both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
 - `mcp-scraper-install` — a human-facing terminal installer card with the branded ASCII intro and copyable install commands. This command is safe to print because it is not an MCP stdio server.
+- `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, local SEO workflow runs, and local HTML reports. This command is safe to print because it is not an MCP stdio server.
 
 ### Terminal installer
 
@@ -23,6 +24,42 @@ npx -y -p mcp-scraper@latest mcp-scraper-install
 
 It prints the MCP Scraper banner, loaded tool groups, Desktop Extension download, Claude Code command, and Codex config. The MCP server commands stay silent so stdout remains valid JSON-RPC for MCP clients.
 
+### Human CLI
+
+Run setup checks and generate agent wiring:
+
+```bash
+npx -y -p mcp-scraper@latest mcp-scraper-cli doctor
+npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
+npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
+```
+
+Run local workflow reports:
+
+```bash
+npx -y -p mcp-scraper@latest mcp-scraper-cli workflow list
+npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com
+npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run local-competitive-audit --query roofers --state TN --min-pop 100000 --per-city 20 --hydrate-top 5 --reviews 50
+npx -y -p mcp-scraper@latest mcp-scraper-cli report open last
+```
+
+Workflow runs save `manifest.json`, `report.html`, CSVs, Markdown, and evidence JSON under `MCP_SCRAPER_OUTPUT_DIR/workflows` or `~/Downloads/mcp-scraper/workflows`.
+
+Run hosted workflow schedules and hosted run history:
+
+```bash
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule create local-competitive-audit --weekly --query roofers --state TN --min-pop 100000 --per-city 20 --webhook https://example.com/mcp-scraper-hook
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule list
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule run <schedule-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule pause <schedule-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule resume <schedule-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli runs list
+npx -y -p mcp-scraper@latest mcp-scraper-cli runs status <run-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli runs download <run-id>
+```
+
+The hosted workflow API is mounted under `/workflows`: `GET /definitions`, `POST /run`, `GET /runs`, `GET /runs/:id`, `GET /runs/:id/artifacts/:artifactId`, `POST /schedules`, `GET /schedules`, `PATCH /schedules/:id`, `DELETE /schedules/:id`, and `POST /schedules/:id/run`. Scheduled dispatch is handled by `/workflows/cron/dispatch` and the main `/cron/tick` route when `CRON_SECRET` is configured. Current hosted artifacts are filesystem-backed for immediate run retrieval; durable public rollout should add S3/R2-compatible object storage.
+
 ### Claude Desktop MCPB
 
 Build the branded one-click bundle:
@@ -31,7 +68,7 @@ Build the branded one-click bundle:
 npm run build:mcpb
 ```
 
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.6`, SHA-256 `d316c1e3744d61c84b8fbbf2b155dba7da4062be52d9024d9fa7adde04c5b73c`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
+The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.9`, SHA-256 `d259e62283f3e4cbd2e5096cd4b2eb9c7a50b5847bc9e90f6ae43d203c5db7f8`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
 
 The MCPB install exposes the same web-intelligence tools as `mcp-scraper` plus all `browser_*` tools from `browser-agent` through one server.
 
@@ -105,9 +142,9 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 - `facebook_ad_search`
 - `facebook_page_intel`
 - `facebook_ad_transcribe`
-- `maps_search` — search Google Maps for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. `maxResults` defaults to 10 and is capped at 50.
+- `maps_search` — search Google Maps for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. In default `proxyMode: "location"`, retryable failures rotate to a new residential proxy and new browser session for up to 5 attempts. `maxResults` defaults to 10 and is capped at 50.
 - `maps_place_intel` — hydrate one known/named Google Maps business with profile details and optional reviews. Use after `maps_search` when a selected candidate needs full details.
-- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus Google Maps searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups. It captures Maps star ratings from list cards, not profile review counts.
+- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus Google Maps searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." In default `proxyMode: "location"`, each city search rotates retryable failures to a new residential proxy and new browser session for up to 5 attempts. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `review_count`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups.
 - `rank_tracker_blueprint` — generate a database schema, cron/heartbeat plan, ingestion workflow, metrics list, and implementation prompt for building rank trackers. It has modes for Maps rankings via `directory_workflow`/`maps_search`, organic rankings via `search_serp`, AI Overview citation tracking, and PAA source presence tracking. This is a local planning tool and does not spend credits.
 - `credits_info`
 
@@ -133,7 +170,9 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 For accurate annotated videos, do not guess annotation times from a script. Start the replay, navigate until each target is visible and stable, call `browser_replay_mark` for each callout, then stop the replay and pass the returned annotations to `browser_replay_annotate` with the returned `source_width` and `source_height`.
 
-For US local SERP tools (`harvest_paa` and `search_serp`), keep `proxyMode` at the default `location` unless you are debugging. Location mode uses fresh residential proxy IDs across retries and treats CAPTCHA, proxy tunnel failure, and wrong-location evidence as retryable before returning.
+For US local SERP tools (`harvest_paa` and `search_serp`), keep `proxyMode` at the default `location` unless you are debugging. Location mode uses fresh residential proxy IDs across retries. When Google shows a CAPTCHA/challenge, browser-service sessions briefly wait for automatic challenge solving first, then rotate to a new proxy/session if the challenge does not clear. Proxy tunnel failure and wrong-location evidence are also retryable before returning.
+
+For Google Maps tools (`maps_search` and `directory_workflow`), keep `proxyMode` at the default `location` for US city/state work. Retryable Maps failures such as CAPTCHA, timeout, proxy tunnel failure, proxy unavailability, or browser-session death rotate to a new residential proxy and a new browser session for up to 5 attempts. Successful structured responses include sanitized attempt telemetry so callers can verify the proxy source, observed city/region, and attempt count without exposing full proxy or browser IDs.
 
 The MCPB bundle and `mcp-scraper-combined` expose both sections through one local MCP server. The split `mcp-scraper` entrypoint exposes only the web-intelligence tools, and the split `browser-agent` entrypoint exposes only the browser-agent tools.