mcp-scraper 0.2.8 → 0.2.10

package/README.md

@@ -6,12 +6,13 @@ MCP Scraper is an MCP server for live web intelligence tools backed by `https://
 
 Use the MCPB Desktop Extension for the branded Claude Desktop install, or use the npm package from any MCP client that can run local stdio commands.
 
-MCP Scraper ships three local stdio entrypoints:
+MCP Scraper ships three local stdio entrypoints plus human-facing helper CLIs:
 
 - `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook, Maps, directory, rank tracker blueprint, and credit tools.
 - `browser-agent` — an agent-controlled live cloud browser with screenshots, clicks, typing, scrolling, live watch URLs, replay links, and MP4 replay download.
 - `mcp-scraper-combined` — one local MCP server that exposes both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
 - `mcp-scraper-install` — a human-facing terminal installer card with the branded ASCII intro and copyable install commands. This command is safe to print because it is not an MCP stdio server.
+- `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, local SEO workflow runs, and local HTML reports. This command is safe to print because it is not an MCP stdio server.
 
 ### Terminal installer
 
@@ -23,6 +24,42 @@ npx -y -p mcp-scraper@latest mcp-scraper-install
 
 It prints the MCP Scraper banner, loaded tool groups, Desktop Extension download, Claude Code command, and Codex config. The MCP server commands stay silent so stdout remains valid JSON-RPC for MCP clients.
 
+### Human CLI
+
+Run setup checks and generate agent wiring:
+
+```bash
+npx -y -p mcp-scraper@latest mcp-scraper-cli doctor
+npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
+npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
+```
+
+Run local workflow reports:
+
+```bash
+npx -y -p mcp-scraper@latest mcp-scraper-cli workflow list
+npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run agent-packet --keyword "roof repair Denver" --domain example.com
+npx -y -p mcp-scraper@latest mcp-scraper-cli workflow run local-competitive-audit --query roofers --state TN --min-pop 100000 --per-city 20 --hydrate-top 5 --reviews 50
+npx -y -p mcp-scraper@latest mcp-scraper-cli report open last
+```
+
+Workflow runs save `manifest.json`, `report.html`, CSVs, Markdown, and evidence JSON under `MCP_SCRAPER_OUTPUT_DIR/workflows` or `~/Downloads/mcp-scraper/workflows`.
+
+Run hosted workflow schedules and hosted run history:
+
+```bash
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule create local-competitive-audit --weekly --query roofers --state TN --min-pop 100000 --per-city 20 --webhook https://example.com/mcp-scraper-hook
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule list
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule run <schedule-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule pause <schedule-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli schedule resume <schedule-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli runs list
+npx -y -p mcp-scraper@latest mcp-scraper-cli runs status <run-id>
+npx -y -p mcp-scraper@latest mcp-scraper-cli runs download <run-id>
+```
+
+The hosted workflow API is mounted under `/workflows`: `GET /definitions`, `POST /run`, `GET /runs`, `GET /runs/:id`, `GET /runs/:id/artifacts/:artifactId`, `POST /schedules`, `GET /schedules`, `PATCH /schedules/:id`, `DELETE /schedules/:id`, and `POST /schedules/:id/run`. Scheduled dispatch is handled by `/workflows/cron/dispatch` and the main `/cron/tick` route when `CRON_SECRET` is configured. Current hosted artifacts are filesystem-backed for immediate run retrieval; durable public rollout should add S3/R2-compatible object storage.
+
 ### Claude Desktop MCPB
 
 Build the branded one-click bundle:
@@ -31,7 +68,7 @@ Build the branded one-click bundle:
 npm run build:mcpb
 ```
 
-The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.8`, SHA-256 `7d9d1bc0cf646a591fa5ed7e07b9bba5fa1ed598667f27cec4bd04e6197d5367`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
+The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.10`, SHA-256 `b6e83f65d461cebfe877b90dd37ae845a50765f7eb6351d1f1d6fe953b68535f`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
 
 The MCPB install exposes the same web-intelligence tools as `mcp-scraper` plus all `browser_*` tools from `browser-agent` through one server.
 
@@ -107,7 +144,7 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 - `facebook_ad_transcribe`
 - `maps_search` — search Google Maps for multiple business/profile candidates. Use for GMB/GBP prospect lists, competitors, categories, and anything needing more than the Google 3-pack. In default `proxyMode: "location"`, retryable failures rotate to a new residential proxy and new browser session for up to 5 attempts. `maxResults` defaults to 10 and is capped at 50.
 - `maps_place_intel` — hydrate one known/named Google Maps business with profile details and optional reviews. Use after `maps_search` when a selected candidate needs full details.
-- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus Google Maps searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." In default `proxyMode: "location"`, each city search rotates retryable failures to a new residential proxy and new browser session for up to 5 attempts. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups. It captures Maps star ratings from list cards, not profile review counts.
+- `directory_workflow` — build city-by-city directory/prospecting datasets from Census place selection plus Google Maps searches. Use it for requests like "all cities over 100k population in Tennessee, then get 20 roofers from Maps." In default `proxyMode: "location"`, each city search rotates retryable failures to a new residential proxy and new browser session for up to 5 attempts. The saved CSV includes `source_location`, `result_position`, `business_name`, `review_stars`, `review_count`, `category`, `address`, `phone`, `hours_status`, `website_url`, `directions_url`, `place_url`, `cid`, `cid_decimal`, Census population, and ZIP groups.
 - `rank_tracker_blueprint` — generate a database schema, cron/heartbeat plan, ingestion workflow, metrics list, and implementation prompt for building rank trackers. It has modes for Maps rankings via `directory_workflow`/`maps_search`, organic rankings via `search_serp`, AI Overview citation tracking, and PAA source presence tracking. This is a local planning tool and does not spend credits.
 - `credits_info`