mcp-scraper 0.2.19 → 0.2.21

package/README.md

@@ -36,7 +36,10 @@ Run setup checks and generate agent wiring:
 
 ```bash
 npx -y -p mcp-scraper@latest mcp-scraper-cli doctor
+npx -y -p mcp-scraper@latest mcp-scraper-cli browser profiles --email seo@example.com
+npx -y -p mcp-scraper@latest mcp-scraper-cli browser import-chrome --email seo@example.com --name seo-example-com
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
+npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude-desktop --browser-mode local --browser-profile seo-example-com
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
 ```
 
@@ -102,7 +105,10 @@ Claude Desktop:
       "command": "npx",
       "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"],
       "env": {
-        "MCP_SCRAPER_API_KEY": "sk_live_your_key"
+        "MCP_SCRAPER_API_KEY": "sk_live_your_key",
+        "MCP_SCRAPER_BROWSER_MODE": "local",
+        "MCP_SCRAPER_BROWSER_PROFILE": "work-accounts",
+        "MCP_SCRAPER_BROWSER_EXECUTABLE": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
       }
     }
   }
@@ -111,6 +117,21 @@ Claude Desktop:
 
 Existing MCP configs that use only `npx -y mcp-scraper` still work for the web intelligence server, but they do not automatically add browser tools. Switch to `mcp-scraper-combined` or add the second `browser-agent` config entry if you want browser tools. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
 
+For local Chrome state, first clone a Chrome profile into MCP Scraper managed storage:
+
+```bash
+mcp-scraper-cli browser import-chrome --email you@example.com --name work-accounts
+mcp-scraper-cli agent install codex --browser-mode local --browser-profile work-accounts
+```
+
+`browser import-chrome` copies the selected local Chrome profile into `~/.mcp-scraper/browser-profiles/<name>/user-data`, skipping cache and lock files. The managed clone can include cookies, local storage, history, session storage, and Chrome password database files. It does not upload the profile. Re-run `mcp-scraper-cli browser sync-profile <name>` after logging into new sites in normal Chrome.
+
+Set `MCP_SCRAPER_BROWSER_MODE=local` and `MCP_SCRAPER_BROWSER_PROFILE=<name>` when you want `browser_open` to launch local Google Chrome against that managed clone. A `profile` argument passed directly to `browser_open` overrides the default for that one session.
+
+Hosted browser profiles are still available. `BROWSER_AGENT_PROFILE_NAME` sets the default saved Kernel browser profile for hosted sessions. For first-time hosted setup, set `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true`, open a browser with the profile, complete login through the watch URL, then call `browser_close` to persist cookies/local storage into the hosted Kernel profile.
+
+Inside MCP clients, use `browser_profile_list` to inspect local Chrome account/profile metadata, `browser_profile_import` to clone a profile for local browser mode, `browser_profile_sync` to refresh an existing clone, and `browser_profile_onboard` only for hosted Kernel setup.
+
 Claude Code:
 
 ```bash
@@ -175,7 +196,11 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 ### Browser-agent tools
 
-- `browser_open` — open a live cloud browser session. Returns a `session_id`, a human `watch_url`, and the raw `live_view_url` when available.
+- `browser_open` — open a browser session. Hosted mode returns a human `watch_url`; local mode opens Google Chrome on this machine against an imported managed profile.
+- `browser_profile_list` — list local Chrome account/profile metadata and suggested managed profile names. This does not read cookies, passwords, browsing history, or copy local Chrome state.
+- `browser_profile_import` — clone a local Chrome profile into `~/.mcp-scraper/browser-profiles` for local browser mode. This copies browser state files but skips cache and locks.
+- `browser_profile_sync` — refresh an existing managed local profile clone from its recorded source Chrome profile.
+- `browser_profile_onboard` — create or load a hosted Kernel browser profile and open a setup browser with profile saving enabled. The user logs in through the `watch_url`, then `browser_close` persists cookies/local storage into the hosted Kernel profile.
 - `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds.
 - `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image.
 - `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds.
@@ -195,15 +220,15 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 For accurate annotated videos, do not guess annotation times from a script. Start the replay, navigate until each target is visible and stable, call `browser_replay_mark` for each callout, then stop the replay and pass the returned annotations to `browser_replay_annotate` with the returned `source_width` and `source_height`.
 
-For US local SERP tools (`harvest_paa` and `search_serp`), keep `proxyMode` at the default `location` unless you are debugging. Location mode uses fresh residential proxy IDs across retries. When Google shows a CAPTCHA/challenge, browser-service sessions briefly wait for automatic challenge solving first, then rotate to a new proxy/session if the challenge does not clear. Proxy tunnel failure and wrong-location evidence are also retryable before returning.
+For general SERP tools (`harvest_paa`, `search_serp`, and hosted SERP capture), omit `proxyMode` for normal use. The default is `configured`, which uses the configured browser-service proxy without city/ZIP targeting for the highest general success rate. Use `proxyMode: "location"` only when the user explicitly needs city/ZIP-targeted residential proxy evidence. When Google shows a CAPTCHA/challenge, browser-service sessions briefly wait for automatic challenge solving first, then rotate to a new proxy/session if the challenge does not clear. Proxy tunnel failure and wrong-location evidence are also retryable before returning.
 
 For Google Maps tools (`maps_search` and `directory_workflow`), keep `proxyMode` at the default `location` for US city/state work. Retryable Maps failures such as CAPTCHA, timeout, proxy tunnel failure, proxy unavailability, or browser-session death rotate to a new residential proxy and a new browser session for up to 5 attempts. Successful structured responses include sanitized attempt telemetry so callers can verify the proxy source, observed city/region, and attempt count without exposing full proxy or browser IDs.
 
 The MCPB bundle and `mcp-scraper-combined` expose both sections through one local MCP server. The split `mcp-scraper` entrypoint exposes only the web-intelligence tools, and the split `browser-agent` entrypoint exposes only the browser-agent tools.
 
-Chaining and planning tools (`maps_search`, `directory_workflow`, `workflow_list`, `workflow_suggest`, `workflow_run`, `workflow_status`, `workflow_artifact_read`, `rank_tracker_blueprint`, `map_site_urls`, `youtube_harvest`, `facebook_ad_search`, `facebook_page_intel`, `facebook_video_transcribe`) return `structuredContent` with the IDs, URLs, CSV paths, transcripts, artifacts, recipe fields, or blueprint fields needed by the next step. All tools carry MCP annotations (`readOnlyHint: true`; live-web tools use `openWorldHint: true`, while local planning and credit tools are closed-world/idempotent).
+All local and hosted MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their local file side effects.
 
-The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 21 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (23 total). Browser-agent tools are local stdio tools backed by the REST API under `https://mcpscraper.dev/agent/*`.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split local `mcp-scraper` stdio server exposes 22 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 21 browser tools. The combined local stdio server exposes 43 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 22 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (24 total). Across all public local and hosted surfaces there are 45 unique tool names.
 
 ## Resources
 
@@ -216,6 +241,12 @@ The `mcp-scraper` and `mcp-scraper-combined` NPX stdio servers also expose saved
 - `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`.
 - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
 - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
+- `MCP_SCRAPER_BROWSER_MODE=local` makes `browser_open` use local Google Chrome instead of the hosted browser service.
+- `MCP_SCRAPER_BROWSER_PROFILE` sets the default managed local browser profile name created by `mcp-scraper-cli browser import-chrome`.
+- `MCP_SCRAPER_BROWSER_EXECUTABLE` optionally points local browser mode at a Chrome executable. On macOS, Google Chrome is recommended so Keychain-backed Chrome state works.
+- `MCP_SCRAPER_BROWSER_PROFILE_DIR` optionally points local browser mode at a direct Chrome-compatible user data directory instead of a managed profile manifest.
+- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved Kernel browser profile for hosted-mode `browser-agent` and `mcp-scraper-combined` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
+- `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup mode. It persists cookies and local storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
 
 Every web intelligence tool call made through `mcp-scraper` or `mcp-scraper-combined` saves a full Markdown report locally by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.