mcp-scraper 0.3.17 → 0.3.18

package/README.md

@@ -6,20 +6,18 @@ MCP Scraper is an MCP server for live web intelligence tools backed by `https://
 
 Use the MCPB Desktop Extension for the branded Claude Desktop install, or use the npm package from any MCP client that can run stdio commands.
 
-MCP Scraper ships three stdio entrypoints plus human-facing helper CLIs:
+MCP Scraper ships one stdio entrypoint plus human-facing helper CLIs:
 
-- `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook ads and organic video transcripts, Maps, directory, rank tracker blueprint, and credit tools.
-- `browser-agent` — agent-controlled direct/no-proxy hosted browser sessions with screenshots, clicks, typing, scrolling, watch URLs, replay links, MP4 replay download, and saved profile setup for authenticated AI visibility workflows.
-- `mcp-scraper-combined` — one context-aware command. In a human terminal it prints the branded ASCII install card; in an MCP client it runs the combined stdio server with both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
+- `mcp-scraper` — the single MCP server with every tool: live web intelligence (SERP, PAA, site extraction, YouTube, Facebook ads and organic video transcripts, Maps, directory, rank tracker blueprint, credits) plus agent-controlled direct/no-proxy hosted browser sessions (screenshots, clicks, typing, scrolling, watch URLs, replay links, MP4 replay download, and saved profile/login setup for authenticated AI visibility workflows). It is context-aware: in a human terminal it prints the branded ASCII install card; in an MCP client it runs as a protocol-clean stdio server. This is the entrypoint used by the MCPB Desktop Extension.
 - `mcp-scraper-install` — explicit alias for the human-facing terminal installer card with the branded ASCII intro and copyable install commands.
 - `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, SEO workflow runs, and HTML reports. This command is safe to print because it is not an MCP stdio server.
 
 ### Terminal installer
 
-Run the combined command when you want the designed terminal install experience:
+Run the command when you want the designed terminal install experience:
 
 ```bash
-npx -y -p mcp-scraper@latest mcp-scraper-combined
+npx -y -p mcp-scraper@latest mcp-scraper
 ```
 
 In a human terminal, it prints the MCP Scraper banner, loaded tool groups, Desktop Extension download, Claude Code command, and Codex config. When launched by an MCP client, the same command detects non-interactive stdio and writes only valid JSON-RPC to stdout.
@@ -41,7 +39,7 @@ npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
 ```
 
-`agent install claude --apply` upserts the Claude Code user-scope `mcp-scraper` entry to `npx -y -p mcp-scraper@latest mcp-scraper-combined`. Fully exit Claude Code and open a new Claude terminal after applying; MCP servers are attached when Claude starts.
+`agent install claude --apply` upserts the Claude Code user-scope `mcp-scraper` entry to `npx -y -p mcp-scraper@latest mcp-scraper`. Fully exit Claude Code and open a new Claude terminal after applying; MCP servers are attached when Claude starts.
 
 Check usage and upgrade concurrency from a normal terminal:
 
@@ -92,7 +90,7 @@ npm run build:mcpb
 
 The generated bundle is written to `build/mcpb/mcp-scraper-<version>.mcpb` and copied to `public/downloads/` for the hosted download. The current public bundle is `https://mcpscraper.dev/downloads/mcp-scraper.mcpb` (`0.2.24`, SHA-256 `2aff0e961e3de920d1fc67230056988a4a9120735f26f234f0ada6242350a4f0`). Install it by opening or dragging it into Claude Desktop. Claude displays the `MCP Scraper` install card, icon, and API-key configuration field from the bundle manifest.
 
-The MCPB install exposes the same web-intelligence tools as `mcp-scraper` plus all `browser_*` tools from `browser-agent` through one server.
+The MCPB install exposes every tool — web-intelligence plus all `browser_*` tools — through the one `mcp-scraper` server.
 
 ### Raw stdio config
 
@@ -103,7 +101,7 @@ Claude Desktop:
   "mcpServers": {
     "mcp-scraper": {
       "command": "npx",
-      "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"],
+      "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper"],
       "env": {
         "MCP_SCRAPER_API_KEY": "sk_live_your_key",
         "BROWSER_AGENT_PROFILE_NAME": "work-accounts"
@@ -113,15 +111,13 @@ Claude Desktop:
 }
 ```
 
-Existing MCP configs that use only `npx -y mcp-scraper` still work for the web intelligence server, but they do not automatically add browser tools. Switch to `mcp-scraper-combined` or add the second `browser-agent` config entry if you want browser tools. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
+`npx -y mcp-scraper@latest` runs the same single server (the package's default bin), so existing configs get every tool, including browser tools, after they pull the latest package. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
 
-For customer AI visibility setup, use the hosted profile connection flow. Open the setup browser at Google, give the user the returned `watch_url`, have them click Sign in and complete login, then call `browser_close` to save cookies and storage into the saved MCP Scraper profile. If the profile later disconnects or shows login controls, run the same setup flow again.
+For customer AI visibility setup, use the hosted profile connection flow: run `browser_profile_connect`, give the user the returned `watch_url`, have them complete login, then poll `browser_profile_list` until the connection is authenticated. One profile can hold many logins — call `browser_profile_connect` again with the same profile and a different domain to stack accounts.
 
-Browser-agent hosted sessions are direct/no-proxy by default. They do not inherit the configured SERP/Maps proxy. Use proxies for SERP/Maps tools when those tools need location or anti-bot handling, not for the interactive login browser.
+Browser hosted sessions are direct/no-proxy by default. They do not inherit the configured SERP/Maps proxy. Use proxies for SERP/Maps tools when those tools need location or anti-bot handling, not for the interactive login browser.
 
-`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions. For first-time setup, use `browser_profile_onboard`, open the returned `watch_url`, complete login, then call `browser_profile_status` until the connection is authenticated.
-
-Inside MCP clients, use `browser_profile_onboard` for customer setup and reconnects. The customer signs in through the hosted `watch_url` so they control the login session directly.
+`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions.
 
 Claude Code one-command setup:
 
@@ -138,14 +134,7 @@ claude mcp list
 Manual Claude Code command:
 
 ```bash
-claude mcp add mcp-scraper --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y -p mcp-scraper@latest mcp-scraper-combined
-```
-
-Split-server raw config still works:
-
-```bash
-claude mcp add mcp-scraper --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y mcp-scraper@latest
-claude mcp add browser-agent --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y -p mcp-scraper@latest browser-agent
+claude mcp add mcp-scraper --scope user --env MCP_SCRAPER_API_KEY=sk_live_your_key -- npx -y -p mcp-scraper@latest mcp-scraper
 ```
 
 Codex config:
@@ -153,21 +142,7 @@ Codex config:
 ```toml
 [mcp_servers.mcp-scraper]
 command = "npx"
-args = ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"]
-env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
-```
-
-Split-server Codex config:
-
-```toml
-[mcp_servers.mcp-scraper]
-command = "npx"
-args = ["-y", "mcp-scraper@latest"]
-env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
-
-[mcp_servers.browser-agent]
-command = "npx"
-args = ["-y", "-p", "mcp-scraper@latest", "browser-agent"]
+args = ["-y", "-p", "mcp-scraper@latest", "mcp-scraper"]
 env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 ```
 
@@ -202,8 +177,8 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 ### Browser-agent tools
 
 - `browser_open` — open a direct/no-proxy hosted browser session. Pass a saved profile name to load an authenticated session. Returns a human `watch_url`.
-- `browser_profile_onboard` — create or load a saved hosted browser profile auth connection and return a user takeover `watch_url`. The user signs in through that URL, then `browser_profile_status` verifies the profile is authenticated. Use this again whenever the profile disconnects.
-- `browser_profile_status` — check whether a hosted browser profile auth connection is authenticated before launching it with `browser_open`.
+- `browser_profile_connect` — save a site login to a profile and return a user takeover `watch_url` on mcpscraper.dev. The user signs in through that URL, then `browser_profile_list` verifies it is authenticated. One profile holds many logins — call again with the same profile and a different domain to stack accounts. Use this again whenever a login disconnects.
+- `browser_profile_list` — list every login saved in a profile with its auth status, account, and note. Poll it after `browser_profile_connect` until a login reads AUTHENTICATED, then launch with `browser_open`.
 - `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds.
 - `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image.
 - `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds.
@@ -228,15 +203,15 @@ For general SERP tools (`harvest_paa`, `search_serp`, and hosted SERP capture),
 
 For Google Maps tools (`maps_search` and `directory_workflow`), keep `proxyMode` at the default `location` for US city/state work. Retryable Maps failures such as CAPTCHA, timeout, proxy tunnel failure, proxy unavailability, or browser-session death rotate to a new residential proxy and a new browser session for up to 5 attempts. Successful structured responses include sanitized attempt telemetry so callers can verify the proxy source, observed city/region, and attempt count without exposing full proxy or browser IDs.
 
-The MCPB bundle and `mcp-scraper-combined` expose both sections through one MCP server. The split `mcp-scraper` entrypoint exposes only the web-intelligence tools, and the split `browser-agent` entrypoint exposes only the browser-agent tools.
+The `mcp-scraper` server (and the MCPB bundle, which runs it) exposes both sections through one MCP server.
 
 All MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split `mcp-scraper` stdio server exposes 24 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 20 browser tools. The combined stdio server exposes 44 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 24 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (26 total). Across all public surfaces there are 46 unique tool names.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The `mcp-scraper` stdio server exposes 45 tools (25 web-intelligence/workflow tools plus 20 browser tools). The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (27 total). Across all public surfaces there are 47 unique tool names.
 
 ## Resources
 
-The `mcp-scraper` and `mcp-scraper-combined` NPX stdio servers also expose saved reports as MCP resources: `resources/list` returns the most recent Markdown reports from your output directory as `report://` URIs, and `resources/read` returns their content — so an MCP client can pull prior research into context without re-scraping or spending credits. The hosted endpoint does not expose resources (it saves no files).
+The `mcp-scraper` NPX stdio server also exposes saved reports as MCP resources: `resources/list` returns the most recent Markdown reports from your output directory as `report://` URIs, and `resources/read` returns their content — so an MCP client can pull prior research into context without re-scraping or spending credits. The hosted endpoint does not expose resources (it saves no files).
 
 ## Environment
 
@@ -245,10 +220,10 @@ The `mcp-scraper` and `mcp-scraper-combined` NPX stdio servers also expose saved
 - `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`.
 - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
 - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
-- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `browser-agent` and `mcp-scraper-combined` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
+- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `mcp-scraper` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
 - `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
 
-Every web intelligence tool call made through `mcp-scraper` or `mcp-scraper-combined` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. AI fan-out exports are saved by `browser_capture_fanout` with `export=true` under `MCP_SCRAPER_OUTPUT_DIR/fanout`, and return relative paths.
+Every web intelligence tool call made through `mcp-scraper` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. AI fan-out exports are saved by `browser_capture_fanout` with `export=true` under `MCP_SCRAPER_OUTPUT_DIR/fanout`, and return relative paths.
 
 ## Updating Existing Installs
 
@@ -257,17 +232,10 @@ Hosted API and website changes deploy immediately to `https://mcpscraper.dev`. S
 Recommended config for update-friendly installs:
 
 ```bash
-npx -y -p mcp-scraper@latest mcp-scraper-combined
+npx -y -p mcp-scraper@latest mcp-scraper
 ```
 
-This is context-aware: in a normal terminal it prints the visible installer and ASCII card; in an MCP client it runs as the silent stdio server. Use `--stdio` or `MCP_SCRAPER_FORCE_STDIO=1` if you need to force server mode from a terminal.
-
-Split-server config:
-
-```bash
-npx -y mcp-scraper@latest
-npx -y -p mcp-scraper@latest browser-agent
-```
+This is context-aware: in a normal terminal it prints the visible installer and ASCII card; in an MCP client it runs as the silent stdio server. Use `--stdio` or `MCP_SCRAPER_FORCE_STDIO=1` if you need to force server mode from a terminal. `npx -y mcp-scraper@latest` (the package default bin) runs the same server.
 
 If a user configured `mcp-scraper@0.2.0`, installed globally with `npm install -g mcp-scraper`, or installed it as a project dependency, they will stay on that version until they update the config or reinstall:
 
@@ -276,19 +244,19 @@ npm update -g mcp-scraper
 npm install mcp-scraper@latest
 ```
 
-Users who do not update can keep using the tools their installed package already advertises, but they will not see newly added stdio tools, schemas, or AI-facing descriptions. For example, a client running an older package cannot call `rank_tracker_blueprint`, `directory_workflow`, or browser tools through stdio even if the hosted API already supports adjacent endpoints. Users who configured only `mcp-scraper` must switch to `mcp-scraper-combined` or add `browser-agent` separately; MCP clients do not auto-create a second server entry from an existing config.
+Users who do not update can keep using the tools their installed package already advertises, but they will not see newly added stdio tools, schemas, or AI-facing descriptions. For example, a client running an older package cannot call `rank_tracker_blueprint`, `directory_workflow`, or browser tools through stdio even if the hosted API already supports adjacent endpoints.
 
 ## Branded One-Click Installs
 
-Raw `npx` MCP server installs are command/config based. `mcp-scraper-combined` is context-aware: terminal TTY prints onboarding text and the ASCII card; MCP-client stdio pipes stay protocol-clean. Do not print marketing text to stdout from an active MCP stdio session; stdout is reserved for JSON-RPC protocol messages.
+Raw `npx` MCP server installs are command/config based. `mcp-scraper` is context-aware: terminal TTY prints onboarding text and the ASCII card; MCP-client stdio pipes stay protocol-clean. Do not print marketing text to stdout from an active MCP stdio session; stdout is reserved for JSON-RPC protocol messages.
 
-For a branded Claude Desktop install, package MCP Scraper as an MCPB Desktop Extension. The repository now builds one combined MCPB bundle with a generated icon, `manifest.json`, bundled runtime dependencies, and `user_config` fields for API-key setup, API URL, and output folder.
+For a branded Claude Desktop install, package MCP Scraper as an MCPB Desktop Extension. The repository builds one MCPB bundle with a generated icon, `manifest.json`, bundled runtime dependencies, and `user_config` fields for API-key setup, API URL, and output folder.
 
 ```bash
 npm run build:mcpb
 ```
 
-The bundle uses `mcp-scraper-combined` internally, so the user installs `MCP Scraper` once and gets web-intelligence tools plus live browser tools in one MCP server.
+The bundle runs `mcp-scraper` internally, so the user installs `MCP Scraper` once and gets web-intelligence tools plus live browser tools in one MCP server.
 
 ## Development