mcp-scraper 0.3.3 → 0.3.5

package/README.md

@@ -9,7 +9,7 @@ Use the MCPB Desktop Extension for the branded Claude Desktop install, or use th
 MCP Scraper ships three local stdio entrypoints plus human-facing helper CLIs:
 
 - `mcp-scraper` — live web intelligence, SERP, PAA, site extraction, YouTube, Facebook ads and organic video transcripts, Maps, directory, rank tracker blueprint, and credit tools.
-- `browser-agent` — agent-controlled browser tools for hosted cloud sessions or local Chrome profile mode, with screenshots, clicks, typing, scrolling, watch URLs, replay links, MP4 replay download, and local profile import/sync helpers.
+- `browser-agent` — agent-controlled direct/no-proxy hosted browser sessions with screenshots, clicks, typing, scrolling, watch URLs, replay links, MP4 replay download, and saved profile setup for authenticated AI visibility workflows.
 - `mcp-scraper-combined` — one context-aware command. In a human terminal it prints the branded ASCII install card; in an MCP client it runs the combined stdio server with both tool sets. This is the entrypoint used by the MCPB Desktop Extension.
 - `mcp-scraper-install` — explicit alias for the human-facing terminal installer card with the branded ASCII intro and copyable install commands.
 - `mcp-scraper-cli` — a human-facing CLI for setup checks, AI-agent config generation, workflow prompts, local SEO workflow runs, and local HTML reports. This command is safe to print because it is not an MCP stdio server.
@@ -36,11 +36,8 @@ Run setup checks and generate agent wiring:
 
 ```bash
 npx -y -p mcp-scraper@latest mcp-scraper-cli doctor
-npx -y -p mcp-scraper@latest mcp-scraper-cli browser profiles --email seo@example.com
-npx -y -p mcp-scraper@latest mcp-scraper-cli browser import-chrome --email seo@example.com --name seo-example-com
 MCP_SCRAPER_API_KEY=sk_live_your_key npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude --apply
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
-npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude-desktop --browser-mode local --browser-profile seo-example-com
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
 ```
 
@@ -109,9 +106,7 @@ Claude Desktop:
       "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"],
       "env": {
         "MCP_SCRAPER_API_KEY": "sk_live_your_key",
-        "MCP_SCRAPER_BROWSER_MODE": "local",
-        "MCP_SCRAPER_BROWSER_PROFILE": "work-accounts",
-        "MCP_SCRAPER_BROWSER_EXECUTABLE": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
+        "BROWSER_AGENT_PROFILE_NAME": "work-accounts"
       }
     }
   }
@@ -120,20 +115,13 @@ Claude Desktop:
 
 Existing MCP configs that use only `npx -y mcp-scraper` still work for the web intelligence server, but they do not automatically add browser tools. Switch to `mcp-scraper-combined` or add the second `browser-agent` config entry if you want browser tools. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
 
-For local Chrome state, first clone a Chrome profile into MCP Scraper managed storage:
+For customer AI visibility setup, use the hosted profile connection flow. Open the setup browser at Google, give the user the returned `watch_url`, have them click Sign in and complete login, then call `browser_close` to save cookies and local storage into the saved MCP Scraper profile. If the profile later disconnects or shows login controls, run the same setup flow again.
 
-```bash
-mcp-scraper-cli browser import-chrome --email you@example.com --name work-accounts
-MCP_SCRAPER_API_KEY=sk_live_your_key mcp-scraper-cli agent install claude --apply --browser-mode local --browser-profile work-accounts
-```
-
-`browser import-chrome` copies the selected local Chrome profile into `~/.mcp-scraper/browser-profiles/<name>/user-data`, skipping cache and lock files. The managed clone can include cookies, local storage, history, session storage, and Chrome password database files. It does not upload the profile. Re-run `mcp-scraper-cli browser sync-profile <name>` after logging into new sites in normal Chrome.
-
-Set `MCP_SCRAPER_BROWSER_MODE=local` and `MCP_SCRAPER_BROWSER_PROFILE=<name>` when you want `browser_open` to launch local Google Chrome against that managed clone. A `profile` argument passed directly to `browser_open` overrides the default for that one session. Instagram tools can use the same managed profile by passing `browserProfile=<name>` or by running the API with local browser mode configured.
+Browser-agent hosted sessions are direct/no-proxy by default. They do not inherit the configured SERP/Maps proxy. Use proxies for SERP/Maps tools when those tools need location or anti-bot handling, not for the interactive login browser.
 
-Hosted browser profiles are still available. `BROWSER_AGENT_PROFILE_NAME` sets the default saved Kernel browser profile for hosted sessions. For first-time hosted setup, set `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true`, open a browser with the profile, complete login through the watch URL, then call `browser_close` to persist cookies/local storage into the hosted Kernel profile.
+`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions. For first-time setup, use `browser_profile_onboard`, open the returned `watch_url`, complete login in the hosted browser, then call `browser_close` to persist cookies/local storage into the saved profile.
 
-Inside MCP clients, use `browser_profile_list` to inspect local Chrome account/profile metadata, `browser_profile_import` to clone a profile for local browser mode, `browser_profile_sync` to refresh an existing clone, and `browser_profile_onboard` only for hosted Kernel setup.
+Inside MCP clients, use `browser_profile_onboard` for customer setup and reconnects. The customer path does not import the user's local Chrome profile: it uses the hosted `watch_url` so the user controls the login session directly.
 
 Claude Code one-command setup:
 
@@ -213,11 +201,11 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 ### Browser-agent tools
 
-- `browser_open` — open a browser session. Hosted mode returns a human `watch_url`; local mode opens Google Chrome on this machine against an imported managed profile.
-- `browser_profile_list` — list local Chrome account/profile metadata and suggested managed profile names. This does not read cookies, passwords, browsing history, or copy local Chrome state.
-- `browser_profile_import` — clone a local Chrome profile into `~/.mcp-scraper/browser-profiles` for local browser mode. This copies browser state files but skips cache and locks.
-- `browser_profile_sync` — refresh an existing managed local profile clone from its recorded source Chrome profile.
-- `browser_profile_onboard` — create or load a hosted Kernel browser profile and open a setup browser with profile saving enabled. The user logs in through the `watch_url`, then `browser_close` persists cookies/local storage into the hosted Kernel profile.
+- `browser_open` — open a direct/no-proxy hosted browser session. Pass a saved profile name to load an authenticated session. Hosted mode returns a human `watch_url`.
+- `browser_profile_onboard` — create or load a saved hosted browser profile and open a direct/no-proxy setup browser at Google with profile saving enabled. The user clicks Sign in through the `watch_url`, then `browser_close` persists cookies/local storage into the saved profile. Use this again whenever the profile disconnects.
+- `browser_profile_list` — developer-only local Chrome profile metadata helper. Do not use for customer auth setup.
+- `browser_profile_import` — developer-only local profile clone helper. Do not use for customer auth setup.
+- `browser_profile_sync` — developer-only local profile refresh helper. Do not use for customer auth setup.
 - `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds.
 - `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image.
 - `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds.
@@ -258,12 +246,12 @@ The `mcp-scraper` and `mcp-scraper-combined` NPX stdio servers also expose saved
 - `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`.
 - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
 - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
-- `MCP_SCRAPER_BROWSER_MODE=local` makes `browser_open` use local Google Chrome instead of the hosted browser service.
-- `MCP_SCRAPER_BROWSER_PROFILE` sets the default managed local browser profile name created by `mcp-scraper-cli browser import-chrome`.
-- `MCP_SCRAPER_BROWSER_EXECUTABLE` optionally points local browser mode at a Chrome executable. On macOS, Google Chrome is recommended so Keychain-backed Chrome state works.
-- `MCP_SCRAPER_BROWSER_PROFILE_DIR` optionally points local browser mode at a direct Chrome-compatible user data directory instead of a managed profile manifest.
-- Instagram tools accept `browserMode` (`auto`, `hosted`, `local`) and `browserProfile`; `browserProfile` uses the same managed local profiles created by `browser_profile_import/sync`.
-- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved Kernel browser profile for hosted-mode `browser-agent` and `mcp-scraper-combined` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
+- `MCP_SCRAPER_BROWSER_MODE=local` is a developer-only override that makes `browser_open` use local Google Chrome instead of the hosted browser service. Do not set this for customer AI visibility setup.
+- `MCP_SCRAPER_BROWSER_PROFILE` is a developer-only local browser profile override. Hosted customer setup uses `BROWSER_AGENT_PROFILE_NAME` and `browser_profile_onboard`.
+- `MCP_SCRAPER_BROWSER_EXECUTABLE` optionally points developer-only local browser mode at a Chrome executable.
+- `MCP_SCRAPER_BROWSER_PROFILE_DIR` optionally points developer-only local browser mode at a direct Chrome-compatible user data directory.
+- Instagram tools accept `browserMode` (`auto`, `hosted`, `local`) and `browserProfile`; local mode is developer-only.
+- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `browser-agent` and `mcp-scraper-combined` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
 - `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup mode. It persists cookies and local storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
 
 Every web intelligence tool call made through `mcp-scraper` or `mcp-scraper-combined` saves a full Markdown report locally by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.

package/dist/bin/api-server.cjs

@@ -22080,7 +22080,7 @@ var PACKAGE_VERSION;
 var init_version = __esm({
   "src/version.ts"() {
     "use strict";
-    PACKAGE_VERSION = "0.3.3";
+    PACKAGE_VERSION = "0.3.5";
   }
 });
 
@@ -24800,11 +24800,16 @@ async function ensureProfile(k, name) {
     if (!isProfileConflict(err)) throw err;
   }
 }
+function explicitProxyId(value) {
+  const trimmed = value?.trim();
+  return trimmed || void 0;
+}
 async function createSession(opts = {}) {
   const k = client();
-  const resolvedProxyId = opts.proxyId ?? browserServiceProxyId();
+  const resolvedProxyId = explicitProxyId(opts.proxyId);
   const resolvedProfileName = opts.profileName ?? browserServiceProfileName();
   const resolvedSaveProfileChanges = opts.saveProfileChanges ?? browserServiceProfileSaveChanges();
+  const disableDefaultProxy = opts.disableDefaultProxy ?? !resolvedProxyId;
   if (resolvedProfileName && resolvedSaveProfileChanges === true) {
     await ensureProfile(k, resolvedProfileName);
   }
@@ -24820,7 +24825,7 @@ async function createSession(opts = {}) {
     } : {}
   });
   const runtimeSessionId = browser.session_id;
-  if (opts.disableDefaultProxy) {
+  if (disableDefaultProxy) {
     try {
       await k.browsers.update(runtimeSessionId, { disable_default_proxy: true });
     } catch {
@@ -25221,7 +25226,7 @@ function buildBrowserAgentRoutes() {
         proxyId: typeof body.proxy_id === "string" ? body.proxy_id : void 0,
         profileName: typeof body.profile === "string" ? body.profile : void 0,
         saveProfileChanges: typeof body.save_profile_changes === "boolean" ? body.save_profile_changes : void 0,
-        disableDefaultProxy: body.disable_default_proxy === true,
+        disableDefaultProxy: typeof body.disable_default_proxy === "boolean" ? body.disable_default_proxy : void 0,
         viewport: body.viewport && typeof body.viewport === "object" ? body.viewport : void 0
       });
       runtimeSessionId = created.runtimeSessionId;
@@ -25653,7 +25658,7 @@ async function selectSession(id) {
 }
 
 async function openSession() {
-  const r = await api('POST', '/sessions', { label: 'console' });
+  const r = await api('POST', '/sessions', { label: 'console', disable_default_proxy: true });
   if (r.ok) { await refreshSessions(); selectSession(r.data.session_id); }
   else alert('Open failed: ' + JSON.stringify(r.data));
 }