mcp-scraper 0.2.20 → 0.2.21

package/README.md

@@ -36,7 +36,10 @@ Run setup checks and generate agent wiring:
 
 ```bash
 npx -y -p mcp-scraper@latest mcp-scraper-cli doctor
+npx -y -p mcp-scraper@latest mcp-scraper-cli browser profiles --email seo@example.com
+npx -y -p mcp-scraper@latest mcp-scraper-cli browser import-chrome --email seo@example.com --name seo-example-com
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent install codex
+npx -y -p mcp-scraper@latest mcp-scraper-cli agent install claude-desktop --browser-mode local --browser-profile seo-example-com
 npx -y -p mcp-scraper@latest mcp-scraper-cli agent prompt agent-packet
 ```
 
@@ -102,7 +105,10 @@ Claude Desktop:
       "command": "npx",
       "args": ["-y", "-p", "mcp-scraper@latest", "mcp-scraper-combined"],
       "env": {
-        "MCP_SCRAPER_API_KEY": "sk_live_your_key"
+        "MCP_SCRAPER_API_KEY": "sk_live_your_key",
+        "MCP_SCRAPER_BROWSER_MODE": "local",
+        "MCP_SCRAPER_BROWSER_PROFILE": "work-accounts",
+        "MCP_SCRAPER_BROWSER_EXECUTABLE": "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"
       }
     }
   }
@@ -111,6 +117,21 @@ Claude Desktop:
 
 Existing MCP configs that use only `npx -y mcp-scraper` still work for the web intelligence server, but they do not automatically add browser tools. Switch to `mcp-scraper-combined` or add the second `browser-agent` config entry if you want browser tools. Use `mcp-scraper@latest` to force npm to resolve the newest published package whenever the MCP client starts a fresh `npx` process.
 
+For local Chrome state, first clone a Chrome profile into MCP Scraper managed storage:
+
+```bash
+mcp-scraper-cli browser import-chrome --email you@example.com --name work-accounts
+mcp-scraper-cli agent install codex --browser-mode local --browser-profile work-accounts
+```
+
+`browser import-chrome` copies the selected local Chrome profile into `~/.mcp-scraper/browser-profiles/<name>/user-data`, skipping cache and lock files. The managed clone can include cookies, local storage, history, session storage, and Chrome password database files. It does not upload the profile. Re-run `mcp-scraper-cli browser sync-profile <name>` after logging into new sites in normal Chrome.
+
+Set `MCP_SCRAPER_BROWSER_MODE=local` and `MCP_SCRAPER_BROWSER_PROFILE=<name>` when you want `browser_open` to launch local Google Chrome against that managed clone. A `profile` argument passed directly to `browser_open` overrides the default for that one session.
+
+Hosted browser profiles are still available. `BROWSER_AGENT_PROFILE_NAME` sets the default saved Kernel browser profile for hosted sessions. For first-time hosted setup, set `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true`, open a browser with the profile, complete login through the watch URL, then call `browser_close` to persist cookies/local storage into the hosted Kernel profile.
+
+Inside MCP clients, use `browser_profile_list` to inspect local Chrome account/profile metadata, `browser_profile_import` to clone a profile for local browser mode, `browser_profile_sync` to refresh an existing clone, and `browser_profile_onboard` only for hosted Kernel setup.
+
 Claude Code:
 
 ```bash
@@ -175,7 +196,11 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 
 ### Browser-agent tools
 
-- `browser_open` — open a live cloud browser session. Returns a `session_id`, a human `watch_url`, and the raw `live_view_url` when available.
+- `browser_open` — open a browser session. Hosted mode returns a human `watch_url`; local mode opens Google Chrome on this machine against an imported managed profile.
+- `browser_profile_list` — list local Chrome account/profile metadata and suggested managed profile names. This does not read cookies, passwords, browsing history, or copy local Chrome state.
+- `browser_profile_import` — clone a local Chrome profile into `~/.mcp-scraper/browser-profiles` for local browser mode. This copies browser state files but skips cache and locks.
+- `browser_profile_sync` — refresh an existing managed local profile clone from its recorded source Chrome profile.
+- `browser_profile_onboard` — create or load a hosted Kernel browser profile and open a setup browser with profile saving enabled. The user logs in through the `watch_url`, then `browser_close` persists cookies/local storage into the hosted Kernel profile.
 - `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds.
 - `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image.
 - `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds.
@@ -203,7 +228,7 @@ The MCPB bundle and `mcp-scraper-combined` expose both sections through one loca
 
 All local and hosted MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their local file side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split local `mcp-scraper` stdio server exposes 22 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 17 browser tools. The combined local stdio server exposes 39 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 22 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (24 total). Across all public local and hosted surfaces there are 41 unique tool names.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split local `mcp-scraper` stdio server exposes 22 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 21 browser tools. The combined local stdio server exposes 43 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 22 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (24 total). Across all public local and hosted surfaces there are 45 unique tool names.
 
 ## Resources
 
@@ -216,6 +241,12 @@ The `mcp-scraper` and `mcp-scraper-combined` NPX stdio servers also expose saved
 - `MCP_SCRAPER_OUTPUT_DIR` is optional and defaults to `~/Downloads/mcp-scraper`.
 - `MCP_SCRAPER_SAVE_REPORTS=false` disables automatic Markdown report files.
 - `MCP_SCRAPER_KEY_PATH` is optional. When no API key env var is set, the server also reads `~/.mcp-scraper-key` for compatibility with older installs.
+- `MCP_SCRAPER_BROWSER_MODE=local` makes `browser_open` use local Google Chrome instead of the hosted browser service.
+- `MCP_SCRAPER_BROWSER_PROFILE` sets the default managed local browser profile name created by `mcp-scraper-cli browser import-chrome`.
+- `MCP_SCRAPER_BROWSER_EXECUTABLE` optionally points local browser mode at a Chrome executable. On macOS, Google Chrome is recommended so Keychain-backed Chrome state works.
+- `MCP_SCRAPER_BROWSER_PROFILE_DIR` optionally points local browser mode at a direct Chrome-compatible user data directory instead of a managed profile manifest.
+- `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved Kernel browser profile for hosted-mode `browser-agent` and `mcp-scraper-combined` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
+- `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup mode. It persists cookies and local storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
 
 Every web intelligence tool call made through `mcp-scraper` or `mcp-scraper-combined` saves a full Markdown report locally by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
 

package/dist/bin/api-server.cjs

@@ -59,6 +59,17 @@ function browserServiceProxyId() {
   const value = (process.env.BROWSER_SERVICE_PROXY_ID ?? process.env.KERNEL_PROXY_ID)?.trim();
   return value || void 0;
 }
+function browserServiceProfileName() {
+  const value = (process.env.BROWSER_AGENT_PROFILE_NAME ?? process.env.BROWSER_SERVICE_PROFILE_NAME ?? process.env.KERNEL_BROWSER_PROFILE_NAME ?? process.env.KERNEL_PROFILE_NAME)?.trim();
+  return value || void 0;
+}
+function browserServiceProfileSaveChanges() {
+  const value = (process.env.BROWSER_AGENT_PROFILE_SAVE_CHANGES ?? process.env.BROWSER_SERVICE_PROFILE_SAVE_CHANGES ?? process.env.KERNEL_BROWSER_PROFILE_SAVE_CHANGES ?? process.env.KERNEL_PROFILE_SAVE_CHANGES)?.trim().toLowerCase();
+  if (!value) return void 0;
+  if (["1", "true", "yes", "on"].includes(value)) return true;
+  if (["0", "false", "no", "off"].includes(value)) return false;
+  return void 0;
+}
 var init_browser_service_env = __esm({
   "src/lib/browser-service-env.ts"() {
     "use strict";
@@ -2465,6 +2476,16 @@ var init_registry = __esm({
   }
 });
 
+// src/attribution-pixel.ts
+var ATTRIBUTION_PIXEL_SRC, ATTRIBUTION_PIXEL_SCRIPT;
+var init_attribution_pixel = __esm({
+  "src/attribution-pixel.ts"() {
+    "use strict";
+    ATTRIBUTION_PIXEL_SRC = "https://hyros-attribution-crm.vercel.app/api/track/sdk?token=35c27ba33dd0c6cc640f9fef415ee3dc9ead67fefe99d269be602ee269900e25";
+    ATTRIBUTION_PIXEL_SCRIPT = `<script src="${ATTRIBUTION_PIXEL_SRC}" async></script>`;
+  }
+});
+
 // src/blog/template.ts
 function faqJsonLd(post) {
   const firstFive = post.sections.flatMap((s) => s.cards).slice(0, 5);
@@ -2599,6 +2620,7 @@ function renderPost(post) {
   <script type="application/ld+json">${faqJsonLd(post)}</script>
   <script type="application/ld+json">${breadcrumbJsonLd(post)}</script>
 
+  ${ATTRIBUTION_PIXEL_SCRIPT}
   <style>${CSS}</style>
 </head>
 <body>
@@ -2742,6 +2764,7 @@ var CSS;
 var init_template = __esm({
   "src/blog/template.ts"() {
     "use strict";
+    init_attribution_pixel();
     CSS = `
     @font-face {
       font-family: "GT Flexa";
@@ -10706,6 +10729,7 @@ function buildTranscriptHtml(result) {
     (ch) => `<div class="chunk"><span class="ts">${fmtSecs(ch.timestamp[0])} \u2192 ${fmtSecs(ch.timestamp[1])}</span><span class="ct">${esc(ch.text.trim())}</span></div>`
   ).join("\n");
   return `<!doctype html><html><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
+${ATTRIBUTION_PIXEL_SCRIPT}
 <style>body{font-family:system-ui,sans-serif;max-width:800px;margin:1.5rem auto;padding:0 1rem;color:#111;line-height:1.7}
 h1{font-size:1.2rem;margin-bottom:.25rem}.meta{color:#666;font-size:.85rem;margin-bottom:1.5rem}
 .full-text{background:#f5f5f3;border-radius:8px;padding:1rem 1.25rem;font-size:.875rem;white-space:pre-wrap;margin-bottom:2rem;word-break:break-word}
@@ -10735,6 +10759,7 @@ var init_youtube_routes = __esm({
     init_CaptionFetcher();
     init_BrowserDriver();
     init_errors();
+    init_attribution_pixel();
     init_api_auth();
     init_server_schemas();
     init_concurrency_gates();
@@ -22677,14 +22702,35 @@ function client() {
   if (!apiKey) throw new Error("Browser backend API key is required");
   return new import_sdk6.default({ apiKey });
 }
+function isProfileConflict(err) {
+  const message = err instanceof Error ? err.message : String(err);
+  return /\b409\b|conflict|already exists/i.test(message);
+}
+async function ensureProfile(k, name) {
+  try {
+    await k.profiles.create({ name });
+  } catch (err) {
+    if (!isProfileConflict(err)) throw err;
+  }
+}
 async function createSession(opts = {}) {
   const k = client();
   const resolvedProxyId = opts.proxyId ?? browserServiceProxyId();
+  const resolvedProfileName = opts.profileName ?? browserServiceProfileName();
+  const resolvedSaveProfileChanges = opts.saveProfileChanges ?? browserServiceProfileSaveChanges();
+  if (resolvedProfileName && resolvedSaveProfileChanges === true) {
+    await ensureProfile(k, resolvedProfileName);
+  }
   const browser = await k.browsers.create({
     stealth: opts.stealth ?? true,
     timeout_seconds: opts.timeoutSeconds ?? DEFAULT_TIMEOUT_SECONDS,
     ...resolvedProxyId ? { proxy_id: resolvedProxyId } : {},
-    ...opts.profileName ? { profile: { name: opts.profileName } } : {}
+    ...resolvedProfileName ? {
+      profile: {
+        name: resolvedProfileName,
+        ...typeof resolvedSaveProfileChanges === "boolean" ? { save_changes: resolvedSaveProfileChanges } : {}
+      }
+    } : {}
   });
   const runtimeSessionId = browser.session_id;
   if (opts.disableDefaultProxy) {
@@ -23075,6 +23121,7 @@ function buildBrowserAgentRoutes() {
         timeoutSeconds,
         proxyId: typeof body.proxy_id === "string" ? body.proxy_id : void 0,
         profileName: typeof body.profile === "string" ? body.profile : void 0,
+        saveProfileChanges: typeof body.save_profile_changes === "boolean" ? body.save_profile_changes : void 0,
         disableDefaultProxy: body.disable_default_proxy === true,
         viewport: body.viewport && typeof body.viewport === "object" ? body.viewport : void 0
       });
@@ -23450,6 +23497,7 @@ function renderConsoleHtml(initialSessionId) {
   .gate button { width: 100%; }
   .muted { color: #6b7689; font-size: 12px; }
 </style>
+${ATTRIBUTION_PIXEL_SCRIPT}
 </head>
 <body>
 <div id="app"></div>
@@ -23599,6 +23647,7 @@ if (state.key) { refreshSessions(); if (state.current) selectSession(state.curre
 var init_browser_agent_console = __esm({
   "src/api/browser-agent-console.ts"() {
     "use strict";
+    init_attribution_pixel();
   }
 });
 
@@ -24157,6 +24206,7 @@ var init_server = __esm({
     init_outbound_sanitize();
     init_registry();
     init_template();
+    init_attribution_pixel();
     init_og();
     import_resend = require("resend");
     init_url_utils();
@@ -25000,6 +25050,7 @@ var init_server = __esm({
   .i6 em{font-family:"GTF";font-weight:700;font-style:italic}
   .i7 em{font-family:"GTF";font-weight:900;font-style:italic}
 </style>
+${ATTRIBUTION_PIXEL_SCRIPT}
 </head>
 <body>