mcp-scraper 0.3.9 → 0.3.11

package/README.md

@@ -119,7 +119,7 @@ For customer AI visibility setup, use the hosted profile connection flow. Open t
 
 Browser-agent hosted sessions are direct/no-proxy by default. They do not inherit the configured SERP/Maps proxy. Use proxies for SERP/Maps tools when those tools need location or anti-bot handling, not for the interactive login browser.
 
-`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions. For first-time setup, use `browser_profile_onboard`, open the returned `watch_url`, complete login in the hosted browser, then call `browser_close` to persist cookies and storage into the saved profile.
+`BROWSER_AGENT_PROFILE_NAME` sets the default saved hosted browser profile for future sessions. For first-time setup, use `browser_profile_onboard`, open the returned `watch_url`, complete login, then call `browser_profile_status` until the connection is authenticated.
 
 Inside MCP clients, use `browser_profile_onboard` for customer setup and reconnects. The customer signs in through the hosted `watch_url` so they control the login session directly.
 
@@ -202,7 +202,8 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 ### Browser-agent tools
 
 - `browser_open` — open a direct/no-proxy hosted browser session. Pass a saved profile name to load an authenticated session. Returns a human `watch_url`.
-- `browser_profile_onboard` — create or load a saved hosted browser profile and open a direct/no-proxy setup browser at Google with profile saving enabled. The user clicks Sign in through the `watch_url`, then `browser_close` persists cookies and storage into the saved profile. Use this again whenever the profile disconnects.
+- `browser_profile_onboard` — create or load a saved hosted browser profile auth connection and return a user takeover `watch_url`. The user signs in through that URL, then `browser_profile_status` verifies the profile is authenticated. Use this again whenever the profile disconnects.
+- `browser_profile_status` — check whether a hosted browser profile auth connection is authenticated before launching it with `browser_open`.
 - `browser_screenshot` — capture a screenshot plus visible text and clickable element center coordinates and DOM bounds.
 - `browser_read` — read the current page text and elements with center coordinates and DOM bounds, without an image.
 - `browser_locate` — locate exact visible DOM elements or text ranges and return screenshot-pixel bounds.
@@ -217,6 +218,7 @@ env = { MCP_SCRAPER_API_KEY = "sk_live_your_key" }
 - `browser_replay_download` — download and save the replay MP4 under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
 - `browser_replay_mark` — while recording, locate a DOM target and return a replay-timed annotation object.
 - `browser_replay_annotate` — download a replay MP4, render timed boxes, circles, underlines, arrows, and labels using annotation objects from `browser_replay_mark` or exact bounds from `browser_locate`, and save a new annotated MP4 under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
+- `browser_capture_fanout` — capture ChatGPT/Claude AI-search fan-out from an open logged-in hosted session. Use `export=true` for durable artifacts. It writes full `fanout.json`, query/source/citation/domain/snippet CSVs, TSV, and `report.html` under `MCP_SCRAPER_OUTPUT_DIR/fanout`, returning paths relative to `MCP_SCRAPER_OUTPUT_DIR` so users can share portable paths.
 - `browser_close`
 - `browser_list_sessions`
 
@@ -230,7 +232,7 @@ The MCPB bundle and `mcp-scraper-combined` expose both sections through one MCP
 
 All MCP tools expose output schemas and return `structuredContent` with the IDs, URLs, CSV paths, transcripts, browser session handles, replay paths, artifacts, recipe fields, or blueprint fields needed by the next step. Browser Agent tools keep a JSON text block for older clients, but structured data is the primary contract. All tools carry MCP annotations; file-writing tools such as replay downloads and annotations state their filesystem side effects.
 
-The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split `mcp-scraper` stdio server exposes 24 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 19 browser tools. The combined stdio server exposes 43 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 24 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (26 total). Across all public surfaces there are 45 unique tool names.
+The canonical tool inventory is generated at `docs/mcp-tool-manifest.generated.json`. The split `mcp-scraper` stdio server exposes 24 web-intelligence/workflow tools. The split `browser-agent` stdio server exposes 20 browser tools. The combined stdio server exposes 44 tools. The hosted MCP endpoint at `https://mcpscraper.dev/mcp` exposes the 24 web-intelligence/workflow tools plus `capture_serp_snapshot` and `capture_serp_page_snapshots` (26 total). Across all public surfaces there are 46 unique tool names.
 
 ## Resources
 
@@ -246,7 +248,7 @@ The `mcp-scraper` and `mcp-scraper-combined` NPX stdio servers also expose saved
 - `BROWSER_AGENT_PROFILE_NAME` is optional and sets the default saved hosted browser profile for `browser-agent` and `mcp-scraper-combined` stdio sessions. Aliases: `BROWSER_SERVICE_PROFILE_NAME`, `KERNEL_BROWSER_PROFILE_NAME`, `KERNEL_PROFILE_NAME`.
 - `BROWSER_AGENT_PROFILE_SAVE_CHANGES=true` is optional hosted setup behavior. It persists cookies and storage back to the named profile when `browser_close` deletes the hosted browser session. Aliases: `BROWSER_SERVICE_PROFILE_SAVE_CHANGES`, `KERNEL_BROWSER_PROFILE_SAVE_CHANGES`, `KERNEL_PROFILE_SAVE_CHANGES`.
 
-Every web intelligence tool call made through `mcp-scraper` or `mcp-scraper-combined` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`.
+Every web intelligence tool call made through `mcp-scraper` or `mcp-scraper-combined` saves a full Markdown report to disk by default and returns the file path in the MCP response. The hosted `/mcp` endpoint returns reports inline only and never writes files. Browser replay downloads are saved by `browser_replay_download` under `MCP_SCRAPER_OUTPUT_DIR/browser-replays`. AI fan-out exports are saved by `browser_capture_fanout` with `export=true` under `MCP_SCRAPER_OUTPUT_DIR/fanout`, and return relative paths.
 
 ## Updating Existing Installs
 

package/dist/bin/api-server.cjs

@@ -21921,7 +21921,7 @@ var PACKAGE_VERSION;
 var init_version = __esm({
   "src/version.ts"() {
     "use strict";
-    PACKAGE_VERSION = "0.3.9";
+    PACKAGE_VERSION = "0.3.11";
   }
 });
 
@@ -24432,7 +24432,9 @@ function writeTable(path6, rows, delimiter) {
 }
 function exportFanout(enriched) {
   const stamp = safe(enriched.capturedAt.replace(/[:.]/g, "-"));
-  const dir = (0, import_node_path11.join)(outputBaseDir3(), "fanout", `${stamp}-${safe(enriched.platform)}`);
+  const outputDir = outputBaseDir3();
+  const relativeDir = (0, import_node_path11.join)("fanout", `${stamp}-${safe(enriched.platform)}`);
+  const dir = (0, import_node_path11.join)(outputDir, relativeDir);
   (0, import_node_fs8.mkdirSync)(dir, { recursive: true });
   const queryRows = enriched.queries.map((q, i) => ({ index: i + 1, query: q }));
   const citationRows = enriched.aggregates.citationOrder.map((c) => {
@@ -24440,25 +24442,32 @@ function exportFanout(enriched) {
     return { rank: c.rank, domain: c.domain, url: c.url, timesCited: c.timesCited, siteType: src?.siteType ?? "", title: src?.title ?? "" };
   });
   const sourceRows2 = enriched.browsedUrls.map((s) => ({ domain: s.domain, url: s.url, cited: s.cited, timesCited: s.timesCited, siteType: s.siteType, title: s.title, round: s.round ?? "" }));
+  const browsedOnlyRows = enriched.browsedOnly.map((s) => ({ domain: s.domain, url: s.url, cited: s.cited, timesCited: s.timesCited, siteType: s.siteType, title: s.title, round: s.round ?? "" }));
+  const snippetRows = enriched.snippets.map((s) => ({ domain: s.domain, url: s.url, title: s.title, text: s.text }));
   const domainRows = enriched.aggregates.topSites.map((d) => ({ domain: d.domain, count: d.count, cited: d.cited, timesCited: d.timesCited, siteType: d.siteType }));
-  const paths = {
-    dir,
-    json: (0, import_node_path11.join)(dir, "fanout.json"),
-    queriesCsv: (0, import_node_path11.join)(dir, "queries.csv"),
-    queriesTsv: (0, import_node_path11.join)(dir, "queries.tsv"),
-    citationsCsv: (0, import_node_path11.join)(dir, "citations.csv"),
-    sourcesCsv: (0, import_node_path11.join)(dir, "sources.csv"),
-    domainsCsv: (0, import_node_path11.join)(dir, "domains.csv"),
-    report: (0, import_node_path11.join)(dir, "report.html")
+  const relativePaths = {
+    relativeTo: "MCP_SCRAPER_OUTPUT_DIR or ~/Downloads/mcp-scraper",
+    dir: relativeDir,
+    json: (0, import_node_path11.join)(relativeDir, "fanout.json"),
+    queriesCsv: (0, import_node_path11.join)(relativeDir, "queries.csv"),
+    queriesTsv: (0, import_node_path11.join)(relativeDir, "queries.tsv"),
+    citationsCsv: (0, import_node_path11.join)(relativeDir, "citations.csv"),
+    sourcesCsv: (0, import_node_path11.join)(relativeDir, "sources.csv"),
+    browsedOnlyCsv: (0, import_node_path11.join)(relativeDir, "browsed-only.csv"),
+    snippetsCsv: (0, import_node_path11.join)(relativeDir, "snippets.csv"),
+    domainsCsv: (0, import_node_path11.join)(relativeDir, "domains.csv"),
+    report: (0, import_node_path11.join)(relativeDir, "report.html")
   };
-  (0, import_node_fs8.writeFileSync)(paths.json, JSON.stringify(enriched, null, 2));
-  writeTable(paths.queriesCsv, queryRows, ",");
-  writeTable(paths.queriesTsv, queryRows, "	");
-  writeTable(paths.citationsCsv, citationRows, ",");
-  writeTable(paths.sourcesCsv, sourceRows2, ",");
-  writeTable(paths.domainsCsv, domainRows, ",");
-  (0, import_node_fs8.writeFileSync)(paths.report, renderReportHtml(enriched));
-  return paths;
+  (0, import_node_fs8.writeFileSync)((0, import_node_path11.join)(outputDir, relativePaths.json), JSON.stringify(enriched, null, 2));
+  writeTable((0, import_node_path11.join)(outputDir, relativePaths.queriesCsv), queryRows, ",");
+  writeTable((0, import_node_path11.join)(outputDir, relativePaths.queriesTsv), queryRows, "	");
+  writeTable((0, import_node_path11.join)(outputDir, relativePaths.citationsCsv), citationRows, ",");
+  writeTable((0, import_node_path11.join)(outputDir, relativePaths.sourcesCsv), sourceRows2, ",");
+  writeTable((0, import_node_path11.join)(outputDir, relativePaths.browsedOnlyCsv), browsedOnlyRows, ",");
+  writeTable((0, import_node_path11.join)(outputDir, relativePaths.snippetsCsv), snippetRows, ",");
+  writeTable((0, import_node_path11.join)(outputDir, relativePaths.domainsCsv), domainRows, ",");
+  (0, import_node_fs8.writeFileSync)((0, import_node_path11.join)(outputDir, relativePaths.report), renderReportHtml(enriched));
+  return relativePaths;
 }
 function esc(s) {
   return String(s).replace(/&/g, "&amp;").replace(/</g, "&lt;").replace(/>/g, "&gt;").replace(/"/g, "&quot;");
@@ -24575,6 +24584,7 @@ async function runFanoutCapture(page, input) {
   const cap = new FanoutCdpCapture();
   await cap.attach(page);
   try {
+    if (input.reset) cap.reset();
     const baseline = cap.answerStreamCount();
     if (input.prompt) await sendPrompt(page, input.prompt).catch(() => false);
     const waitMs = input.wait_ms ?? (input.prompt ? 9e4 : 8e3);
@@ -24639,10 +24649,68 @@ async function ensureProfile(k, name) {
     if (!isProfileConflict(err)) throw err;
   }
 }
+function profileConnectionMatches(connection, profileName, domain) {
+  if (!connection || typeof connection !== "object") return false;
+  const data = connection;
+  return data.profile_name === profileName && data.domain === domain;
+}
+async function findProfileConnection(k, profileName, domain) {
+  for await (const connection of k.auth.connections.list({ profile_name: profileName, domain })) {
+    if (profileConnectionMatches(connection, profileName, domain)) return connection;
+  }
+  return null;
+}
+async function findOrCreateProfileConnection(k, opts) {
+  try {
+    return await k.auth.connections.create({
+      domain: opts.domain,
+      profile_name: opts.profileName,
+      ...opts.loginUrl ? { login_url: opts.loginUrl } : {}
+    });
+  } catch (err) {
+    if (!isProfileConflict(err)) throw err;
+    const existing = await findProfileConnection(k, opts.profileName, opts.domain);
+    if (existing) return existing;
+    throw err;
+  }
+}
+function profileOnboardResult(connection, hostedUrl = null, liveViewUrl = null) {
+  return {
+    connectionId: String(connection.id),
+    profileName: String(connection.profile_name),
+    domain: String(connection.domain),
+    status: String(connection.status),
+    flowStatus: typeof connection.flow_status === "string" ? connection.flow_status : null,
+    flowStep: typeof connection.flow_step === "string" ? connection.flow_step : null,
+    flowExpiresAt: typeof connection.flow_expires_at === "string" ? connection.flow_expires_at : null,
+    hostedUrl: hostedUrl ?? (typeof connection.hosted_url === "string" ? connection.hosted_url : null),
+    liveViewUrl: liveViewUrl ?? (typeof connection.live_view_url === "string" ? connection.live_view_url : null),
+    browserSessionId: typeof connection.browser_session_id === "string" ? connection.browser_session_id : null,
+    postLoginUrl: typeof connection.post_login_url === "string" ? connection.post_login_url : null
+  };
+}
 function explicitProxyId(value) {
   const trimmed = value?.trim();
   return trimmed || void 0;
 }
+async function startProfileOnboarding(opts) {
+  const k = client();
+  const connection = await findOrCreateProfileConnection(k, opts);
+  const login = await k.auth.connections.login(connection.id);
+  const state = await k.auth.connections.retrieve(connection.id);
+  return profileOnboardResult(state, login.hosted_url ?? null, login.live_view_url ?? null);
+}
+async function getProfileOnboardingStatus(input) {
+  const k = client();
+  let connection = null;
+  if (input.connectionId) {
+    connection = await k.auth.connections.retrieve(input.connectionId);
+  } else if (input.profileName && input.domain) {
+    connection = await findProfileConnection(k, input.profileName, input.domain);
+  }
+  if (!connection) throw new Error("profile auth connection not found");
+  return profileOnboardResult(connection);
+}
 async function createSession(opts = {}) {
   const k = client();
   const resolvedProxyId = explicitProxyId(opts.proxyId);
@@ -25045,6 +25113,56 @@ function buildBrowserAgentRoutes() {
     return next();
   });
   app2.use("*", auth);
+  app2.post("/profiles/onboard", async (c) => {
+    const body = await c.req.json().catch(() => ({}));
+    const profileName = typeof body.profile === "string" ? body.profile.trim() : "";
+    const domain = typeof body.domain === "string" ? body.domain.trim().replace(/^https?:\/\//, "").replace(/\/.*$/, "") : "chatgpt.com";
+    const loginUrl = typeof body.login_url === "string" ? body.login_url.trim() : void 0;
+    if (!profileName) return c.json({ error: "profile is required" }, 400);
+    if (!domain) return c.json({ error: "domain is required" }, 400);
+    try {
+      const result = await startProfileOnboarding({ profileName, domain, loginUrl });
+      return c.json({
+        connection_id: result.connectionId,
+        profile: result.profileName,
+        domain: result.domain,
+        status: result.status,
+        flow_status: result.flowStatus,
+        flow_step: result.flowStep,
+        flow_expires_at: result.flowExpiresAt,
+        hosted_url: result.hostedUrl,
+        live_view_url: result.liveViewUrl,
+        browser_session_id: result.browserSessionId,
+        post_login_url: result.postLoginUrl
+      });
+    } catch (err) {
+      return c.json(failure(err), 502);
+    }
+  });
+  app2.post("/profiles/status", async (c) => {
+    const body = await c.req.json().catch(() => ({}));
+    const connectionId = typeof body.connection_id === "string" ? body.connection_id.trim() : void 0;
+    const profileName = typeof body.profile === "string" ? body.profile.trim() : void 0;
+    const domain = typeof body.domain === "string" ? body.domain.trim().replace(/^https?:\/\//, "").replace(/\/.*$/, "") : void 0;
+    try {
+      const result = await getProfileOnboardingStatus({ connectionId, profileName, domain });
+      return c.json({
+        connection_id: result.connectionId,
+        profile: result.profileName,
+        domain: result.domain,
+        status: result.status,
+        flow_status: result.flowStatus,
+        flow_step: result.flowStep,
+        flow_expires_at: result.flowExpiresAt,
+        hosted_url: result.hostedUrl,
+        live_view_url: result.liveViewUrl,
+        browser_session_id: result.browserSessionId,
+        post_login_url: result.postLoginUrl
+      });
+    } catch (err) {
+      return c.json(failure(err), 404);
+    }
+  });
   app2.post("/sessions", async (c) => {
     const user = c.get("user");
     if (Number(user.balance_mc ?? 0) < BROWSER_OPEN_MIN_BALANCE_MC) {
@@ -25148,6 +25266,7 @@ function buildBrowserAgentRoutes() {
       prompt: typeof body.prompt === "string" ? body.prompt : void 0,
       wait_ms: typeof body.wait_ms === "number" ? body.wait_ms : void 0,
       first_party_domain: typeof body.first_party_domain === "string" ? body.first_party_domain : void 0,
+      reset: body.reset === true,
       export: body.export === true
     };
     const t0 = Date.now();