chrometools-mcp 3.5.6 → 3.6.0

package/CHANGELOG.md

@@ -2,6 +2,21 @@
 
 All notable changes to this project will be documented in this file.
 
+## [3.6.0] - 2026-06-25
+
+### Added
+- **iframe automation (`switchFrame` / `listFrames`)** — Automate pages whose UI lives in an `<iframe>`, including **cross-origin** ones (resolved over CDP, bypassing Same-Origin Policy). `switchFrame({ frameUrl | frameSelector })` sets an active frame so `click`, `type`, `hover`, `selectOption`, `pressKey`, `scrollTo`, `waitForElement`, `analyzePage`, `findElementsByText`, `smartFindElement`, `executeScript` all run inside it; call `switchFrame()` with no args to reset. `listFrames()` lists frames; `analyzePage` also reports a `frames` array when >1 frame exists. Active frame auto-resets on `navigateTo`
+- **Browser connection env vars** — `CHROMETOOLS_BROWSER_WS_ENDPOINT` (connect directly to a CDP WebSocket), `CHROMETOOLS_DEBUG_PORT` (default 9222), `CHROMETOOLS_USER_DATA_DIR` (use a real/logged-in Chrome profile instead of a blank temp one), `CHROMETOOLS_CHROME_PATH`. Lets the server attach to your authenticated Chrome session — required for sites that need real cookies/cross-subdomain session
+- **`smartFindElement({ minConfidence })`** — Confidence threshold (default 0.6) gating auto-`action`. If the best match is below it or too close to the runner-up, the action is skipped and candidates are returned with an `actionSkipped` reason. Candidate coverage broadened (`[onclick]`, `[role=menuitem]`, `[role=tab]`, nav links); scoring penalizes text that doesn't match the description and rewards nav/menu context
+- **`click({ waitForRouteChange })`** — For SPAs: after click, waits for `location.pathname+search` to change and reports `routeChanged:true/false` (never fails the click on timeout)
+
+### Changed
+- **`executeScript` top-level `return`** — Now runs the snippet as-is first and only re-wraps in an async IIFE if the engine reports an "Illegal return statement". Handles `const x=…; return x`, callbacks containing `function`, and bare expressions — replaces the fragile `^return` + no-`function` heuristic
+- **`navigateTo` network summary** — Lists only XHR/Fetch requests (static assets hidden and counted), capped at 12 with a `… N more` note, instead of dumping every chunk/CSS/font
+
+### Fixed
+- **`findElementsByText({ action })` in iframes** — The action path resolved the selector against the main frame even after `switchFrame`, causing `Element not found for action`. Now resolves inside the active frame
+
 ## [3.5.6] - 2026-05-28
 
 ### Added

package/README.md

@@ -219,13 +219,14 @@ The Chrome Extension is **required** for scenario recording and other advanced f
 - [Installation](#installation)
   - [Chrome Extension Setup](#chrome-extension-setup)
 - [AI Optimization Features](#ai-optimization-features)- [Scenario Recorder](#scenario-recorder)  - Visual UI-based recording with smart optimization
-- [Available Tools](#available-tools) - **49+ Tools Total**
+- [Available Tools](#available-tools) - **51+ Tools Total**
   - [AI-Powered Tools](#ai-powered-tools)  - smartFindElement, analyzePage, getElementDetails, findElementsByText
   - [Core Tools](#1-core-tools) - ping, openBrowser
   - [Interaction Tools](#2-interaction-tools) - click, type, scrollTo, selectOption, selectFromGroup, drag, scrollHorizontal
   - [Inspection Tools](#3-inspection-tools) - getElement, getComputedCss, getBoxModel, screenshot
   - [Advanced Tools](#4-advanced-tools) - executeScript, getConsoleLogs, listNetworkRequests, getNetworkRequest, filterNetworkRequests, hover, pressKey, setStyles, setViewport, getViewport, navigateTo
   - [Tab Management Tools](#5-tab-management-tools)  - listTabs, switchTab
+  - [Frame Tools](#5a-frame-tools-iframe-automation)  - listFrames, switchFrame (cross-origin iframe automation)
   - [Recorder Tools](#7-recorder-tools)  - enableRecorder, executeScenario, listScenarios, searchScenarios, getScenarioInfo, deleteScenario, exportScenarioAsCode, appendScenarioToFile, generatePageObject
   - [API / Swagger Tools](#8-api--swagger-tools) - loadSwagger, generateApiModels
 - [Typical Workflow Example](#typical-workflow-example)
@@ -357,6 +358,8 @@ executeScenario({ name: "login_flow", parameters: { email: "user@test.com" } })
 - **Parameters**:
   - `description` (required): Natural language (e.g., "login button", "email field")
   - `maxResults` (optional): Max candidates to return (default: 5)
+  - `minConfidence` (optional): Confidence threshold (0–1, default **0.6**) for auto-executing `action`. If the best match scores below this — or is too close to the runner-up — the action is **skipped** and candidates are returned with an `actionSkipped` reason instead. Prevents auto-clicking the wrong control (e.g. a primary form submit when you asked for a menu item). Lower it to act on weaker matches.
+- **Candidate coverage**: besides `button`/`input`/`a`/`[role=button]`, also considers `[onclick]`, `[role=menuitem]`, `[role=tab]`, and links inside `nav`/`[role=navigation]`/`[role=menu]` — so menu items rendered as `div`/`span[onclick]` are found. Scoring penalizes candidates whose text doesn't match the description and rewards navigation/menu context.
 - **Use case**: When you don't know the exact selector
 - **Returns**: Ranked candidates with confidence scores, selectors, and reasoning
 - **Example**:
@@ -519,6 +522,7 @@ Click an element with optional result screenshot. **PREFERRED**: Use APOM ID fro
   - `networkWaitTimeout` (optional): Custom network wait timeout in ms (default: 10000). Only used if skipNetworkWait is false.
   - `waitForSelector` (optional): CSS selector to wait for **after** the click — atomic click+wait. Use for dropdowns/popups that render into a React Portal and otherwise race with the next MCP call. Example: `click({ id: 'button_47', waitForSelector: '#menu-popup-root > div' })`.
   - `waitTimeoutMs` (optional): Timeout for `waitForSelector` in ms (default: 2000). On timeout the click still succeeds but the result text reports `⚠️ WAIT_TIMEOUT`.
+  - `waitForRouteChange` (optional): For SPAs (React Router etc.). After the click, waits for `location.pathname + location.search` to change relative to before, then reports `Route changed: "/a" → "/b"` or `⚠️ ROUTE_UNCHANGED`. Because SPAs navigate via `history.pushState`, plain network diagnostics may not register the navigation — this makes "success" mean the view actually changed, not just that the click was delivered. Never fails the click on timeout. For view changes that don't alter the URL, use `waitForSelector` instead.
   - `autoAnalyzeAfter` (optional): After click, automatically diff APOM and append the delta to the result text (e.g. `+3 appeared: button_42:"Статистика", button_43:"Настройки", link_44:"Удалить"`). New element ids are pre-registered so the next `click({ id })`/`type({ id })` call works **without an extra `analyzePage`**. Designed for the dropdown/menu pattern: one MCP call instead of three.
 - **Use case**: Buttons, links, form submissions, Django admin forms
 - **Returns**: Confirmation text + optional screenshot + network diagnostics
@@ -725,7 +729,8 @@ Execute arbitrary JavaScript in page context with optional screenshot.
 - **Use case**: Complex interactions, custom manipulations
 - **Returns**: Execution result + optional screenshot
 - **Performance**: 2-10x faster without screenshot
-- **Top-level `return`**: snippets that start with `return ...` (e.g. `return document.title`) are auto-wrapped in an async IIFE — no need to manually wrap in `(() => { ... })()`. Scripts that declare a `function` are left unmodified so implicit-return patterns keep working.
+- **Top-level `return`**: any snippet using a top-level `return` (e.g. `return document.title`, `const x = 1; return x`, or code that contains a `function` in a callback) just works — it's run as-is first, and only re-wrapped in an async IIFE if the engine reports an "Illegal return statement". Bare expressions like `document.title` still return their value. No manual `(() => { ... })()` wrapping needed.
+- **Frames**: runs inside the active frame after `switchFrame` (see [Frame Tools](#5a-frame-tools-iframe-automation)); defaults to the main frame.
 
 #### getConsoleLogs
 Retrieve browser console logs (log, warn, error, etc.).
@@ -848,7 +853,8 @@ Navigate to different URL while keeping browser instance.
   - `url` (required)
   - `waitUntil` (optional): load event type
 - **Use case**: Moving between pages in workflow
-- **Returns**: New page title
+- **Returns**: New page title. The post-navigation network summary lists only **XHR/Fetch** requests (static assets — JS chunks, CSS, fonts, images — are hidden and counted) and is capped at the first 12 with a `… N more` note, so a heavy SPA load doesn't bury the signal.
+- **Note**: resets the active frame back to the main frame (see [Frame Tools](#5a-frame-tools-iframe-automation)).
 
 ### 5. Tab Management Tools 
 Tools for managing multiple browser tabs. New tabs opened via `window.open()`, `target="_blank"`, or user actions are automatically detected and tracked.
@@ -891,6 +897,45 @@ switchTab({ tab: 0 })
 switchTab({ tab: "google.com" })
 ```
 
+### 5a. Frame Tools (iframe automation)
+
+Tools for automating pages whose UI lives inside an `<iframe>` — including **cross-origin** iframes (e.g. a widget hosted on a different subdomain). Cross-origin frames can't be reached from page JavaScript (Same-Origin Policy blocks `iframe.contentDocument`), but ChromeTools resolves them over CDP, so the SOP doesn't apply.
+
+By default all tools operate on the **main frame**. After `switchFrame`, the element tools — `click`, `type`, `hover`, `selectOption`, `pressKey`, `scrollTo`, `waitForElement`, `analyzePage`, `findElementsByText`, `smartFindElement`, `executeScript` — run **inside the selected frame** until you reset. The active frame is reset automatically on `navigateTo`.
+
+#### listFrames
+List all frames on the current page so you can discover which one to switch into.
+- **Parameters**: None
+- **Returns**: `{ count, activeFrame, frames: [{ url, name, isMain }], hint }`
+- **Use case**: Find a cross-origin iframe (e.g. `app.example.com`) before switching into it
+
+#### switchFrame
+Set the active frame for subsequent element tools. Call with **no arguments** to reset back to the main frame.
+- **Parameters** (provide one, or none to reset):
+  - `frameUrl` (optional): substring matched against each frame's URL (e.g. `"app.example.com"`). Selects the first match.
+  - `frameSelector` (optional): CSS selector of the `<iframe>` element; its content frame becomes active.
+- **Returns**: `{ active: <frame url>, matcher, frames }` (or `{ active: null }` on reset)
+- **Use case**: Enter a cross-origin iframe to click/fill a form rendered there
+
+```javascript
+// Discover frames
+listFrames()
+// → { count: 2, frames: [ {url:".../app", isMain:true}, {url:"https://app.example.com/...", isMain:false} ] }
+
+// Enter the cross-origin iframe
+switchFrame({ frameUrl: "app.example.com" })
+
+// Now element tools target the iframe
+analyzePage()                       // returns the iframe's APOM tree
+findElementsByText({ text: "Save" }) // searches inside the iframe
+click({ id: "button_3" })            // clicks inside the iframe
+
+// Back to the main document
+switchFrame()
+```
+
+> `analyzePage` also includes a `frames` array in its output whenever the page has more than one frame, so the agent can discover iframes without a separate `listFrames` call.
+
 ### 6. Figma Tools 
 Design-to-code validation, file browsing, design system extraction, and comparison tools with automatic 3 MB compression.
 
@@ -1614,14 +1659,14 @@ Each tool definition is sent to the AI in every request, consuming context token
 | Group | Description | Tools (count) |
 |-------|-------------|---------------|
 | `core` | Basic tools | `ping`, `openBrowser` (2) |
-| `interaction` | User interaction | `click`, `type`, `scrollTo`, `waitForElement`, `hover` (5) |
+| `interaction` | User interaction & frames | `click`, `type`, `scrollTo`, `waitForElement`, `hover`, `selectOption`, `selectFromGroup`, `drag`, `scrollHorizontal`, `switchFrame`, `listFrames` (11) |
 | `inspection` | Page inspection | `getComputedCss`, `getBoxModel`, `screenshot`, `saveScreenshot` (4) |
 | `debug` | Debugging & network | `getConsoleLogs`, `listNetworkRequests`, `getNetworkRequest`, `filterNetworkRequests` (4) |
 | `advanced` | Advanced automation & AI | `executeScript`, `setStyles`, `setViewport`, `getViewport`, `navigateTo`, `smartFindElement`, `analyzePage`, `findElementsByText` (8) |
 | `recorder` | Scenario recording | `enableRecorder`, `executeScenario`, `listScenarios`, `searchScenarios`, `getScenarioInfo`, `deleteScenario`, `exportScenarioAsCode`, `appendScenarioToFile`, `generatePageObject` (9) |
 | `figma` | Figma integration | `getFigmaFrame`, `compareFigmaToElement`, `getFigmaSpecs`, `parseFigmaUrl`, `listFigmaPages`, `searchFigmaFrames`, `getFigmaComponents`, `getFigmaStyles`, `getFigmaColorPalette`, `convertFigmaToCode` (10) |
 
-**Total:** 42 tools across 7 groups
+**Total:** 48 tools across 7 groups
 
 **Configuration:**
 
@@ -1746,6 +1791,48 @@ To use Figma tools, you need to configure your Figma Personal Access Token.
 
 ---
 
+## Browser Connection (use your real, logged-in Chrome)
+
+By default ChromeTools connects to a Chrome with remote debugging on port **9222**, and if none is found it launches a fresh Chrome with a **temporary profile** (no cookies, no logins). To automate a site that needs your **authenticated session** (and to reach cross-origin iframes that depend on it), point ChromeTools at your own Chrome via environment variables — all optional:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `CHROMETOOLS_BROWSER_WS_ENDPOINT` | _(unset)_ | Direct CDP WebSocket URL (e.g. `ws://127.0.0.1:9222/devtools/browser/<id>`). When set, ChromeTools connects straight to it, skipping port discovery and launch. |
+| `CHROMETOOLS_DEBUG_PORT` | `9222` | Remote-debugging port to discover/connect to (and to launch with). |
+| `CHROMETOOLS_USER_DATA_DIR` | `<temp>/chrome-mcp-profile` | Chrome profile used when launching a new instance. Point at a real (or cloned) profile to reuse its cookies/logins. |
+| `CHROMETOOLS_CHROME_PATH` | platform default | Path to the Chrome executable. |
+
+**Quick path — attach to your already-logged-in Chrome:**
+
+1. Fully close Chrome (a profile can't open a debug port while another Chrome holds it).
+2. Launch it with remote debugging on your normal profile:
+   ```bash
+   # Windows
+   chrome.exe --remote-debugging-port=9222
+   # macOS
+   "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome" --remote-debugging-port=9222
+   ```
+3. Log into the target site in that Chrome. ChromeTools will `connect()` to it — cross-origin iframes load authenticated, and `listTabs` sees your real tabs.
+
+```json
+{
+  "mcpServers": {
+    "chrometools": {
+      "type": "stdio",
+      "command": "npx",
+      "args": ["chrometools-mcp"],
+      "env": {
+        "CHROMETOOLS_USER_DATA_DIR": "C:/Users/you/chrome-mcp-real-profile"
+      }
+    }
+  }
+}
+```
+
+Without any of these variables, behavior is unchanged.
+
+---
+
 ## WSL Setup Guide
 
 If you're using **Windows Subsystem for Linux (WSL)**, special configuration is required to display Chrome GUI windows.

package/browser/browser-manager.js

@@ -9,7 +9,7 @@ import { spawn } from 'child_process';
 import http from 'http';
 import path from 'path';
 import { fileURLToPath } from 'url';
-import { getChromePath, getTempDir, isWSL, CHROME_DEBUG_PORT } from '../utils/platform-utils.js';
+import { getChromePath, getUserDataDir, isWSL, CHROME_DEBUG_PORT } from '../utils/platform-utils.js';
 import { handleNewTab, openPages, lastPage } from './page-manager.js';
 
 // Get extension path (use forward slashes for Chrome on Windows)
@@ -97,6 +97,30 @@ export async function getBrowser() {
         let browser;
         let endpoint;
 
+        // Highest priority: explicit CDP WebSocket endpoint (e.g. a real
+        // logged-in Chrome the user already launched). Skips port-discovery
+        // and spawning entirely — connect straight to the live session.
+        const wsEndpoint = process.env.CHROMETOOLS_BROWSER_WS_ENDPOINT;
+        if (wsEndpoint) {
+          browser = await puppeteer.connect({
+            browserWSEndpoint: wsEndpoint,
+            defaultViewport: null,
+          });
+          debugLog("Connected via CHROMETOOLS_BROWSER_WS_ENDPOINT:", wsEndpoint);
+          connectedToExistingChrome = true;
+          console.error("[chrometools-mcp] Connected to Chrome via CHROMETOOLS_BROWSER_WS_ENDPOINT.");
+
+          browser.on('disconnected', () => {
+            debugLog("Browser disconnected");
+            browserPromise = null;
+            connectedToExistingChrome = false;
+          });
+
+          setupNewTabTracking(browser);
+          scheduleBridgeReconnect();
+          return browser;
+        }
+
         // Try to connect to existing Chrome with remote debugging
         try {
           endpoint = await getChromeWebSocketEndpoint(CHROME_DEBUG_PORT, 2);
@@ -132,7 +156,7 @@ export async function getBrowser() {
 
         // Launch new Chrome with remote debugging enabled
         const chromePath = getChromePath();
-        const userDataDir = `${getTempDir()}/chrome-mcp-profile`;
+        const userDataDir = getUserDataDir();
 
         debugLog("Chrome path:", chromePath);
         debugLog("User data dir:", userDataDir);

package/browser/page-manager.js

@@ -346,6 +346,95 @@ export function setLastPage(page) {
   lastPage = page;
 }
 
+// ─────────────────────────────────────────────────────────────────────────────
+// Active frame state (P0-1: cross-origin iframe support)
+//
+// Tools default to the main frame. switchFrame() stores a *matcher* (not a Frame
+// object — Frames get invalidated when the iframe re-navigates) keyed by Page.
+// getTargetFrame() re-resolves the matcher against page.frames() on every call,
+// so it survives iframe reloads. Cleared on top-level navigation.
+// ─────────────────────────────────────────────────────────────────────────────
+const activeFrames = new WeakMap(); // Page -> { frameUrl?, frameSelector? }
+
+/**
+ * Set (or clear) the active frame matcher for a page.
+ * @param {Page} page
+ * @param {{frameUrl?: string, frameSelector?: string}|null} matcher - null resets to main frame
+ */
+export function setActiveFrame(page, matcher) {
+  if (!matcher || (!matcher.frameUrl && !matcher.frameSelector)) {
+    activeFrames.delete(page);
+  } else {
+    activeFrames.set(page, matcher);
+  }
+}
+
+/**
+ * Clear the active frame for a page (back to main frame).
+ * @param {Page} page
+ */
+export function clearActiveFrame(page) {
+  activeFrames.delete(page);
+}
+
+/**
+ * Get the matcher currently stored for a page (or null).
+ * @param {Page} page
+ * @returns {{frameUrl?: string, frameSelector?: string}|null}
+ */
+export function getActiveFrameMatcher(page) {
+  return activeFrames.get(page) || null;
+}
+
+/**
+ * Resolve the execution context (Frame) for a page based on its active matcher.
+ * Returns the main frame when no active frame is set. A Frame exposes the same
+ * surface tools rely on (.evaluate, .$, .$$, .click, .type, .waitForSelector),
+ * so handlers can use the returned context in place of `page` for DOM work.
+ * @param {Page} page
+ * @returns {Promise<Frame>} - Target Puppeteer Frame
+ */
+export async function getTargetFrame(page) {
+  const matcher = activeFrames.get(page);
+  if (!matcher) {
+    return page.mainFrame();
+  }
+
+  if (matcher.frameSelector) {
+    const handle = await page.$(matcher.frameSelector);
+    const frame = handle ? await handle.contentFrame() : null;
+    if (!frame) {
+      throw new Error(
+        `Active frame (selector "${matcher.frameSelector}") not found. Call listFrames() and switchFrame() again, or switchFrame() with no args to reset.`
+      );
+    }
+    return frame;
+  }
+
+  // frameUrl: substring match against all frames
+  const frame = page.frames().find(f => f.url().includes(matcher.frameUrl));
+  if (!frame) {
+    throw new Error(
+      `Active frame (url contains "${matcher.frameUrl}") not found. Call listFrames() and switchFrame() again, or switchFrame() with no args to reset.`
+    );
+  }
+  return frame;
+}
+
+/**
+ * List all frames on a page with metadata (for listFrames tool / analyzePage output).
+ * @param {Page} page
+ * @returns {Array<{url: string, name: string, isMain: boolean}>}
+ */
+export function listFramesForPage(page) {
+  const main = page.mainFrame();
+  return page.frames().map(f => ({
+    url: f.url(),
+    name: f.name() || '',
+    isMain: f === main,
+  }));
+}
+
 /**
  * Setup a new page with all monitoring (console, network, recorder)
  * Used for both explicitly created pages and auto-detected new tabs

package/element-finder-utils.js

@@ -177,6 +177,12 @@ function analyzeButtonContextInPage(element) {
 
     // Primary button indicators
     isPrimary: /primary|main|btn-primary|button-primary/i.test(element.className || ''),
+
+    // Navigation/menu context (P1-3): menu items and nav links must be able to
+    // out-score a form's primary submit button when the user asked for navigation.
+    hasMenuRole: element.getAttribute('role') === 'menuitem' || element.getAttribute('role') === 'tab',
+    hasOnClick: element.hasAttribute('onclick'),
+    isNavContext: element.closest('nav, [role="navigation"], [role="menu"], [role="menubar"], [role="tablist"]') !== null,
   };
 
   // Check if it's the last button in form
@@ -217,6 +223,10 @@ function scoreSubmitButton(element, context, description) {
   if (context.hasSubmitIcon) score += 5;        // submit icon
   if (context.isPrimary) score += 15;           // primary button style
 
+  // Navigation/menu context bonus (P1-3)
+  if (context.hasMenuRole) score += 15;         // role=menuitem/tab
+  if (context.isNavContext) score += 10;        // inside nav/menu container
+
   // Visibility bonus
   if (context.isVisible) score += 10;
 
@@ -235,6 +245,22 @@ function scoreSubmitButton(element, context, description) {
     score -= 20;
   }
 
+  // Text-mismatch penalty (P1-3): when the description carries actual words but
+  // the element's text neither contains nor is contained by it AND it doesn't hit
+  // a submit keyword, it's almost certainly the wrong control (e.g. asked
+  // "Настройки", element says "Пополнить"). Without this, a primary form submit
+  // wins on structural points alone and gets auto-clicked.
+  if (descLower.trim().length > 0 && text.length > 0) {
+    const textMatchesDesc = text.includes(descLower) || descLower.includes(text);
+    if (!textMatchesDesc && !matchesSubmitKeyword(context.text, description)) {
+      // Strong enough to overcome a full house of structural submit points
+      // (type=submit + in-form + last + primary + visible ≈ +115), so a button
+      // whose text is unrelated to the request can't be auto-clicked. Submit
+      // keywords (Save/Войти/…) are exempt above, so real submits aren't harmed.
+      score -= 60;
+    }
+  }
+
   return score;
 }
 
@@ -572,6 +598,18 @@ function explainScore(element, context, description, score) {
   if (context.isLink && !matchesSubmitKeyword(context.text, description)) {
     reasons.push('link without submit keyword (-20)');
   }
+  if (context.hasMenuRole) reasons.push('menu/tab role (+15)');
+  if (context.isNavContext) reasons.push('nav/menu context (+10)');
+  {
+    const t = (context.text || '').toLowerCase();
+    const d = description.toLowerCase();
+    if (d.trim().length > 0 && t.length > 0) {
+      const m = t.includes(d) || d.includes(t);
+      if (!m && !matchesSubmitKeyword(context.text, description)) {
+        reasons.push('text ≠ description (-60)');
+      }
+    }
+  }
 
   // Common reasons
   if (context.inForm) reasons.push('in form (+20)');