native-devtools-mcp 0.5.1 → 0.7.0

package/README.md

@@ -1,10 +1,10 @@
 # native-devtools-mcp
 
-`native-devtools-mcp` is a Model Context Protocol (MCP) server for computer use on macOS, Windows, and Android. It gives AI agents and MCP clients direct control over native desktop apps and Android devices through screenshots, OCR, accessibility-based text lookup, input simulation, window management, and ADB.
+`native-devtools-mcp` is a Model Context Protocol (MCP) server for computer use on macOS, Windows, and Android. It gives AI agents and MCP clients direct control over native desktop apps, Chrome/Electron browsers, and Android devices through screenshots, OCR, accessibility-based text lookup, input simulation, window management, Chrome DevTools Protocol (CDP), and ADB.
 
-Use it when browser-only automation is not enough: Electron apps, system dialogs, desktop tools, native app testing, and Android device workflows. It works with [Claude Desktop](https://claude.ai/download), [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Cursor](https://cursor.com), and other MCP-compatible clients.
+Use it when browser-only automation is not enough: Electron apps (Signal, Discord, VS Code), Chrome browser automation, system dialogs, desktop tools, native app testing, and Android device workflows. It works with [Claude Desktop](https://claude.ai/download), [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Cursor](https://cursor.com), and other MCP-compatible clients.
 
-Useful for MCP-based computer use, desktop automation, UI automation, native app testing, e2e testing, RPA, screen reading, mouse and keyboard control, and Android device automation.
+Useful for MCP-based computer use, desktop automation, browser automation, UI automation, native app testing, e2e testing, RPA, screen reading, mouse and keyboard control, Chrome DevTools Protocol automation, and Android device automation.
 
 ```bash
 npx -y native-devtools-mcp
@@ -15,6 +15,7 @@ npx -y native-devtools-mcp
 - `click`, `type_text`, `scroll`, `launch_app`, `quit_app`, and window management
 - `element_at_point` for inspecting accessible UI elements at screen coordinates
 - `load_image` + `find_image` for non-text UI elements such as icons and custom controls
+- Chrome/Electron automation via CDP: snapshots, click, fill, navigate, type, and tab management
 - Android screenshots, text lookup, input, and app control over ADB
 - Local execution: screenshots and input stay on the machine
 
@@ -25,7 +26,7 @@ npx -y native-devtools-mcp
 ![Platform](https://img.shields.io/badge/platform-macOS%20%7C%20Windows%20%7C%20Android-blue?style=flat-square)
 ![Downloads](https://img.shields.io/npm/dt/native-devtools-mcp?style=flat-square)
 
-[Features](#-features) • [Installation](#-installation) • [Getting Started](#-getting-started) • [Recipes](#-recipes-and-examples) • [Security & Trust](#-security--trust) • [For AI Agents](#-for-ai-agents-llms) • [Android](#-android-support)
+[Features](#-features) • [Installation](#-installation) • [Getting Started](#-getting-started) • [Recipes](#-recipes-and-examples) • [Security & Trust](#-security--trust) • [For AI Agents](#-for-ai-agents-llms) • [Chrome/Electron (CDP)](#-browser-automation-cdp) • [Android](#-android-support)
 
 <div align="center">
 <table>
@@ -50,10 +51,12 @@ npx -y native-devtools-mcp
 - **🧩 Template Matching:** Find non-text UI elements (icons, shapes) using `load_image` + `find_image`, returning precise click coordinates.
 - **🔒 Local & Private:** 100% local execution. No screenshots or data are ever sent to external servers.
 - **📱 Android Support:** Connect to Android devices over ADB for screenshots, input simulation, UI element search, and app management — all from the same MCP server.
-- **🔍 Hover Tracking:** Track cursor hover transitions across UI elements in real-time. Configurable dwell threshold filters pass-through noise — designed for LLMs observing user navigation patterns. macOS only.
+- **🔍 Hover Tracking:** Track cursor hover transitions across UI elements in real-time. Configurable dwell threshold filters pass-through noise — designed for LLMs observing user navigation patterns.
+- **🌐 Browser Automation (CDP):** Connect to Chrome/Electron apps via Chrome DevTools Protocol. Take accessibility tree snapshots, click elements by UID, evaluate JavaScript, and manage tabs — all without a separate Node.js server.
 - **🔌 Dual-Mode Interaction:**
     1.  **Visual/Native:** Works with *any* app via screenshots & coordinates (Universal).
     2.  **AppDebugKit:** Deep integration for supported apps to inspect the UI tree (DOM-like structure).
+    3.  **CDP:** Connect to Chrome/Electron via `--remote-debugging-port` for DOM-level element targeting and JS evaluation.
 
 ## 🤖 For AI Agents (LLMs)
 
@@ -67,8 +70,10 @@ This MCP server is designed to be **highly discoverable and usable** by AI model
 3.  `find_text`: A shortcut to find text on screen and get its coordinates immediately. Uses the platform **accessibility API** (macOS Accessibility / Windows UI Automation) for precise element-level matching, with OCR fallback.
 4.  `element_at_point`: Inspect the accessibility element at given screen coordinates — returns name, role, label, value, bounds, pid, and app_name. Note: privacy-focused Electron apps (e.g. Signal) may restrict their AX tree, returning only a container — use `take_screenshot` with OCR as a fallback.
 5.  `load_image` / `find_image`: Template matching for non-text UI elements (icons, shapes), returning screen coordinates for clicking.
-6.  `start_hover_tracking` / `get_hover_events` / `stop_hover_tracking`: Track cursor hover transitions across UI elements. Configurable dwell threshold filters pass-throughs. macOS only.
-7.  `launch_app` / `quit_app`: Launch apps with optional CLI args, or gracefully/forcefully quit them.
+6.  `start_hover_tracking` / `get_hover_events` / `stop_hover_tracking`: Track cursor hover transitions across UI elements. Configurable dwell threshold filters pass-throughs.
+7.  `start_recording` / `stop_recording`: Record the frontmost app's window at ~5fps as timestamped JPEG frames. Automatically follows app switches.
+8.  `launch_app` / `quit_app`: Launch apps with optional CLI args, or gracefully/forcefully quit them.
+9.  `cdp_connect` / `cdp_take_snapshot` / `cdp_click` / `cdp_fill` / `cdp_navigate`: Connect to Chrome or Electron apps via CDP for DOM-level automation — snapshots, clicking, typing, navigation, and tab management without a separate Node.js server.
 
 ## 📦 Installation
 
@@ -247,6 +252,27 @@ Use `find_image` when the target is **not text** (icons, toggles, custom control
 
 Optional inputs like `mask_id`, `search_region`, `scales`, and `rotations` can improve precision and performance.
 
+## 🌐 Browser Automation (CDP)
+
+Connect to Chrome or Electron apps via the Chrome DevTools Protocol for DOM-level automation — more reliable than coordinate-based clicking for web content.
+
+```bash
+# Launch Chrome with remote debugging
+launch_app(app_name="Google Chrome", args=["--remote-debugging-port=9222", "--user-data-dir=/tmp/chrome-profile"])
+
+# Connect and automate
+cdp_connect(port=9222)
+cdp_navigate(url="https://example.com")
+cdp_take_snapshot()           # accessibility tree with element UIDs
+cdp_fill(uid="10", value="search query")
+cdp_press_key(key="Enter")
+cdp_wait_for(text=["Results"])
+```
+
+**16 CDP tools** — click, hover, fill, type, press key, navigate, handle dialogs, manage tabs, evaluate JS, and more. Works with Chrome 136+, Chromium, and Electron apps (Signal, Discord, VS Code, Slack). See [`AGENTS.md`](./AGENTS.md) for full tool reference.
+
+**Chrome 136+ note:** Requires `--user-data-dir=<path>` alongside `--remote-debugging-port` (Chrome silently ignores the debug port with the default profile). Electron apps only need `--remote-debugging-port`.
+
 ## 📱 Android Support
 
 Android support is built-in. The MCP server communicates with Android devices over ADB (USB or Wi-Fi), providing screenshots, input simulation, UI element search, and app management.
@@ -344,12 +370,15 @@ graph TD
 | | Input | `CGEvent` (CoreGraphics) |
 | | Text Search (`find_text`) | `Accessibility API` (primary), Vision OCR (fallback) |
 | | Element Inspection (`element_at_point`) | `AXUIElementCopyElementAtPosition` + AX tree walk fallback (Accessibility API) |
-| | Hover Tracking (`start_hover_tracking`) | `CGEvent` cursor + Accessibility API polling (macOS only) |
+| | Hover Tracking (`start_hover_tracking`) | `CGEvent` cursor + Accessibility API polling |
+| | Screen Recording (`start_recording`) | `CGWindowListCreateImage` at configurable fps |
 | | OCR | `VNRecognizeTextRequest` (Vision Framework) |
 | **Windows** | Screenshots | `BitBlt` (GDI) |
 | | Input | `SendInput` (Win32) |
 | | Text Search (`find_text`) | `UI Automation` (primary), WinRT OCR (fallback) |
 | | Element Inspection (`element_at_point`) | `IUIAutomation::ElementFromPoint` (UI Automation) |
+| | Hover Tracking (`start_hover_tracking`) | `GetCursorPos` + UI Automation polling |
+| | Screen Recording (`start_recording`) | `BitBlt` (GDI) at configurable fps |
 | | OCR | `Windows.Media.Ocr` (WinRT) |
 | **Android** | Screenshots | `screencap` / ADB framebuffer |
 | | Input | `adb shell input` (tap, swipe, text, keyevent) |
@@ -390,6 +419,7 @@ Works out of the box on **Windows 10/11**.
 *   `find_text` uses **UI Automation (UIA)** as the primary search mechanism, querying the accessibility tree for element names. This is the same accessibility-first approach used on macOS (with the Accessibility API). Falls back to OCR automatically when UIA finds no matches.
 *   OCR uses the built-in Windows Media OCR engine (offline).
 *   **Note:** Cannot interact with "Run as Administrator" windows unless the MCP server itself is also running as Administrator.
+*   **Screen Recording Performance:** Screen recording uses GDI/BitBlt at configurable fps (default 5). For higher fps requirements or game capture scenarios, DXGI Desktop Duplication API would provide hardware-accelerated capture — this is a planned future upgrade.
 
 ## 📜 License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "native-devtools-mcp",
-  "version": "0.5.1",
+  "version": "0.7.0",
   "mcpName": "io.github.sh3ll3x3c/native-devtools",
   "description": "MCP server for native app testing — screenshot, OCR, click, type, find_text, template matching. macOS, Windows & Android.",
   "license": "MIT",
@@ -53,8 +53,8 @@
     "bin"
   ],
   "optionalDependencies": {
-    "@sh3ll3x3c/native-devtools-mcp-darwin-arm64": "0.5.1",
-    "@sh3ll3x3c/native-devtools-mcp-win32-x64": "0.5.1"
+    "@sh3ll3x3c/native-devtools-mcp-darwin-arm64": "0.7.0",
+    "@sh3ll3x3c/native-devtools-mcp-win32-x64": "0.7.0"
   },
   "engines": {
     "node": ">=18"