native-devtools-mcp 0.5.0 → 0.6.0

package/README.md

@@ -1,22 +1,33 @@
 # native-devtools-mcp
 
-<div align="center">
+`native-devtools-mcp` is a Model Context Protocol (MCP) server for computer use on macOS, Windows, and Android. It gives AI agents and MCP clients direct control over native desktop apps and Android devices through screenshots, OCR, accessibility-based text lookup, input simulation, window management, and ADB.
 
-![Version](https://img.shields.io/npm/v/native-devtools-mcp?style=flat-square)
-![License](https://img.shields.io/npm/l/native-devtools-mcp?style=flat-square)
-![Platform](https://img.shields.io/badge/platform-macOS%20%7C%20Windows%20%7C%20Android-blue?style=flat-square)
-![Downloads](https://img.shields.io/npm/dt/native-devtools-mcp?style=flat-square)
+Use it when browser-only automation is not enough: Electron apps, system dialogs, desktop tools, native app testing, and Android device workflows. It works with [Claude Desktop](https://claude.ai/download), [Claude Code](https://docs.anthropic.com/en/docs/claude-code), [Cursor](https://cursor.com), and other MCP-compatible clients.
 
-**Give your AI agent "eyes" and "hands" for native desktop and mobile applications.**
+Useful for MCP-based computer use, desktop automation, UI automation, native app testing, e2e testing, RPA, screen reading, mouse and keyboard control, and Android device automation.
 
-A Model Context Protocol (MCP) server that provides **Computer Use** capabilities: screenshots, OCR, input simulation, and window management — for **native desktop apps** and **Android devices**, not just browsers.
+```bash
+npx -y native-devtools-mcp
+```
 
-**Works with:** [Claude Desktop](https://claude.ai/download) • [Claude Code](https://docs.anthropic.com/en/docs/claude-code) • [Cursor](https://cursor.com) • Any MCP-compatible client
+**Core capabilities**
+- Screenshots, OCR, and accessibility-first `find_text`
+- `click`, `type_text`, `scroll`, `launch_app`, `quit_app`, and window management
+- `element_at_point` for inspecting accessible UI elements at screen coordinates
+- `load_image` + `find_image` for non-text UI elements such as icons and custom controls
+- Android screenshots, text lookup, input, and app control over ADB
+- Local execution: screenshots and input stay on the machine
 
-[//]: # "Search keywords: MCP, MCP server, Model Context Protocol, computer use, desktop automation, UI automation, native app testing, test automation, e2e testing, RPA, screenshots, OCR, template matching, accessibility, mouse, keyboard, screen reading, macOS, Windows, Android, ADB, mobile testing, Claude, Claude Code, Cursor, AI agent, native-devtools-mcp"
+**For AI agents:** Read [`AGENTS.md`](./AGENTS.md) for tool definitions, workflow patterns, and machine-readable usage guidance.
 
-[Features](#-features) • [Installation](#-installation) • [Getting Started](#-getting-started) • [Security & Trust](#-security--trust) • [For AI Agents](#-for-ai-agents-llms) • [Android](#-android-support)
+![Version](https://img.shields.io/npm/v/native-devtools-mcp?style=flat-square)
+![License](https://img.shields.io/npm/l/native-devtools-mcp?style=flat-square)
+![Platform](https://img.shields.io/badge/platform-macOS%20%7C%20Windows%20%7C%20Android-blue?style=flat-square)
+![Downloads](https://img.shields.io/npm/dt/native-devtools-mcp?style=flat-square)
+
+[Features](#-features) • [Installation](#-installation) • [Getting Started](#-getting-started) • [Recipes](#-recipes-and-examples) • [Security & Trust](#-security--trust) • [For AI Agents](#-for-ai-agents-llms) • [Android](#-android-support)
 
+<div align="center">
 <table>
 <tr>
 <td align="center"><strong>macOS</strong></td>
@@ -27,7 +38,6 @@ A Model Context Protocol (MCP) server that provides **Computer Use** capabilitie
 <td><img src="windows-demo-1.gif" width="450" alt="Windows Demo"></td>
 </tr>
 </table>
-
 </div>
 
 ---
@@ -40,6 +50,7 @@ A Model Context Protocol (MCP) server that provides **Computer Use** capabilitie
 - **🧩 Template Matching:** Find non-text UI elements (icons, shapes) using `load_image` + `find_image`, returning precise click coordinates.
 - **🔒 Local & Private:** 100% local execution. No screenshots or data are ever sent to external servers.
 - **📱 Android Support:** Connect to Android devices over ADB for screenshots, input simulation, UI element search, and app management — all from the same MCP server.
+- **🔍 Hover Tracking:** Track cursor hover transitions across UI elements in real-time. Configurable dwell threshold filters pass-through noise — designed for LLMs observing user navigation patterns.
 - **🔌 Dual-Mode Interaction:**
     1.  **Visual/Native:** Works with *any* app via screenshots & coordinates (Universal).
     2.  **AppDebugKit:** Deep integration for supported apps to inspect the UI tree (DOM-like structure).
@@ -56,6 +67,9 @@ This MCP server is designed to be **highly discoverable and usable** by AI model
 3.  `find_text`: A shortcut to find text on screen and get its coordinates immediately. Uses the platform **accessibility API** (macOS Accessibility / Windows UI Automation) for precise element-level matching, with OCR fallback.
 4.  `element_at_point`: Inspect the accessibility element at given screen coordinates — returns name, role, label, value, bounds, pid, and app_name. Note: privacy-focused Electron apps (e.g. Signal) may restrict their AX tree, returning only a container — use `take_screenshot` with OCR as a fallback.
 5.  `load_image` / `find_image`: Template matching for non-text UI elements (icons, shapes), returning screen coordinates for clicking.
+6.  `start_hover_tracking` / `get_hover_events` / `stop_hover_tracking`: Track cursor hover transitions across UI elements. Configurable dwell threshold filters pass-throughs.
+7.  `start_recording` / `stop_recording`: Record the frontmost app's window at ~5fps as timestamped JPEG frames. Automatically follows app switches.
+8.  `launch_app` / `quit_app`: Launch apps with optional CLI args, or gracefully/forcefully quit them.
 
 ## 📦 Installation
 
@@ -119,6 +133,18 @@ Then restart your MCP client and you're ready to go.
 > { "permissions": { "allow": ["mcp__native-devtools__*"] } }
 > ```
 
+## 📚 Recipes and Examples
+
+- [Recipes and Examples Index](./examples/README.md)
+- [Claude Desktop Setup](./examples/claude-desktop-setup.md)
+- [Claude Code Setup](./examples/claude-code-setup.md)
+- [Cursor Setup](./examples/cursor-setup.md)
+- [End-to-End Desktop Flow](./examples/end-to-end-desktop-flow.md)
+- [Native App Click Flow](./examples/native-app-click-flow.md)
+- [OCR Fallback and Element Inspection](./examples/ocr-fallback-and-element-inspection.md)
+- [Template Matching Flow](./examples/template-matching-flow.md)
+- [Android Quickstart](./examples/android-quickstart.md)
+
 <details>
 <summary><strong>Manual configuration (without setup)</strong></summary>
 
@@ -319,11 +345,15 @@ graph TD
 | | Input | `CGEvent` (CoreGraphics) |
 | | Text Search (`find_text`) | `Accessibility API` (primary), Vision OCR (fallback) |
 | | Element Inspection (`element_at_point`) | `AXUIElementCopyElementAtPosition` + AX tree walk fallback (Accessibility API) |
+| | Hover Tracking (`start_hover_tracking`) | `CGEvent` cursor + Accessibility API polling |
+| | Screen Recording (`start_recording`) | `CGWindowListCreateImage` at configurable fps |
 | | OCR | `VNRecognizeTextRequest` (Vision Framework) |
 | **Windows** | Screenshots | `BitBlt` (GDI) |
 | | Input | `SendInput` (Win32) |
 | | Text Search (`find_text`) | `UI Automation` (primary), WinRT OCR (fallback) |
 | | Element Inspection (`element_at_point`) | `IUIAutomation::ElementFromPoint` (UI Automation) |
+| | Hover Tracking (`start_hover_tracking`) | `GetCursorPos` + UI Automation polling |
+| | Screen Recording (`start_recording`) | `BitBlt` (GDI) at configurable fps |
 | | OCR | `Windows.Media.Ocr` (WinRT) |
 | **Android** | Screenshots | `screencap` / ADB framebuffer |
 | | Input | `adb shell input` (tap, swipe, text, keyevent) |
@@ -364,6 +394,7 @@ Works out of the box on **Windows 10/11**.
 *   `find_text` uses **UI Automation (UIA)** as the primary search mechanism, querying the accessibility tree for element names. This is the same accessibility-first approach used on macOS (with the Accessibility API). Falls back to OCR automatically when UIA finds no matches.
 *   OCR uses the built-in Windows Media OCR engine (offline).
 *   **Note:** Cannot interact with "Run as Administrator" windows unless the MCP server itself is also running as Administrator.
+*   **Screen Recording Performance:** Screen recording uses GDI/BitBlt at configurable fps (default 5). For higher fps requirements or game capture scenarios, DXGI Desktop Duplication API would provide hardware-accelerated capture — this is a planned future upgrade.
 
 ## 📜 License
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "native-devtools-mcp",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "mcpName": "io.github.sh3ll3x3c/native-devtools",
   "description": "MCP server for native app testing — screenshot, OCR, click, type, find_text, template matching. macOS, Windows & Android.",
   "license": "MIT",
@@ -53,8 +53,8 @@
     "bin"
   ],
   "optionalDependencies": {
-    "@sh3ll3x3c/native-devtools-mcp-darwin-arm64": "0.5.0",
-    "@sh3ll3x3c/native-devtools-mcp-win32-x64": "0.5.0"
+    "@sh3ll3x3c/native-devtools-mcp-darwin-arm64": "0.6.0",
+    "@sh3ll3x3c/native-devtools-mcp-win32-x64": "0.6.0"
   },
   "engines": {
     "node": ">=18"