@harusame64/desktop-touch-mcp 0.15.7 → 1.0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (5) hide show
  1. package/LICENSE +21 -21
  2. package/README.ja.md +612 -516
  3. package/README.md +723 -685
  4. package/bin/launcher.js +83 -7
  5. package/package.json +79 -78
package/README.md CHANGED
@@ -1,694 +1,732 @@
1
- # desktop-touch-mcp
2
-
3
- [![desktop-touch-mcp MCP server](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp/badges/card.svg)](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp)
4
-
5
- [日本語](README.ja.md)
6
-
7
- > **Stop pasting screenshots. Let Claude see and control your desktop directly.**
8
-
9
- An MCP server that gives Claude eyes and hands on Windows — 57 tools covering screenshots, mouse, keyboard, Windows UI Automation, Chrome DevTools Protocol, clipboard, desktop notifications, SmartScroll, and a Reactive Perception Graph for safe multi-step automation, designed from the ground up for LLM efficiency.
10
-
11
- > *v0.15: **82× average speedup** via Rust native engine — UIA focus queries in 2 ms, SSE2-accelerated image diffing at 13–15× native speed. Zero-config: the engine auto-loads when present, with transparent PowerShell fallback.*
1
+ # desktop-touch-mcp
2
+
3
+ [![desktop-touch-mcp MCP server](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp/badges/card.svg)](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp)
4
+
5
+ [日本語](README.ja.md)
6
+
7
+ > **Beyond Coordinate Roulette: LLM-native Windows automation with a Semantic World-Graph and Auto-Perception.**
8
+
9
+ An MCP server that gives Claude eyes and hands on Windows. It moves beyond pixel-guessing by grounding all interactions in a **Semantic World-Graph** (`desktop_discover`) and verifying every action with **Auto-Perception** guards. Optimized into 28 high-signal tools covering screenshots, background input (WM_CHAR), UIA, Chrome CDP, terminal, and token-efficient P-frame diffing.
10
+
11
+ > *v0.15: **82× average speedup** via Rust native engine — UIA focus queries in 2 ms, SSE2-accelerated image diffing at 13–15× native speed. Zero-config: the engine auto-loads when present, with transparent PowerShell fallback.*
12
12
  > *v0.15.5: **Pinned release verification** — the npm launcher now fetches only the matching GitHub Release tag and verifies the Windows runtime zip before extraction.*
13
-
14
- ---
15
-
16
- ## Features
17
-
18
- - **⚡ High-performance Rust Native Core** — The UIA bridge and image-diff engine are written in Rust (`napi-rs` + `windows-rs`) and loaded as a native `.node` addon. Direct COM calls from a dedicated MTA thread eliminate PowerShell process spawning — `getFocusedElement` completes in **2 ms** (160× faster), and `getUiElements` returns full trees in **~100 ms** with a batch BFS algorithm that minimizes cross-process RPC. Image-diff operations use **SSE2 SIMD** for 13–15× throughput. When the native engine is unavailable, every function transparently falls back to PowerShell — zero config required.
19
- - **🎯 Set-of-Marks (SoM) visual fallback** — Games, RDP sessions, and non-accessible Electron apps return clickable elements even when UIA is completely blind. `screenshot(detail="text")` automatically detects UIA sparsity and activates a Hybrid Non-CDP pipeline: Rust-powered grayscale + bilinear upscale → Windows OCR → clustering → red bounding-box annotation with numbered badges (`[1]`, `[2]`…). Two parallel representations returned: a visual PNG for spatial orientation and a semantic `elements[]` list with `clickAt` coords — no CDP required.
20
- - **LLM-native design** — Built around how LLMs think, not how humans click. `run_macro` batches multiple operations into a single API call; `diffMode` sends only the windows that changed since the last frame. Minimal tokens, minimal round-trips.
21
- - **Reactive Perception Graph** — Register a `lensId` for a window or browser tab, pass it to action tools, and get guard-checked `post.perception` feedback after each action. It reduces repeated `screenshot` / `get_context` calls and prevents wrong-window typing or stale-coordinate clicks.
22
- - **Full CJK support** — Uses Win32 `GetWindowTextW` for window titles, avoiding nut-js garbling. IME bypass input supported for Japanese/Chinese/Korean environments.
23
- - **3-tier token reduction** — `detail="image"` (~443 tok) / `detail="text"` (~100–300 tok) / `diffMode=true` (~160 tok). Send pixels only when you actually need to see them.
24
- - **1:1 coordinate mode** — `dotByDot=true` captures at native resolution (WebP). Image pixel = screen coordinate — no scale math needed. With `origin`+`scale` passed to `mouse_click`, the server converts coords for you — eliminating off-by-one / scale bugs.
25
- - **Browser capture data reduction** — `grayscale=true` (~50% size), `dotByDotMaxDimension=1280` (auto-scaled with coord preservation), and `windowTitle + region` sub-crops help exclude browser chrome and other irrelevant pixels. Typical reduction for heavy captures: 50–70%.
26
- - **Chromium smart fallback** — `detail="text"` on Chrome/Edge/Brave auto-skips UIA (prohibitively slow there) and runs Windows OCR. `hints.chromiumGuard` + `hints.ocrFallbackFired` flag the path taken.
27
- - **UIA element extraction** — `detail="text"` returns button names and `clickAt` coords as JSON. Claude can click the right element without ever looking at a screenshot.
28
- - **Auto-dock CLI** — `dock_window` snaps any window to a screen corner with always-on-top. Set `DESKTOP_TOUCH_DOCK_TITLE='@parent'` to auto-dock the terminal hosting Claude on MCP startup — the process-tree walker finds the right window regardless of title.
29
- - **Emergency stop (Failsafe)** — Move the mouse to the **top-left corner (within 10px of 0,0)** to immediately terminate the MCP server.
30
-
31
- ---
32
-
33
- ## Requirements
34
-
35
- | | |
36
- |---|---|
37
- | OS | Windows 10 / 11 (64-bit) |
38
- | Node.js | v20+ recommended (tested on v22+) |
39
- | PowerShell | 5.1+ (bundled with Windows) — used only as fallback when the Rust native engine is unavailable |
40
- | Claude CLI | `claude` command must be available |
41
-
42
- > **Note:** nut-js native bindings require the Visual C++ Redistributable.
43
- > Download from [Microsoft](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist) if not already installed.
44
-
45
- ---
46
-
47
- ## Installation
48
-
49
- ```bash
50
- npx -y @harusame64/desktop-touch-mcp
51
- ```
52
-
13
+
14
+ ---
15
+
16
+ ## Features
17
+
18
+ - **⚡ High-performance Rust Native Core** — The UIA bridge and image-diff engine are written in Rust (`napi-rs` + `windows-rs`) and loaded as a native `.node` addon. Direct COM calls from a dedicated MTA thread eliminate PowerShell process spawning — `getFocusedElement` completes in **2 ms** (160× faster), and `getUiElements` returns full trees in **~100 ms** with a batch BFS algorithm that minimizes cross-process RPC. Image-diff operations use **SSE2 SIMD** for 13–15× throughput. When the native engine is unavailable, every function transparently falls back to PowerShell — zero config required.
19
+ - **🎯 Set-of-Marks (SoM) visual fallback** — Games, RDP sessions, and non-accessible Electron apps return clickable elements even when UIA is completely blind. `screenshot(detail="text")` automatically detects UIA sparsity and activates a Hybrid Non-CDP pipeline: Rust-powered grayscale + bilinear upscale → Windows OCR → clustering → red bounding-box annotation with numbered badges (`[1]`, `[2]`…). Two parallel representations returned: a visual PNG for spatial orientation and a semantic `elements[]` list with `clickAt` coords — no CDP required.
20
+ - **LLM-native design** — Built around how LLMs think, not how humans click. `run_macro` batches multiple operations into a single API call; `diffMode` sends only the windows that changed since the last frame. Minimal tokens, minimal round-trips.
21
+ - **Reactive Perception Graph** — Register a `lensId` for a window or browser tab, pass it to action tools, and get guard-checked `post.perception` feedback after each action. It reduces repeated `screenshot` / `desktop_state` calls and prevents wrong-window typing or stale-coordinate clicks.
22
+ - **Full CJK support** — Uses Win32 `GetWindowTextW` for window titles, avoiding nut-js garbling. IME bypass input supported for Japanese/Chinese/Korean environments.
23
+ - **3-tier token reduction** — `detail="image"` (~443 tok) / `detail="text"` (~100–300 tok) / `diffMode=true` (~160 tok). Send pixels only when you actually need to see them.
24
+ - **1:1 coordinate mode** — `dotByDot=true` captures at native resolution (WebP). Image pixel = screen coordinate — no scale math needed. With `origin`+`scale` passed to `mouse_click`, the server converts coords for you — eliminating off-by-one / scale bugs.
25
+ - **Browser capture data reduction** — `grayscale=true` (~50% size), `dotByDotMaxDimension=1280` (auto-scaled with coord preservation), and `windowTitle + region` sub-crops help exclude browser chrome and other irrelevant pixels. Typical reduction for heavy captures: 50–70%.
26
+ - **Chromium smart fallback** — `detail="text"` on Chrome/Edge/Brave auto-skips UIA (prohibitively slow there) and runs Windows OCR. `hints.chromiumGuard` + `hints.ocrFallbackFired` flag the path taken.
27
+ - **UIA element extraction** — `detail="text"` returns button names and `clickAt` coords as JSON. Claude can click the right element without ever looking at a screenshot.
28
+ - **Auto-dock CLI** — `window_dock(action='dock')` snaps any window to a screen corner with always-on-top. Set `DESKTOP_TOUCH_DOCK_TITLE='@parent'` to auto-dock the terminal hosting Claude on MCP startup — the process-tree walker finds the right window regardless of title.
29
+ - **Emergency stop (Failsafe)** — Move the mouse to the **top-left corner (within 10px of 0,0)** to immediately terminate the MCP server.
30
+
31
+ ---
32
+
33
+ ## Requirements
34
+
35
+ | | |
36
+ |---|---|
37
+ | OS | Windows 10 / 11 (64-bit) |
38
+ | Node.js | v20+ recommended (tested on v22+) |
39
+ | PowerShell | 5.1+ (bundled with Windows) — used only as fallback when the Rust native engine is unavailable |
40
+ | Claude CLI | `claude` command must be available |
41
+
42
+ > **Note:** nut-js native bindings require the Visual C++ Redistributable.
43
+ > Download from [Microsoft](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist) if not already installed.
44
+
45
+ ---
46
+
47
+ ## Installation
48
+
49
+ ```bash
50
+ npx -y @harusame64/desktop-touch-mcp
51
+ ```
52
+
53
53
  The npm launcher resolves runtime strictly by npm package version. For package `X.Y.Z`, it fetches only GitHub Release tag `vX.Y.Z`, downloads `desktop-touch-mcp-windows.zip`, verifies its SHA256 digest, and only then expands it under `%USERPROFILE%\.desktop-touch-mcp`. Verified cached releases are reused on later runs.
54
54
 
55
55
  Set `DESKTOP_TOUCH_MCP_HOME` to override the cache root directory.
56
-
57
- ### Register with Claude CLI
58
-
59
- Add to `~/.claude.json` under `mcpServers`:
60
-
61
- ```json
62
- {
63
- "mcpServers": {
64
- "desktop-touch": {
65
- "type": "stdio",
66
- "command": "npx",
67
- "args": ["-y", "@harusame64/desktop-touch-mcp"]
68
- }
69
- }
70
- }
71
- ```
72
-
73
- **No system prompt needed.** The command reference is automatically injected into Claude via the MCP `initialize` response's `instructions` field.
74
-
75
- ### Register with other clients (HTTP mode)
76
-
77
- Clients that require an HTTP endpoint (GPT Desktop, VS Code Copilot, Cursor, etc.) can use the built-in Streamable HTTP transport:
78
-
79
- ```bash
80
- npx -y @harusame64/desktop-touch-mcp --http
81
- # or with a custom port:
82
- npx -y @harusame64/desktop-touch-mcp --http --port 8080
83
- ```
84
-
85
- The server starts at `http://127.0.0.1:23847/mcp` (localhost only). Register the URL in your MCP client settings. A health check is available at `http://127.0.0.1:<port>/health`.
86
-
87
- In HTTP mode the system tray icon shows the active URL and provides quick-copy and open-in-browser shortcuts.
88
-
89
- ### Development install
90
-
91
- ```bash
92
- git clone https://github.com/Harusame64/desktop-touch-mcp.git
93
- cd desktop-touch-mcp
94
- npm install
95
- ```
96
-
56
+
57
+ ### Register with Claude CLI
58
+
59
+ Add to `~/.claude.json` under `mcpServers`:
60
+
61
+ ```json
62
+ {
63
+ "mcpServers": {
64
+ "desktop-touch": {
65
+ "type": "stdio",
66
+ "command": "npx",
67
+ "args": ["-y", "@harusame64/desktop-touch-mcp"]
68
+ }
69
+ }
70
+ }
71
+ ```
72
+
73
+ **No system prompt needed.** The command reference is automatically injected into Claude via the MCP `initialize` response's `instructions` field.
74
+
75
+ ### Register with other clients (HTTP mode)
76
+
77
+ Clients that require an HTTP endpoint (GPT Desktop, VS Code Copilot, Cursor, etc.) can use the built-in Streamable HTTP transport:
78
+
79
+ ```bash
80
+ npx -y @harusame64/desktop-touch-mcp --http
81
+ # or with a custom port:
82
+ npx -y @harusame64/desktop-touch-mcp --http --port 8080
83
+ ```
84
+
85
+ The server starts at `http://127.0.0.1:23847/mcp` (localhost only). Register the URL in your MCP client settings. A health check is available at `http://127.0.0.1:<port>/health`.
86
+
87
+ In HTTP mode the system tray icon shows the active URL and provides quick-copy and open-in-browser shortcuts.
88
+
89
+ ### Development install
90
+
91
+ ```bash
92
+ git clone https://github.com/Harusame64/desktop-touch-mcp.git
93
+ cd desktop-touch-mcp
94
+ npm install
95
+ ```
96
+
97
97
  Build after install:
98
98
 
99
99
  ```bash
100
100
  npm run build
101
101
  ```
102
-
103
- For a local checkout, register the built server directly:
104
-
105
- ```json
106
- {
107
- "mcpServers": {
108
- "desktop-touch": {
109
- "type": "stdio",
110
- "command": "node",
111
- "args": ["D:/path/to/desktop-touch-mcp/dist/index.js"]
112
- }
113
- }
114
- }
115
- ```
116
-
117
- > **Note:** Replace `D:/path/to/desktop-touch-mcp` with the actual path where you cloned this repository.
118
-
119
- ---
120
-
121
- ## Tools (57 total)
122
-
123
- > 📖 **Full command reference**: [`docs/system-overview.md`](docs/system-overview.md) — every tool's parameters, response shape, coordinate math, layer-buffer strategy, and engineering notes in one place.
124
-
125
-
126
- ### Screenshot (5)
127
- | Tool | Description |
128
- |---|---|
129
- | `screenshot` | Main capture. Supports `detail`, `dotByDot`, `dotByDotMaxDimension`, `grayscale`, `region` sub-crop, `diffMode`. `detail="text"` auto-activates the SoM pipeline when UIA is blind (games, RDP, custom Electron) |
130
- | `screenshot_background` | Capture a background window without focusing it (PrintWindow API) |
131
- | `screenshot_ocr` | Windows.Media.Ocr on a window; returns word-level text + screen clickAt coords |
132
- | `get_screen_info` | Monitor layout, DPI, cursor position |
133
- | `scroll_capture` | Full-page stitch by scrolling (MAE overlap detection + 10% fallback) |
134
-
135
- ### Window management (4)
136
- | Tool | Description |
137
- |---|---|
138
- | `get_windows` | List all windows in Z-order |
139
- | `get_active_window` | Info about the focused window |
140
- | `focus_window` | Bring a window to foreground by partial title match |
141
- | `dock_window` | Snap a window to a screen corner at a small size + always-on-top (for keeping CLI visible) |
142
-
143
- ### Mouse (5)
144
- | Tool | Description |
145
- |---|---|
146
- | `mouse_move` / `mouse_click` / `mouse_drag` | Move, click, drag. `doubleClick` / `tripleClick` (line-select). Accept `speed` and `homing` parameters |
147
- | `scroll` | Scroll in any direction. Accepts `speed` and `homing` parameters |
148
- | `get_cursor_position` | Current cursor coordinates |
149
-
150
- ### Keyboard (2)
151
- | Tool | Description |
152
- |---|---|
153
- | `keyboard_type` | Type text. `use_clipboard=true` bypasses IME (required for em-dash / smart quotes). `replaceAll=true` sends Ctrl+A before typing. Non-ASCII symbols trigger clipboard mode automatically (opt-out: `forceKeystrokes=true`) |
154
- | `keyboard_press` | Key combos (`ctrl+c`, `alt+f4`, etc.) |
155
-
156
- ### UI Automation (4)
157
- | Tool | Description |
158
- |---|---|
159
- | `get_ui_elements` | Full UIA element tree for a window |
160
- | `click_element` | Click a button by name or automationId — no coordinates needed |
161
- | `set_element_value` | Write directly to a text field |
162
- | `scope_element` | High-res zoom crop of an element + its child tree |
163
-
164
- ### Browser CDP (12)
165
- | Tool | Description |
166
- |---|---|
167
- | `browser_launch` | Launch Chrome/Edge/Brave with `--remote-debugging-port` and wait for the CDP endpoint (idempotent) |
168
- | `browser_connect` | Connect to Chrome/Edge via CDP; lists open tabs with `active:true/false` |
169
- | `browser_find_element` | CSS selector → exact physical screen coords |
170
- | `browser_click_element` | Find DOM element + click in one step |
171
- | `browser_eval` | Evaluate JS expression in the browser tab |
172
- | `browser_fill_input` | Fill React/Vue/Svelte controlled inputs via CDP — works where `browser_eval` value assignment doesn't update framework state |
173
- | `browser_get_dom` | Get outerHTML of element or `document.body` |
174
- | `browser_get_interactive` | Enumerate links / buttons / inputs + **ARIA toggles** with `state.{checked,pressed,selected,expanded}`; each element includes `viewportPosition` |
175
- | `browser_get_app_state` | **SPA state extractor** — one CDP call that scans `__NEXT_DATA__`, `__NUXT_DATA__`, `__REMIX_CONTEXT__`, `__APOLLO_STATE__`, GitHub `react-app` embeddedData, JSON-LD, `window.__INITIAL_STATE__` |
176
- | `browser_search` | Grep DOM by text / regex / role / ariaLabel / selector with confidence ranking |
177
- | `browser_navigate` | Navigate via CDP `Page.navigate`; `waitForLoad:true` (default) returns once `readyState==='complete'` |
178
- | `browser_disconnect` | Close cached CDP WebSocket sessions |
179
-
180
- All `browser_*` tools that touch the DOM accept `includeContext:false` to omit the trailing `activeTab:` / `readyState:` lines (saves ~150 tok/call on chained invocations). Within a 500 ms window, consecutive calls reuse one tab-context fetch automatically.
181
-
182
- ### Workspace (2)
183
- | Tool | Description |
184
- |---|---|
185
- | `workspace_snapshot` | All windows: thumbnails + UI summaries in one call |
186
- | `workspace_launch` | Launch an app and auto-detect the new window |
187
-
188
- ### Context / Wait / History (8)
189
- | Tool | Description |
190
- |---|---|
191
- | `get_context` | Lightweight snapshot of focused window, element, cursor, and page state |
192
- | `get_history` | Retrieve recent tool invocation history |
193
- | `get_document_state` | Chrome page state (URL/title/readyState/scroll) via CDP |
194
- | `engine_status` | Returns which backend is active: `uia` (native Rust or powershell) and `imageDiff` (native Rust SSE2 or typescript). Diagnostic — call once per session when troubleshooting performance |
195
- | `wait_until` | Server-side wait for window/focus/terminal/browser DOM state changes |
196
- | `events_subscribe` / `events_poll` / `events_unsubscribe` / `events_list` | Subscribe to and poll window appearance/disappearance/focus events |
197
-
198
- ### Terminal (2)
199
- | Tool | Description |
200
- |---|---|
201
- | `terminal_read` | Read text from Windows Terminal / PowerShell / cmd / WSL via UIA/OCR. Supports `sinceMarker` for diff reads |
202
- | `terminal_send` | Send commands to a terminal. Uses clipboard paste by default for IME safety |
203
-
204
- ### Pin / Macro (3)
205
- | Tool | Description |
206
- |---|---|
207
- | `pin_window` / `unpin_window` | Always-on-top toggle |
208
- | `run_macro` | Execute up to 50 steps sequentially in one MCP call |
209
-
210
- ### Clipboard (2)
211
- | Tool | Description |
212
- |---|---|
213
- | `clipboard_read` | Read the current Windows clipboard text (non-text payloads return empty string) |
214
- | `clipboard_write` | Write text to the Windows clipboard; full Unicode / emoji / CJK support |
215
-
216
- ### Notification (1)
217
- | Tool | Description |
218
- |---|---|
219
- | `notification_show` | Show a Windows system tray balloon notification — useful to alert the user when a long-running task finishes |
220
-
221
- ### Scroll (2)
222
- | Tool | Description |
223
- |---|---|
224
- | `scroll_to_element` | Scroll a named element into the viewport without computing scroll amounts. Chrome path: `selector` + `block` alignment. Native path: `name` + `windowTitle` via UIA ScrollItemPattern |
225
- | `smart_scroll` | **SmartScroll** — unified scroll dispatcher: CDP → UIA → image binary-search fallback. Handles nested containers, virtualised lists (TanStack/React Virtualized), sticky-header occlusion, and image-only environments. Returns `pageRatio`, `ancestors[]`, and hash-verified `scrolled` |
226
-
227
- ---
228
-
229
- ## Browser CDP automation
230
-
231
- For web automation, connect Chrome or Edge with the remote debugging port enabled — no Selenium or Playwright needed.
232
-
233
- ```bash
234
- # Launch Chrome in CDP mode
235
- chrome.exe --remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp
236
- ```
237
-
238
- ```
239
- browser_launch() → launch Chrome/Edge/Brave in debug mode (idempotent)
240
- browser_connect() → list open tabs + get tabIds
241
- browser_find_element("#submit") → CSS selector → physical screen coords
242
- browser_click_element("#submit") → find + click in one step (auto-focuses browser)
243
- browser_eval("document.title") → evaluate JS, returns result
244
- browser_fill_input("#email", "user@example.com") → fill React/Vue/Svelte controlled input (state-safe)
245
- browser_get_dom("#main", maxLength=5000)→ outerHTML, truncated to maxLength chars
246
- browser_get_interactive() → links/buttons/inputs + ARIA toggles + viewportPosition per element
247
- browser_get_app_state() → one-shot SPA state (Next/Nuxt/Remix/Apollo/GitHub react-app/Redux SSR)
248
- browser_search(by="text", pattern="...")→ grep DOM with confidence ranking
249
- browser_navigate("https://example.com") → navigate via CDP (no address bar interaction)
250
- browser_disconnect() → clean up WebSocket sessions
251
- ```
252
-
253
- For chained calls in the same tab, pass `includeContext:false` to omit the activeTab/readyState annotation (~150 tok/call saved). Boolean / object params accept the LLM-friendly string spellings (`"true"`, `"{}"`).
254
-
255
- Coordinates returned by `browser_find_element` account for the browser chrome (tab strip + address bar height) and `devicePixelRatio`, so they can be passed directly to `mouse_click` without any scaling.
256
-
257
- **Recommended web workflow:**
258
- ```
259
- browser_connect() → browser_get_dom() → browser_find_element(selector) → browser_click_element(selector)
260
- ```
261
-
262
- ---
263
-
264
- ## Auto-dock CLI on startup
265
-
266
- Keep Claude CLI visible while operating other apps full-screen. Set env vars in your MCP config and the docked window auto-snaps into place every MCP startup.
267
-
268
- ```json
269
- {
270
- "mcpServers": {
271
- "desktop-touch": {
272
- "type": "stdio",
273
- "command": "npx",
274
- "args": ["-y", "@harusame64/desktop-touch-mcp"],
275
- "env": {
276
- "DESKTOP_TOUCH_DOCK_TITLE": "@parent",
277
- "DESKTOP_TOUCH_DOCK_CORNER": "bottom-right",
278
- "DESKTOP_TOUCH_DOCK_WIDTH": "480",
279
- "DESKTOP_TOUCH_DOCK_HEIGHT": "360",
280
- "DESKTOP_TOUCH_DOCK_PIN": "true"
281
- }
282
- }
283
- }
284
- }
285
- ```
286
-
287
- | Env var | Default | Notes |
288
- |---|---|---|
289
- | `DESKTOP_TOUCH_DOCK_TITLE` | *(unset = off)* | `@parent` walks the MCP process tree to find the hosting terminal — immune to title / branch / project changes. Or use a literal substring. |
290
- | `DESKTOP_TOUCH_DOCK_CORNER` | `bottom-right` | `top-left` / `top-right` / `bottom-left` / `bottom-right` |
291
- | `DESKTOP_TOUCH_DOCK_WIDTH` / `HEIGHT` | `480` / `360` | px (`"480"`) or ratio of work area (`"25%"`) — 4K/8K auto-adapts |
292
- | `DESKTOP_TOUCH_DOCK_PIN` | `true` | Always-on-top toggle |
293
- | `DESKTOP_TOUCH_DOCK_MONITOR` | primary | Monitor id from `get_screen_info` |
294
- | `DESKTOP_TOUCH_DOCK_SCALE_DPI` | `false` | If true, multiply px values by `dpi / 96` (opt-in per-monitor scaling) |
295
- | `DESKTOP_TOUCH_DOCK_MARGIN` | `8` | Screen-edge padding (px) |
296
- | `DESKTOP_TOUCH_DOCK_TIMEOUT_MS` | `5000` | Max wait for the target window to appear |
297
-
298
- > **Input routing gotcha:** when a pinned window is active (e.g. Claude CLI), `keyboard_type` / `keyboard_press` send keys to it, **not** the app you wanted to type into. Always call `focus_window(title=...)` before keyboard operations, then verify `isActive=true` via `screenshot(detail='meta')`.
299
-
300
- ### Reactive Perception Graph (4)
301
-
302
- | Tool | Description |
303
- |---|---|
304
- | `perception_register` | Register a live perception lens on a window or browser tab. Returns a `lensId` to pass to action tools |
305
- | `perception_read` | Force-refresh the lens and return a full perception envelope when attention is dirty/stale/blocked |
306
- | `perception_forget` | Release a lens when the workflow ends or the target was replaced |
307
- | `perception_list` | List active lenses so Claude can reuse or clean up existing tracking |
308
-
309
- Reactive Perception Graph is desktop-touch's low-cost situational awareness layer. It keeps the target identity, focus, rect, readiness, and guard state alive across actions so Claude does not need to re-check everything with a screenshot after every small move.
310
-
311
- ```
312
- # Register a lens on the target window or browser tab
313
- perception_register({name:"editor", target:{kind:"window", match:{titleIncludes:"Notepad"}}})
314
- → {lensId:"perc-1", ...}
315
-
316
- # Pass lensId to action tools. Guards run before the action;
317
- # compact feedback arrives in post.perception after the action.
318
- keyboard_type({text:"hello", windowTitle:"Notepad", lensId:"perc-1"})
319
- → post.perception: {attention:"ok", guards:{...}, latest:{target:{title, rect, foreground}}}
320
-
321
- # If the app restarts or focus moves away, guards fail closed before unsafe input:
322
- keyboard_type({text:"x", lensId:"perc-1"})
323
- → {ok:false, code:"GuardFailed", suggest:["Re-register lens for the new process instance"]}
324
- ```
325
-
326
- `lensId` is opt-in on all action tools (`keyboard_type`, `keyboard_press`, `mouse_click`, `mouse_drag`, `click_element`, `set_element_value`, `browser_click_element`, `browser_navigate`, `browser_eval`). Omitting `lensId` preserves existing behavior exactly.
327
-
328
- ---
329
-
330
- ## Mouse homing correction
331
-
332
- When Claude calls `screenshot(detail='text')` to read coordinates and then `mouse_click` seconds later, the target window may have moved. The homing system corrects this automatically.
333
-
334
- | Tier | How to enable | Latency | What it does |
335
- |------|--------------|---------|--------------|
336
- | 1 | Always-on (if cache exists) | <1ms | Applies (dx, dy) offset when window moved |
337
- | 2 | Pass `windowTitle` hint | ~100ms | Auto-focuses window if it went behind another |
338
- | 3 | Pass `elementName`/`elementId` + `windowTitle` | 1–3s | UIA re-query for fresh coords on resize |
339
-
340
- ```
341
- # Tier 1 only (automatic)
342
- mouse_click(x=500, y=300)
343
-
344
- # Tier 1 + 2: also bring window to front if hidden
345
- mouse_click(x=500, y=300, windowTitle="Notepad")
346
-
347
- # Tier 1 + 2 + 3: also re-query UIA if window resized
348
- mouse_click(x=500, y=300, windowTitle="Notepad", elementName="Save")
349
-
350
- # Traction control OFF — no correction
351
- mouse_click(x=500, y=300, homing=false)
352
- ```
353
-
354
- The `homing` parameter is available on `mouse_click`, `mouse_move`, `mouse_drag`, and `scroll`. The cache is updated automatically on every `screenshot()`, `get_windows()`, `focus_window()`, and `workspace_snapshot()` call.
355
-
356
- ### `mouse_click` image-local coords (origin + scale)
357
-
358
- When you take a `dotByDot` screenshot with `dotByDotMaxDimension`, the response prints the `origin` and `scale` values. Instead of computing screen coords manually, copy them into `mouse_click`:
359
-
360
- ```
361
- # Screenshot response:
362
- # origin: (0, 120) | scale: 0.6667
363
- # To click image pixel (ix, iy): mouse_click(x=ix, y=iy, origin={x:0, y:120}, scale=0.6667)
364
-
365
- mouse_click(x=640, y=300, origin={x:0, y:120}, scale=0.6667, windowTitle="Chrome")
366
- # Server converts: screen = (0 + 640/0.6667, 120 + 300/0.6667) = (960, 570)
367
- ```
368
-
369
- This eliminates a whole class of off-by-one and scale bugs. Without origin/scale, `x`/`y` remain absolute screen pixels (unchanged behavior).
370
-
371
- ---
372
-
373
- ## `screenshot` key parameters
374
-
375
- ```
376
- detail="image" — PNG/WebP pixels (default)
377
- detail="text" — UIA element JSON + clickAt coords (no image, ~100–300 tok)
378
- detail="meta" — Title + region only (cheapest, ~20 tok/window)
379
- dotByDot=true — 1:1 WebP; image_px + origin = screen_px
380
- dotByDotMaxDimension=N — cap longest edge (response includes scale for coord math)
381
- grayscale=true — ~50% smaller for text-heavy captures (code/AWS console)
382
- region={x,y,w,h} — with windowTitle: window-local coords (exclude browser chrome)
383
- without: virtual screen coords
384
- diffMode=true — I-frame first call, P-frame (changed windows only) after (~160 tok)
385
- ocrFallback="auto" — detail='text' auto-fires Windows OCR on uiaSparse or empty
386
- ```
387
-
388
- **Recommended Chrome combo** (50–70% data reduction):
389
- ```
390
- screenshot(windowTitle="Chrome",
391
- dotByDot=true, dotByDotMaxDimension=1280, grayscale=true,
392
- region={x:0, y:120, width:1920, height:900}) # skip browser chrome
393
- ```
394
-
395
- **Recommended workflow:**
396
- ```
397
- workspace_snapshot() → full orientation (resets diff buffer)
398
- screenshot(detail="text", windowTitle=X) → get actionable[].clickAt coords
399
- mouse_click(x, y) → click directly, no math needed
400
- screenshot(diffMode=true) → check only what changed (~160 tok)
401
- ```
402
-
403
- ---
404
-
405
- ## Security
406
-
407
- ### Emergency stop (Failsafe)
408
-
409
- **Move the mouse to the top-left corner of the screen (within 10px of 0,0) to immediately terminate the MCP server.**
410
-
411
- - **Per-tool check**: `checkFailsafe()` runs before every tool handler
412
- - **Background monitor**: 500ms polling as a backup for long-running operations
413
- - Trigger radius: 10px
414
-
415
- ### Blocked operations
416
-
417
- **`workspace_launch` blocklist:**
418
- `cmd.exe`, `powershell.exe`, `pwsh.exe`, `wscript.exe`, `cscript.exe`, `mshta.exe`, `regsvr32.exe`, `rundll32.exe`, `msiexec.exe`, `bash.exe`, `wsl.exe` are blocked.
419
- Script extensions (`.bat`, `.ps1`, `.vbs`, etc.) are rejected. Arguments containing `;`, `&`, `|`, `` ` ``, `$(`, `${` are also rejected.
420
-
421
- **`keyboard_press` blocklist:**
422
- `Win+R` (Run dialog), `Win+X` (admin menu), `Win+S` (search), `Win+L` (lock screen) are blocked.
423
-
424
- ### PowerShell injection protection
425
-
426
- All `-like` patterns in the UIA bridge PowerShell fallback path are sanitized with `escapeLike()`, which escapes wildcard characters (`*`, `?`, `[`, `]`) before they reach PowerShell. When the Rust native engine is active, PowerShell is not invoked for UIA operations.
427
-
428
- ### Allowlist for `workspace_launch`
429
-
430
- Shell interpreters are blocked by default. To allow specific executables, create an allowlist file:
431
-
432
- **File locations (searched in order):**
433
- 1. Path in `DESKTOP_TOUCH_ALLOWLIST` environment variable
434
- 2. `~/.claude/desktop-touch-allowlist.json`
435
- 3. `desktop-touch-allowlist.json` in the server's working directory
436
-
437
- **Format:**
438
- ```json
439
- {
440
- "allowedExecutables": [
441
- "pwsh.exe",
442
- "C:\\Tools\\myapp.exe"
443
- ]
444
- }
445
- ```
446
-
447
- Changes take effect immediately — no restart needed.
448
-
449
- ---
450
-
451
- ## Mouse movement speed
452
-
453
- All mouse tools (`mouse_move`, `mouse_click`, `mouse_drag`, `scroll`) accept an optional `speed` parameter:
454
-
455
- | Value | Behavior |
456
- |---|---|
457
- | Omitted | Uses the configured default (see below) |
458
- | `0` | Instant teleport — `setPosition()`, no animation |
459
- | `1–N` | Animated movement at N px/sec |
460
-
461
- **Default speed** is 1500 px/sec. Change it permanently via the `DESKTOP_TOUCH_MOUSE_SPEED` environment variable:
462
-
463
- ```json
464
- {
465
- "mcpServers": {
466
- "desktop-touch": {
467
- "type": "stdio",
468
- "command": "npx",
469
- "args": ["-y", "@harusame64/desktop-touch-mcp"],
470
- "env": {
471
- "DESKTOP_TOUCH_MOUSE_SPEED": "3000"
472
- }
473
- }
474
- }
475
- }
476
- ```
477
-
478
- Common values: `0` = teleport, `1500` = default gentle, `3000` = fast, `5000` = very fast.
479
-
480
- ---
481
-
482
- ## Force-Focus (AttachThreadInput)
483
-
484
- Windows foreground-stealing protection can prevent `SetForegroundWindow` from succeeding when another window (such as a pinned Claude CLI) is in the foreground. This causes subsequent keystrokes or clicks to land in the wrong window — a silent failure.
485
-
486
- `mouse_click`, `keyboard_type`, `keyboard_press`, and `terminal_send` all accept a `forceFocus` parameter that bypasses this protection using `AttachThreadInput`:
487
-
488
- ```json
489
- {
490
- "name": "mouse_click",
491
- "arguments": {
492
- "x": 500,
493
- "y": 300,
494
- "windowTitle": "Google Chrome",
495
- "forceFocus": true
496
- }
497
- }
498
- ```
499
-
500
- If the force attempt is refused despite `AttachThreadInput`, the response includes `hints.warnings: ["ForceFocusRefused"]`.
501
-
502
- **Global default via environment variable:**
503
-
504
- ```json
505
- {
506
- "mcpServers": {
507
- "desktop-touch": {
508
- "env": {
509
- "DESKTOP_TOUCH_FORCE_FOCUS": "1"
510
- }
511
- }
512
- }
513
- }
514
- ```
515
-
516
- Setting `DESKTOP_TOUCH_FORCE_FOCUS=1` makes `forceFocus: true` the default for all four tools without changing each call.
517
-
518
- **Known tradeoffs:**
519
-
520
- - During the ~10ms `AttachThreadInput` window, key state and mouse capture are shared between the two threads. In rapid macro sequences this can cause a race condition (rare in practice).
521
- - Disable `forceFocus` (or unset the env var) when the user is manually operating another app to avoid unexpected focus shifts.
522
-
523
- ---
524
-
525
- ## Auto Guard (v0.12+)
526
-
527
- Action tools (`mouse_click`, `mouse_drag`, `keyboard_type`, `keyboard_press`, `click_element`, `set_element_value`, `browser_click_element`, `browser_navigate`) automatically guard each action when you pass `windowTitle` / `tabId`:
528
-
529
- - Verifies target window identity (process restart / HWND replacement detected)
530
- - Confirms click coordinates are inside the target window rect
531
- - Returns `post.perception.status` on every response — including failures — so the LLM can recover without a screenshot
532
-
533
- **Disabling auto guard** — set `DESKTOP_TOUCH_AUTO_GUARD=0` to restore v0.11.12 behavior (no auto guard):
534
-
535
- ```json
536
- {
537
- "mcpServers": {
538
- "desktop-touch": {
539
- "type": "stdio",
540
- "command": "npx",
541
- "args": ["-y", "@harusame64/desktop-touch-mcp"],
542
- "env": {
543
- "DESKTOP_TOUCH_AUTO_GUARD": "0"
544
- }
545
- }
546
- }
547
- }
548
- ```
549
-
550
- When auto guard is enabled (default), `post.perception.status` will be one of:
551
-
552
- | Status | Meaning |
553
- |---|---|
554
- | `ok` | Guard passed — target verified |
555
- | `unguarded` | `windowTitle` not provided; action ran without guard |
556
- | `target_not_found` | No window matched the given title |
557
- | `ambiguous_target` | Multiple windows matched; use a more specific title |
558
- | `identity_changed` | Window was replaced (process restart / HWND change) |
559
- | `unsafe_coordinates` | Click coordinates are outside the target window rect |
560
- | `needs_escalation` | Use `browser_click_element` or specify `windowTitle` |
561
-
562
- When `unsafe_coordinates` or `identity_changed` is returned, the response may include a `suggestedFix.fixId`. Pass that `fixId` to the relevant tool call to approve the recovery:
563
-
564
- ```json
565
- { "name": "mouse_click", "arguments": { "fixId": "fix-..." } }
566
- { "name": "keyboard_type", "arguments": { "fixId": "fix-...", "text": "hello" } }
567
- { "name": "click_element", "arguments": { "fixId": "fix-..." } }
568
- { "name": "browser_click_element", "arguments": { "fixId": "fix-..." } }
569
- ```
570
-
571
- The fix is one-shot and expires in 15 seconds. The server revalidates the target process identity before executing.
572
-
573
- ---
574
-
575
- ## v0.13 Additions
576
-
577
- ### Target-Identity Timeline
578
-
579
- The server tracks a semantic timeline of what happened to each target window/tab. Recent events are included in:
580
-
581
- - `get_history` → `recentTargetKeys`: array of 3 most recently active target keys (compact, no event bodies)
582
- - `perception_read(lensId)` → `recentEvents`: up to 10 events for that lens's target, each with `tsMs`, `semantic`, `summary`
583
-
584
- Enable the MCP resources below to browse timelines:
585
-
586
- ```json
587
- { "env": { "DESKTOP_TOUCH_PERCEPTION_RESOURCES": "1" } }
588
- ```
589
-
590
- MCP resources available when enabled:
591
-
592
- | URI | Content |
593
- |---|---|
594
- | `perception://target/{targetKey}/timeline` | Semantic event timeline for a target |
595
- | `perception://targets/recent` | Most recently active target keys |
596
- | `perception://lens/{lensId}/summary` | Lens attention/guard state |
597
-
598
- ### Manual Lens Eviction: FIFO → LRU
599
-
600
- Manual lenses (created via `perception_register`) are now evicted by **least-recently-used** instead of insertion order. Using `perception_read`, `evaluatePreToolGuards`, or `buildEnvelopeFor` on a lens promotes it. The hard limit of 16 active lenses is unchanged.
601
-
602
- ### browser_eval Structured Mode
603
-
604
- Pass `withPerception: true` to receive a structured JSON response with `post.perception` instead of raw text:
605
-
606
- ```json
607
- { "name": "browser_eval", "arguments": { "expression": "document.title", "withPerception": true } }
608
- ```
609
-
610
- Returns `{ ok: true, result: "...", post: { perception: { status: "ok", ... } } }`.
611
-
612
- ### mouse_drag Cross-Window Guard
613
-
614
- `mouse_drag` now guards both start and end coordinates. Drags that cross window boundaries (or reach the desktop wallpaper) are blocked by default. To allow intentional cross-window or range-selection drags:
615
-
616
- ```json
617
- { "name": "mouse_drag", "arguments": { "startX": 100, "startY": 100, "endX": 900, "endY": 900, "allowCrossWindowDrag": true } }
618
- ```
619
-
620
- ---
621
-
622
- ## Performance (v0.15 — Rust Native Engine)
623
-
624
- The Rust native engine (`@harusame64/desktop-touch-engine`) replaces PowerShell process spawning with direct COM calls over a persistent MTA thread. It loads automatically as a `.node` addon — no configuration needed.
625
-
626
- ### UIA Benchmark (vs PowerShell baseline)
627
-
628
- | Function | Rust Native | PowerShell | Speedup |
629
- |---|---|---|---|
630
- | `getFocusedElement` | **2.2 ms** | 366 ms | **163.9×** |
631
- | `getUiElements` (Explorer, ~60 elements) | **106.5 ms** | 346 ms | **3.3×** |
632
- | **Weighted average** | | | **~82×** |
633
-
634
- ### Image Diff Benchmark (SSE2 SIMD)
635
-
636
- | Function | Rust (SSE2) | TypeScript | Speedup |
637
- |---|---|---|---|
638
- | `computeChangeFraction` (1920×1080) | **0.26 ms** | 3.8 ms | **~15×** |
639
- | `dHash` (perceptual hash) | **0.09 ms** | 1.2 ms | **~13×** |
640
-
641
- ### Architecture
642
-
643
- ```
644
- Claude CLI / MCP Client
645
- │ stdio or HTTP (MCP protocol)
646
- ▼
647
- desktop-touch-mcp (TypeScript)
648
- │
649
- ├── Rust Native Engine (.node addon) ← NEW in v0.15
650
- │ ├── UIA: 13 functions via napi-rs + windows-rs 0.62
651
- │ │ └── Dedicated COM thread (MTA) + batch BFS algorithm
652
- │ └── Image: SSE2 SIMD pixel diff + perceptual hashing
653
- │
654
- └── PowerShell Fallback (automatic)
655
- └── Activates transparently if .node is unavailable
656
- ```
657
-
658
- ### Why `getUiElements` is 3.3× (not 160×)
659
-
660
- The 160× speedup on `getFocusedElement` comes from eliminating PowerShell process startup (~200 ms) and .NET assembly loading. For `getUiElements`, the bottleneck shifts to the **UIA provider** inside the target application (e.g., Explorer) — it must enumerate its UI tree regardless of who asks. The Rust engine uses a **batch BFS algorithm** (`FindAllBuildCache` + `TreeScope_Children`) that minimizes cross-process RPC calls and supports `maxElements` early exit, making it dramatically faster on large trees (VS Code, browsers with 1000+ elements).
661
-
662
- ---
663
-
664
- ## Known limitations
665
-
666
- | Limitation | Detail | Workaround |
667
- |---|---|---|
668
- | Games / video players may return black or hang in background capture | DirectX fullscreen apps may not work even with `PW_RENDERFULLCONTENT` | Retry with `screenshot_background(fullContent=false)`; if still black, use foreground `screenshot` |
669
- | UIA call overhead | ~2 ms (focus) / ~100 ms (tree) via Rust native engine; ~300 ms via PowerShell fallback | Rust engine loads automatically; `workspace_snapshot` uses a 2 s timeout internally |
670
- | Chrome / WinUI3 UIA elements are empty | Chromium exposes only limited UIA | `screenshot(detail='text')` auto-detects Chromium and falls back to Windows OCR (`hints.chromiumGuard=true`). For richer DOM access use `browser_connect` + `browser_find_element` |
671
- | Chromium title-regex misses when sites rewrite `document.title` | Guard relies on the ` - Google Chrome` suffix being present; some sites push it off the end of a long title | Title is treated as plain Chrome (UIA runs). OCR path is still reachable via `ocrFallback='always'` or when UIA returns `<5` elements (`uiaSparse`) |
672
- | `browser_*` CDP tools need Chrome launched with `--remote-debugging-port` | If Chrome is already running on the default profile without the flag, `browser_launch` / `browser_connect` fail. The CDP E2E suite (`tests/e2e/browser-cdp.test.ts`) will also fail in that state | Close Chrome first, then `browser_launch` will relaunch it in debug mode, or start Chrome manually with `--remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp` |
673
- | Layer buffer TTL | Buffer auto-clears after 90s of inactivity → next `diffMode` becomes an I-frame | After long waits, call `workspace_snapshot` to explicitly reset the buffer |
674
- | `keyboard_type` / `keyboard_press` follow focus | When `dock_window(pin=true)` keeps another window on top (e.g. Claude CLI), keystrokes may be absorbed by that window | Call `focus_window(title=...)` first and verify `isActive=true` via `screenshot(detail='meta')` before sending keys |
675
- | `keyboard_type` em-dash / smart quotes in Chrome/Edge | Non-ASCII punctuation (em-dash `—`, en-dash `–`, smart quotes `"" ''`) can be intercepted as keyboard accelerators, shifting focus to the address bar | Always use `use_clipboard=true` when the text contains such characters |
676
- | `browser_eval` on React / Vue / Svelte inputs | Setting `element.value = ...` or dispatching synthetic events does not update the framework's internal state | Use `browser_fill_input(selector, value)` — it uses native prototype setter + InputEvent which does update React/Vue/Svelte state |
677
-
678
- ---
679
-
680
- ## Token cost reference
681
-
682
- | Mode | Tokens | Use case |
683
- |---|---|---|
684
- | `screenshot` (768px PNG) | ~443 tok | General visual check |
685
- | `screenshot(dotByDot=true)` window | ~800 tok | Precise clicking (no coordinate math) |
686
- | `screenshot(diffMode=true)` | ~160 tok | Post-action diff |
687
- | `screenshot(detail="text")` | ~100–300 tok | UI interaction (no image) |
688
- | `workspace_snapshot` | ~2000 tok | Full session orientation |
689
-
690
- ---
691
-
692
- ## License
693
-
694
- MIT
102
+
103
+ For a local checkout, register the built server directly:
104
+
105
+ ```json
106
+ {
107
+ "mcpServers": {
108
+ "desktop-touch": {
109
+ "type": "stdio",
110
+ "command": "node",
111
+ "args": ["D:/path/to/desktop-touch-mcp/dist/index.js"]
112
+ }
113
+ }
114
+ }
115
+ ```
116
+
117
+ > **Note:** Replace `D:/path/to/desktop-touch-mcp` with the actual path where you cloned this repository.
118
+
119
+ ---
120
+
121
+ ## Tools (28 Optimized Tools)
122
+
123
+ > 📖 **Full Reference**: [`docs/system-overview.md`](docs/system-overview.md) — Exhaustive guide on parameters, return schemas, and coordinate math.
124
+
125
+ ### 🌐 World-Graph V2 (Primary Path)
126
+ | Tool | Description |
127
+ |---|---|
128
+ | `desktop_discover` | Observe the desktop. Returns interactive entities with leases (UIA, CDP, Terminal, Visual SoM). |
129
+ | `desktop_act` | Perform actions (click, type, drag, select) on entities via lease validation. Returns semantic diffs. |
130
+
131
+ ### 👁️ Observation & State
132
+ | Tool | Description |
133
+ |---|---|
134
+ | `desktop_state` | Lightweight check of focus, active window, cursor, and Auto-Perception attention signal. |
135
+ | `screenshot` | Multi-mode capture: `detail='text'` (UIA/OCR), `diffMode` (P-frame), `dotByDot` (1:1), and `background`. |
136
+ | `workspace_snapshot` | Instant session orientation: all window thumbnails + UI summaries in one call. |
137
+ | `server_status` | Diagnostic check for native engine health and feature activation. |
138
+
139
+ ### ⌨️ Input & Control
140
+ | Tool | Description |
141
+ |---|---|
142
+ | `keyboard` | Send keyboard input. Supports background input (WM_CHAR) and IME-safe clipboard bypass. |
143
+ | `mouse_click` / `mouse_drag` | Precision coordinate-based interaction with homing and force-focus protection. |
144
+ | `scroll` | Multi-strategy: `raw` (notches), `to_element`, `smart` (virtual lists), and `capture` (stitch). |
145
+ | `click_element` | Legacy UIA-based click by name/ID (fallback when entities are unavailable). |
146
+
147
+ ### 🌐 Browser CDP (Chrome/Edge/Brave)
148
+ | Tool | Description |
149
+ |---|---|
150
+ | `browser_open` / `browser_navigate` | Idempotent debug-mode launch and reliable navigation. |
151
+ | `browser_click` / `browser_fill` / `browser_form` | High-level DOM interaction stable across repaints and framework re-renders. |
152
+ | `browser_eval` | Deep inspection via `js` (scripting), `dom` (HTML), and `appState` (SPA data extraction). |
153
+ | `browser_overview` / `browser_search` / `browser_locate` | Semantic discovery, grep-like DOM search, and pixel-accurate coordinate lookup. |
154
+
155
+ ### 🛠️ Utilities & Workflow
156
+ | Tool | Description |
157
+ |---|---|
158
+ | `terminal` | Unified command execution: `run` (send + wait + read), `read` (OCR/UIA), and `send`. |
159
+ | `wait_until` | Efficient server-side polling for window, focus, text, or URL state changes. |
160
+ | `window_dock` / `focus_window` | Window management: `pin` (always-on-top), `unpin`, `dock` (corner snap), and `focus`. |
161
+ | `workspace_launch` | Launch apps and auto-detect new HWNDs (supports localized titles). |
162
+ | `run_macro` | Batch up to 50 operations into a single round-trip for maximum efficiency. |
163
+ | `clipboard` / `notification_show` | System-level text exchange and user alerts. |
164
+
165
+ ---
166
+
167
+ ## Standard workflow (v1.0.0)
168
+
169
+ The v2 World-Graph surface (`desktop_discover` / `desktop_act`) is the recommended dispatch path. The four-call shape works for native apps, browsers, and terminals identically.
170
+
171
+ ```
172
+ desktop_state → orient: focused window/element, modal, attention signal
173
+ desktop_discover → find actionable entities (returns lease + windows[])
174
+ desktop_act(lease, …) → act on entity (returns attention + post.perception)
175
+ desktop_state → confirm the world changed as expected
176
+ ```
177
+
178
+ Clicking — priority order:
179
+
180
+ ```
181
+ browser_click(selector) → Chrome / Edge (CDP, stable across repaints)
182
+ desktop_act(lease, action='click') → native / dialog / visual (entity-based; use after desktop_discover)
183
+ click_element(name | automationId) → native UIA fallback if desktop_act returns ok:false
184
+ mouse_click(x, y, origin?, scale?) → pixel last resort; origin+scale from dotByDot screenshots only
185
+ ```
186
+
187
+ Recovery hints — read `response.attention` after every observation and `response.warnings[]` on `desktop_discover` / `desktop_act`. Common reasons:
188
+
189
+ - `lease_expired` / `lease_generation_mismatch` / `lease_digest_mismatch` / `entity_not_found` → re-call `desktop_discover`
190
+ - `modal_blocking` → dismiss via `click_element`, then retry
191
+ - `entity_outside_viewport` → `scroll(action='to_element' | 'raw')`, then re-call `desktop_discover`
192
+ - `executor_failed` → fall back to `click_element` / `mouse_click` / `browser_click`
193
+
194
+ Lease lifecycle:
195
+
196
+ - Each `desktop_discover` response carries `softExpiresAtMs` (≈ 60 % of the TTL window). Past that timestamp the LLM should consider re-calling `desktop_discover` even though the lease is still technically valid — `lease.expiresAtMs` is the only correctness wall.
197
+ - TTL adapts to `view` mode (`action`/`explore`/`debug`), entity count, and response payload size. Cap is 60 s.
198
+ - Set `DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1` to fall back to the v1 tool surface (`get_windows` / `get_ui_elements` / `set_element_value`) for troubleshooting only — V2 is the recommended default.
199
+
200
+ ---
201
+
202
+ ## Browser CDP automation
203
+
204
+ For web automation, connect Chrome or Edge with the remote debugging port enabled — no Selenium or Playwright needed.
205
+
206
+ ```bash
207
+ # Launch Chrome in CDP mode
208
+ chrome.exe --remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp
209
+ ```
210
+
211
+ ```
212
+ browser_open({launch:{}}) → spawn-if-needed Chrome in debug mode + list tabs (idempotent)
213
+ browser_open() → connect-only (fail if no CDP endpoint live)
214
+ browser_locate({selector:"#submit"}) → CSS selector → physical screen coords
215
+ browser_click({selector:"#submit"}) → find + click in one step (auto-focuses browser)
216
+ browser_eval({action:"js", expression:"document.title"}) → evaluate JS, returns result
217
+ browser_eval({action:"dom", selector:"#main", maxLength:5000}) → outerHTML, truncated to maxLength chars
218
+ browser_eval({action:"appState"}) → one-shot SPA state (Next/Nuxt/Remix/Apollo/GitHub react-app/Redux SSR)
219
+ browser_fill({selector:"#email", value:"user@example.com"}) → fill React/Vue/Svelte controlled input (state-safe)
220
+ browser_overview() → links/buttons/inputs + ARIA toggles + viewportPosition per element
221
+ browser_search({by:"text", pattern:"..."}) → grep DOM with confidence ranking
222
+ browser_navigate({url:"https://example.com"}) → navigate via CDP (no address bar interaction)
223
+ ```
224
+
225
+ For chained calls in the same tab, pass `includeContext:false` to omit the activeTab/readyState annotation (~150 tok/call saved). Boolean / object params accept the LLM-friendly string spellings (`"true"`, `"{}"`).
226
+
227
+ Coordinates returned by `browser_locate` account for the browser chrome (tab strip + address bar height) and `devicePixelRatio`, so they can be passed directly to `mouse_click` without any scaling.
228
+
229
+ **Recommended web workflow:**
230
+ ```
231
+ browser_open({launch:{}}) → browser_eval({action:"dom"}) → browser_locate(selector) → browser_click(selector)
232
+ ```
233
+
234
+ ---
235
+
236
+ ## Auto-dock CLI on startup
237
+
238
+ Keep Claude CLI visible while operating other apps full-screen. Set env vars in your MCP config and the docked window auto-snaps into place every MCP startup.
239
+
240
+ ```json
241
+ {
242
+ "mcpServers": {
243
+ "desktop-touch": {
244
+ "type": "stdio",
245
+ "command": "npx",
246
+ "args": ["-y", "@harusame64/desktop-touch-mcp"],
247
+ "env": {
248
+ "DESKTOP_TOUCH_DOCK_TITLE": "@parent",
249
+ "DESKTOP_TOUCH_DOCK_CORNER": "bottom-right",
250
+ "DESKTOP_TOUCH_DOCK_WIDTH": "480",
251
+ "DESKTOP_TOUCH_DOCK_HEIGHT": "360",
252
+ "DESKTOP_TOUCH_DOCK_PIN": "true"
253
+ }
254
+ }
255
+ }
256
+ }
257
+ ```
258
+
259
+ | Env var | Default | Notes |
260
+ |---|---|---|
261
+ | `DESKTOP_TOUCH_DOCK_TITLE` | *(unset = off)* | `@parent` walks the MCP process tree to find the hosting terminal — immune to title / branch / project changes. Or use a literal substring. |
262
+ | `DESKTOP_TOUCH_DOCK_CORNER` | `bottom-right` | `top-left` / `top-right` / `bottom-left` / `bottom-right` |
263
+ | `DESKTOP_TOUCH_DOCK_WIDTH` / `HEIGHT` | `480` / `360` | px (`"480"`) or ratio of work area (`"25%"`) — 4K/8K auto-adapts |
264
+ | `DESKTOP_TOUCH_DOCK_PIN` | `true` | Always-on-top toggle |
265
+ | `DESKTOP_TOUCH_DOCK_MONITOR` | primary | Monitor id from `desktop_state({includeScreen:true})` |
266
+ | `DESKTOP_TOUCH_DOCK_SCALE_DPI` | `false` | If true, multiply px values by `dpi / 96` (opt-in per-monitor scaling) |
267
+ | `DESKTOP_TOUCH_DOCK_MARGIN` | `8` | Screen-edge padding (px) |
268
+ | `DESKTOP_TOUCH_DOCK_TIMEOUT_MS` | `5000` | Max wait for the target window to appear |
269
+
270
+ > **Input routing gotcha:** when a pinned window is active (e.g. Claude CLI), `keyboard(action='type')` / `keyboard(action='press')` send keys to it, **not** the app you wanted to type into. Always call `focus_window(title=...)` before keyboard operations, then verify `isActive=true` via `screenshot(detail='meta')`.
271
+
272
+ ### Auto Perception (always-on)
273
+
274
+ Phase 4 privatizes the explicit `perception_*` tool family — the v0.12 Auto
275
+ Perception layer attaches an `attention` signal to every `desktop_state` and
276
+ `desktop_act` response automatically. Action tools also auto-guard when given
277
+ a `windowTitle`. There is no longer a need to register / read / forget lenses
278
+ manually.
279
+
280
+ ```
281
+ # desktop_state always returns the attention signal
282
+ desktop_state() → {focusedWindow, focusedElement, modal, attention:"ok", ...}
283
+
284
+ # Action tools auto-guard when windowTitle is given:
285
+ keyboard({action:"type", text:"hello", windowTitle:"Notepad"})
286
+ → post.perception:{status:"ok"} // unsafe input blocked if guards fail
287
+
288
+ # When attention is dirty / stale / settling, refresh with desktop_state:
289
+ desktop_state() // re-evaluates attention via Auto Perception
290
+ ```
291
+
292
+ For advanced pinned-target workflows, the `lensId` parameter remains on action
293
+ tools (`keyboard`, `mouse_click`, `mouse_drag`, `click_element`,
294
+ `browser_click`, `browser_navigate`, `browser_eval`, `desktop_act`). Omit
295
+ `lensId` for the normal Auto Perception path. The underlying registry, hot
296
+ target cache, and sensor loop are unchanged; only the explicit
297
+ `perception_register / perception_read / perception_forget / perception_list`
298
+ tools were retired.
299
+
300
+ ---
301
+
302
+ ## Mouse homing correction
303
+
304
+ When Claude calls `screenshot(detail='text')` to read coordinates and then `mouse_click` seconds later, the target window may have moved. The homing system corrects this automatically.
305
+
306
+ | Tier | How to enable | Latency | What it does |
307
+ |------|--------------|---------|--------------|
308
+ | 1 | Always-on (if cache exists) | <1ms | Applies (dx, dy) offset when window moved |
309
+ | 2 | Pass `windowTitle` hint | ~100ms | Auto-focuses window if it went behind another |
310
+ | 3 | Pass `elementName`/`elementId` + `windowTitle` | 1–3s | UIA re-query for fresh coords on resize |
311
+
312
+ ```
313
+ # Tier 1 only (automatic)
314
+ mouse_click(x=500, y=300)
315
+
316
+ # Tier 1 + 2: also bring window to front if hidden
317
+ mouse_click(x=500, y=300, windowTitle="Notepad")
318
+
319
+ # Tier 1 + 2 + 3: also re-query UIA if window resized
320
+ mouse_click(x=500, y=300, windowTitle="Notepad", elementName="Save")
321
+
322
+ # Traction control OFF — no correction
323
+ mouse_click(x=500, y=300, homing=false)
324
+ ```
325
+
326
+ The `homing` parameter is available on `mouse_click`, `mouse_drag`, and `scroll`. The cache is updated automatically on every `screenshot()`, `desktop_discover()`, `focus_window()`, and `workspace_snapshot()` call.
327
+
328
+ ### `mouse_click` image-local coords (origin + scale)
329
+
330
+ When you take a `dotByDot` screenshot with `dotByDotMaxDimension`, the response prints the `origin` and `scale` values. Instead of computing screen coords manually, copy them into `mouse_click`:
331
+
332
+ ```
333
+ # Screenshot response:
334
+ # origin: (0, 120) | scale: 0.6667
335
+ # To click image pixel (ix, iy): mouse_click(x=ix, y=iy, origin={x:0, y:120}, scale=0.6667)
336
+
337
+ mouse_click(x=640, y=300, origin={x:0, y:120}, scale=0.6667, windowTitle="Chrome")
338
+ # Server converts: screen = (0 + 640/0.6667, 120 + 300/0.6667) = (960, 570)
339
+ ```
340
+
341
+ This eliminates a whole class of off-by-one and scale bugs. Without origin/scale, `x`/`y` remain absolute screen pixels (unchanged behavior).
342
+
343
+ ---
344
+
345
+ ## `screenshot` key parameters
346
+
347
+ ```
348
+ detail="image" — PNG/WebP pixels (default)
349
+ detail="text" — UIA element JSON + clickAt coords (no image, ~100–300 tok)
350
+ detail="meta" — Title + region only (cheapest, ~20 tok/window)
351
+ dotByDot=true — 1:1 WebP; image_px + origin = screen_px
352
+ dotByDotMaxDimension=N — cap longest edge (response includes scale for coord math)
353
+ grayscale=true — ~50% smaller for text-heavy captures (code/AWS console)
354
+ region={x,y,w,h} — with windowTitle: window-local coords (exclude browser chrome)
355
+ without: virtual screen coords
356
+ diffMode=true — I-frame first call, P-frame (changed windows only) after (~160 tok)
357
+ ocrFallback="auto" — detail='text' auto-fires Windows OCR on uiaSparse or empty
358
+ ```
359
+
360
+ **Recommended Chrome combo** (50–70% data reduction):
361
+ ```
362
+ screenshot(windowTitle="Chrome",
363
+ dotByDot=true, dotByDotMaxDimension=1280, grayscale=true,
364
+ region={x:0, y:120, width:1920, height:900}) # skip browser chrome
365
+ ```
366
+
367
+ **Recommended workflow:**
368
+ ```
369
+ workspace_snapshot() → full orientation (resets diff buffer)
370
+ screenshot(detail="text", windowTitle=X) → get actionable[].clickAt coords
371
+ mouse_click(x, y) → click directly, no math needed
372
+ screenshot(diffMode=true) → check only what changed (~160 tok)
373
+ ```
374
+
375
+ ---
376
+
377
+ ## Security
378
+
379
+ ### Emergency stop (Failsafe)
380
+
381
+ **Move the mouse to the top-left corner of the screen (within 10px of 0,0) to immediately terminate the MCP server.**
382
+
383
+ - **Per-tool check**: `checkFailsafe()` runs before every tool handler
384
+ - **Background monitor**: 500ms polling as a backup for long-running operations
385
+ - Trigger radius: 10px
386
+
387
+ ### Blocked operations
388
+
389
+ **`workspace_launch` blocklist:**
390
+ `cmd.exe`, `powershell.exe`, `pwsh.exe`, `wscript.exe`, `cscript.exe`, `mshta.exe`, `regsvr32.exe`, `rundll32.exe`, `msiexec.exe`, `bash.exe`, `wsl.exe` are blocked.
391
+ Script extensions (`.bat`, `.ps1`, `.vbs`, etc.) are rejected. Arguments containing `;`, `&`, `|`, `` ` ``, `$(`, `${` are also rejected.
392
+
393
+ **`keyboard(action='press')` blocklist:**
394
+ `Win+R` (Run dialog), `Win+X` (admin menu), `Win+S` (search), `Win+L` (lock screen) are blocked.
395
+
396
+ ### PowerShell injection protection
397
+
398
+ All `-like` patterns in the UIA bridge PowerShell fallback path are sanitized with `escapeLike()`, which escapes wildcard characters (`*`, `?`, `[`, `]`) before they reach PowerShell. When the Rust native engine is active, PowerShell is not invoked for UIA operations.
399
+
400
+ ### Allowlist for `workspace_launch`
401
+
402
+ Shell interpreters are blocked by default. To allow specific executables, create an allowlist file:
403
+
404
+ **File locations (searched in order):**
405
+ 1. Path in `DESKTOP_TOUCH_ALLOWLIST` environment variable
406
+ 2. `~/.claude/desktop-touch-allowlist.json`
407
+ 3. `desktop-touch-allowlist.json` in the server's working directory
408
+
409
+ **Format:**
410
+ ```json
411
+ {
412
+ "allowedExecutables": [
413
+ "pwsh.exe",
414
+ "C:\\Tools\\myapp.exe"
415
+ ]
416
+ }
417
+ ```
418
+
419
+ Changes take effect immediately — no restart needed.
420
+
421
+ ---
422
+
423
+ ## Mouse movement speed
424
+
425
+ All mouse tools (`mouse_click`, `mouse_drag`, `scroll`) accept an optional `speed` parameter:
426
+
427
+ | Value | Behavior |
428
+ |---|---|
429
+ | Omitted | Uses the configured default (see below) |
430
+ | `0` | Instant teleport — `setPosition()`, no animation |
431
+ | `1–N` | Animated movement at N px/sec |
432
+
433
+ **Default speed** is 1500 px/sec. Change it permanently via the `DESKTOP_TOUCH_MOUSE_SPEED` environment variable:
434
+
435
+ ```json
436
+ {
437
+ "mcpServers": {
438
+ "desktop-touch": {
439
+ "type": "stdio",
440
+ "command": "npx",
441
+ "args": ["-y", "@harusame64/desktop-touch-mcp"],
442
+ "env": {
443
+ "DESKTOP_TOUCH_MOUSE_SPEED": "3000"
444
+ }
445
+ }
446
+ }
447
+ }
448
+ ```
449
+
450
+ Common values: `0` = teleport, `1500` = default gentle, `3000` = fast, `5000` = very fast.
451
+
452
+ ---
453
+
454
+ ## Force-Focus (AttachThreadInput)
455
+
456
+ Windows foreground-stealing protection can prevent `SetForegroundWindow` from succeeding when another window (such as a pinned Claude CLI) is in the foreground. This causes subsequent keystrokes or clicks to land in the wrong window — a silent failure.
457
+
458
+ `mouse_click`, `keyboard(action='type')`, `keyboard(action='press')`, and `terminal(action='send')` all accept a `forceFocus` parameter that bypasses this protection using `AttachThreadInput`:
459
+
460
+ ```json
461
+ {
462
+ "name": "mouse_click",
463
+ "arguments": {
464
+ "x": 500,
465
+ "y": 300,
466
+ "windowTitle": "Google Chrome",
467
+ "forceFocus": true
468
+ }
469
+ }
470
+ ```
471
+
472
+ If the force attempt is refused despite `AttachThreadInput`, the response includes `hints.warnings: ["ForceFocusRefused"]`.
473
+
474
+ **Global default via environment variable:**
475
+
476
+ ```json
477
+ {
478
+ "mcpServers": {
479
+ "desktop-touch": {
480
+ "env": {
481
+ "DESKTOP_TOUCH_FORCE_FOCUS": "1"
482
+ }
483
+ }
484
+ }
485
+ }
486
+ ```
487
+
488
+ Setting `DESKTOP_TOUCH_FORCE_FOCUS=1` makes `forceFocus: true` the default for all four tools without changing each call.
489
+
490
+ **Known tradeoffs:**
491
+
492
+ - During the ~10ms `AttachThreadInput` window, key state and mouse capture are shared between the two threads. In rapid macro sequences this can cause a race condition (rare in practice).
493
+ - Disable `forceFocus` (or unset the env var) when the user is manually operating another app to avoid unexpected focus shifts.
494
+
495
+ ---
496
+
497
+ ## Auto Guard (v0.12+)
498
+
499
+ Action tools (`mouse_click`, `mouse_drag`, `keyboard(action='type'/'press')`, `click_element`, `desktop_act`, `browser_click`, `browser_navigate`) automatically guard each action when you pass `windowTitle` / `tabId`:
500
+
501
+ - Verifies target window identity (process restart / HWND replacement detected)
502
+ - Confirms click coordinates are inside the target window rect
503
+ - Returns `post.perception.status` on every response — including failures — so the LLM can recover without a screenshot
504
+
505
+ **Disabling auto guard** — set `DESKTOP_TOUCH_AUTO_GUARD=0` to restore v0.11.12 behavior (no auto guard):
506
+
507
+ ```json
508
+ {
509
+ "mcpServers": {
510
+ "desktop-touch": {
511
+ "type": "stdio",
512
+ "command": "npx",
513
+ "args": ["-y", "@harusame64/desktop-touch-mcp"],
514
+ "env": {
515
+ "DESKTOP_TOUCH_AUTO_GUARD": "0"
516
+ }
517
+ }
518
+ }
519
+ }
520
+ ```
521
+
522
+ When auto guard is enabled (default), `post.perception.status` will be one of:
523
+
524
+ | Status | Meaning |
525
+ |---|---|
526
+ | `ok` | Guard passed — target verified |
527
+ | `unguarded` | `windowTitle` not provided; action ran without guard |
528
+ | `target_not_found` | No window matched the given title |
529
+ | `ambiguous_target` | Multiple windows matched; use a more specific title |
530
+ | `identity_changed` | Window was replaced (process restart / HWND change) |
531
+ | `unsafe_coordinates` | Click coordinates are outside the target window rect |
532
+ | `needs_escalation` | Use `browser_click` or specify `windowTitle` |
533
+
534
+ When `unsafe_coordinates` or `identity_changed` is returned, the response may include a `suggestedFix.fixId`. Pass that `fixId` to the relevant tool call to approve the recovery:
535
+
536
+ ```json
537
+ { "name": "mouse_click", "arguments": { "fixId": "fix-..." } }
538
+ { "name": "keyboard(action='type')", "arguments": { "fixId": "fix-...", "text": "hello" } }
539
+ { "name": "click_element", "arguments": { "fixId": "fix-..." } }
540
+ { "name": "browser_click", "arguments": { "fixId": "fix-..." } }
541
+ ```
542
+
543
+ The fix is one-shot and expires in 15 seconds. The server revalidates the target process identity before executing.
544
+
545
+ ---
546
+
547
+ ## v0.13 Additions
548
+
549
+ ### Target-Identity Timeline
550
+
551
+ The server tracks a semantic timeline of what happened to each target window/tab. Recent events are included in:
552
+
553
+ - `get_history` → `recentTargetKeys`: array of 3 most recently active target keys (compact, no event bodies)
554
+ - `perception_read(lensId)` → `recentEvents`: up to 10 events for that lens's target, each with `tsMs`, `semantic`, `summary`
555
+
556
+ Enable the MCP resources below to browse timelines:
557
+
558
+ ```json
559
+ { "env": { "DESKTOP_TOUCH_PERCEPTION_RESOURCES": "1" } }
560
+ ```
561
+
562
+ MCP resources available when enabled:
563
+
564
+ | URI | Content |
565
+ |---|---|
566
+ | `perception://target/{targetKey}/timeline` | Semantic event timeline for a target |
567
+ | `perception://targets/recent` | Most recently active target keys |
568
+ | `perception://lens/{lensId}/summary` | Lens attention/guard state |
569
+
570
+ ### Manual Lens Eviction: FIFO → LRU
571
+
572
+ Manual lenses (created via `perception_register`) are now evicted by **least-recently-used** instead of insertion order. Using `perception_read`, `evaluatePreToolGuards`, or `buildEnvelopeFor` on a lens promotes it. The hard limit of 16 active lenses is unchanged.
573
+
574
+ ### browser_eval Structured Mode
575
+
576
+ Pass `withPerception: true` to receive a structured JSON response with `post.perception` instead of raw text:
577
+
578
+ ```json
579
+ { "name": "browser_eval", "arguments": { "expression": "document.title", "withPerception": true } }
580
+ ```
581
+
582
+ Returns `{ ok: true, result: "...", post: { perception: { status: "ok", ... } } }`.
583
+
584
+ ### mouse_drag Cross-Window Guard
585
+
586
+ `mouse_drag` now guards both start and end coordinates. Drags that cross window boundaries (or reach the desktop wallpaper) are blocked by default. To allow intentional cross-window or range-selection drags:
587
+
588
+ ```json
589
+ { "name": "mouse_drag", "arguments": { "startX": 100, "startY": 100, "endX": 900, "endY": 900, "allowCrossWindowDrag": true } }
590
+ ```
591
+
592
+ ---
593
+
594
+ ## Performance (v0.15 — Rust Native Engine)
595
+
596
+ The Rust native engine (`@harusame64/desktop-touch-engine`) replaces PowerShell process spawning with direct COM calls over a persistent MTA thread. It loads automatically as a `.node` addon — no configuration needed.
597
+
598
+ ### UIA Benchmark (vs PowerShell baseline)
599
+
600
+ | Function | Rust Native | PowerShell | Speedup |
601
+ |---|---|---|---|
602
+ | `getFocusedElement` | **2.2 ms** | 366 ms | **163.9×** |
603
+ | `getUiElements` (Explorer, ~60 elements) | **106.5 ms** | 346 ms | **3.3×** |
604
+ | **Weighted average** | | | **~82×** |
605
+
606
+ ### Image Diff Benchmark (SSE2 SIMD)
607
+
608
+ | Function | Rust (SSE2) | TypeScript | Speedup |
609
+ |---|---|---|---|
610
+ | `computeChangeFraction` (1920×1080) | **0.26 ms** | 3.8 ms | **~15×** |
611
+ | `dHash` (perceptual hash) | **0.09 ms** | 1.2 ms | **~13×** |
612
+
613
+ ### Architecture
614
+
615
+ ```
616
+ Claude CLI / MCP Client
617
+ │ stdio or HTTP (MCP protocol)
618
+ ▼
619
+ desktop-touch-mcp (TypeScript)
620
+ │
621
+ ├── Rust Native Engine (.node addon) ← NEW in v0.15
622
+ │ ├── UIA: 13 functions via napi-rs + windows-rs 0.62
623
+ │ │ └── Dedicated COM thread (MTA) + batch BFS algorithm
624
+ │ └── Image: SSE2 SIMD pixel diff + perceptual hashing
625
+ │
626
+ └── PowerShell Fallback (automatic)
627
+ └── Activates transparently if .node is unavailable
628
+ ```
629
+
630
+ ### Why `getUiElements` is 3.3× (not 160×)
631
+
632
+ The 160× speedup on `getFocusedElement` comes from eliminating PowerShell process startup (~200 ms) and .NET assembly loading. For `getUiElements`, the bottleneck shifts to the **UIA provider** inside the target application (e.g., Explorer) — it must enumerate its UI tree regardless of who asks. The Rust engine uses a **batch BFS algorithm** (`FindAllBuildCache` + `TreeScope_Children`) that minimizes cross-process RPC calls and supports `maxElements` early exit, making it dramatically faster on large trees (VS Code, browsers with 1000+ elements).
633
+
634
+ ---
635
+
636
+ ## UI Operating Layer (V2)
637
+
638
+ > **Status: Default ON since v0.17.** `desktop_discover` and `desktop_act` are available out of the box.
639
+
640
+ V2 introduces two new tools that replace coordinate-based clicking with entity-based interaction:
641
+
642
+ | Tool | Description |
643
+ |---|---|
644
+ | `desktop_discover` | Observe a window or browser tab. Returns interactive entities with leases — no raw screen coordinates. Supports UIA (native), CDP (browser), terminal, and visual GPU lanes. |
645
+ | `desktop_act` | Interact with an entity returned by `desktop_discover`. Validates the lease before executing. Returns a semantic diff (`entity_disappeared`, `modal_appeared`, `focus_shifted`, …). |
646
+
647
+ ### Clicking — priority order
648
+
649
+ When multiple tools could perform the same click, prefer them in this order:
650
+
651
+ 1. `browser_click(selector)` — Chrome / Edge over CDP (stable across repaints)
652
+ 2. `desktop_act(lease)` — native windows, dialogs, visual-only targets (entity-based; use after `desktop_discover`)
653
+ 3. `click_element(name | automationId)` — native UIA fallback when `desktop_act` returns `ok:false`
654
+ 4. `mouse_click(x, y)` — pixel-level last resort (`origin` + `scale` from `dotByDot` screenshots only)
655
+
656
+ ### Disabling V2 (kill switch)
657
+
658
+ To hide `desktop_discover` / `desktop_act` from the tool catalog, add the disable flag and restart:
659
+
660
+ ```json
661
+ {
662
+ "mcpServers": {
663
+ "desktop-touch": {
664
+ "type": "stdio",
665
+ "command": "npx",
666
+ "args": ["-y", "@harusame64/desktop-touch-mcp"],
667
+ "env": {
668
+ "DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2": "1"
669
+ }
670
+ }
671
+ }
672
+ }
673
+ ```
674
+
675
+ All V1 tools continue to work without interruption — no reinstall required. Remove the env entry and restart to re-enable.
676
+
677
+ Flag semantics (exact-match: only the literal string `"1"` counts):
678
+
679
+ | `DISABLE_FUKUWARAI_V2` | `ENABLE_FUKUWARAI_V2` | V2 state |
680
+ |---|---|---|
681
+ | unset / not `"1"` | unset / not `"1"` | **ON** (default) |
682
+ | unset / not `"1"` | `"1"` | ON (legacy flag — see below) |
683
+ | `"1"` | any | **OFF** — DISABLE wins |
684
+
685
+ ### Deprecated: `DESKTOP_TOUCH_ENABLE_FUKUWARAI_V2`
686
+
687
+ This was the opt-in switch in v0.16.x. From v0.17 it is accepted for compatibility but no longer required — the server prints a deprecation warning on startup when it is set. It will be removed in v0.18. Remove it from your config when you upgrade.
688
+
689
+ ### Recovery when V2 fails
690
+
691
+ If `desktop_act` returns `ok: false`, read `reason` and follow the built-in recovery hints in the tool description. Common paths:
692
+
693
+ - `lease_expired` / `*_mismatch` / `entity_not_found` → re-call `desktop_discover`
694
+ - `modal_blocking` → dismiss the modal with `click_element`, then retry
695
+ - `entity_outside_viewport` → `scroll` / `scroll(action='to_element')`, then re-call `desktop_discover`
696
+ - `executor_failed` → fall back to `click_element` / `mouse_click` / `browser_click`
697
+
698
+ For `desktop_discover` warnings (`visual_provider_unavailable`, `visual_provider_warming`, `cdp_provider_failed`, …), V1 tools (`screenshot`, `click_element`, `get_ui_elements`, `terminal(action='send')`, …) remain available as an escape hatch.
699
+
700
+ ---
701
+
702
+ ## Known limitations
703
+
704
+ | Limitation | Detail | Workaround |
705
+ |---|---|---|
706
+ | Games / video players may return black or hang in background capture | DirectX fullscreen apps may not work even with `PW_RENDERFULLCONTENT` | Retry with `screenshot_background(fullContent=false)`; if still black, use foreground `screenshot` |
707
+ | UIA call overhead | ~2 ms (focus) / ~100 ms (tree) via Rust native engine; ~300 ms via PowerShell fallback | Rust engine loads automatically; `workspace_snapshot` uses a 2 s timeout internally |
708
+ | Chrome / WinUI3 UIA elements are empty | Chromium exposes only limited UIA | `screenshot(detail='text')` auto-detects Chromium and falls back to Windows OCR (`hints.chromiumGuard=true`). For richer DOM access use `browser_open` + `browser_locate` |
709
+ | Chromium title-regex misses when sites rewrite `document.title` | Guard relies on the ` - Google Chrome` suffix being present; some sites push it off the end of a long title | Title is treated as plain Chrome (UIA runs). OCR path is still reachable via `ocrFallback='always'` or when UIA returns `<5` elements (`uiaSparse`) |
710
+ | `browser_*` CDP tools need Chrome launched with `--remote-debugging-port` | If Chrome is already running on the default profile without the flag, `browser_open` fails. The CDP E2E suite (`tests/e2e/browser-cdp.test.ts`) will also fail in that state | Close Chrome first, then `browser_open({launch:{}})` will relaunch it in debug mode, or start Chrome manually with `--remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp` |
711
+ | Layer buffer TTL | Buffer auto-clears after 90s of inactivity → next `diffMode` becomes an I-frame | After long waits, call `workspace_snapshot` to explicitly reset the buffer |
712
+ | `keyboard(action='type')` / `keyboard(action='press')` follow focus | When `window_dock(action='dock')(pin=true)` keeps another window on top (e.g. Claude CLI), keystrokes may be absorbed by that window | Call `focus_window(title=...)` first and verify `isActive=true` via `screenshot(detail='meta')` before sending keys |
713
+ | `keyboard(action='type')` em-dash / smart quotes in Chrome/Edge | Non-ASCII punctuation (em-dash `—`, en-dash `–`, smart quotes `"" ''`) can be intercepted as keyboard accelerators, shifting focus to the address bar | Always use `use_clipboard=true` when the text contains such characters |
714
+ | `browser_eval(action='js')` on React / Vue / Svelte inputs | Setting `element.value = ...` or dispatching synthetic events does not update the framework's internal state | Use `browser_fill(selector, value)` — it uses native prototype setter + InputEvent which does update React/Vue/Svelte state |
715
+
716
+ ---
717
+
718
+ ## Token cost reference
719
+
720
+ | Mode | Tokens | Use case |
721
+ |---|---|---|
722
+ | `screenshot` (768px PNG) | ~443 tok | General visual check |
723
+ | `screenshot(dotByDot=true)` window | ~800 tok | Precise clicking (no coordinate math) |
724
+ | `screenshot(diffMode=true)` | ~160 tok | Post-action diff |
725
+ | `screenshot(detail="text")` | ~100–300 tok | UI interaction (no image) |
726
+ | `workspace_snapshot` | ~2000 tok | Full session orientation |
727
+
728
+ ---
729
+
730
+ ## License
731
+
732
+ MIT