@harusame64/desktop-touch-mcp 1.12.1 → 1.12.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.ja.md +737 -737
- package/README.md +861 -861
- package/bin/launcher.js +3 -3
- package/package.json +6 -6
package/README.md
CHANGED
|
@@ -1,861 +1,861 @@
|
|
|
1
|
-
# desktop-touch-mcp
|
|
2
|
-
|
|
3
|
-
[](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp)
|
|
4
|
-
|
|
5
|
-
[日本語](README.ja.md)
|
|
6
|
-
|
|
7
|
-
> **Computer-use MCP server for Windows.** Lets Claude, Cursor, or any MCP client see and operate your Windows 10/11 desktop — screenshots, UI Automation, Chrome CDP, keyboard / mouse, terminal — with **semantic discover-then-act targeting** that avoids pixel-coordinate guessing, and **per-action perception guards** that catch wrong-window typing before it happens.
|
|
8
|
-
|
|
9
|
-
```bash
|
|
10
|
-
npx -y @harusame64/desktop-touch-mcp
|
|
11
|
-
```
|
|
12
|
-
|
|
13
|
-
32 tools, native Rust engine (UIA in 2 ms), zero-config PowerShell fallback, full CJK support, MIT licensed. Add the snippet above to your Claude / Cursor / VS Code Copilot config and Claude can drive Notepad, Excel, Chrome, Windows Terminal, and any other app on your machine.
|
|
14
|
-
|
|
15
|
-
> **Why this over pixel-clicking?** Two ideas run through every tool: **discover-then-act** — `desktop_discover` returns interactive entities with short-lived leases instead of raw coordinates, so `desktop_act` operates on *what* you mean, not *where* it was — and **per-action perception guards** that verify the target window's identity and bounds before input lands, catching wrong-window typing and stale-coordinate clicks before they happen.
|
|
16
|
-
>
|
|
17
|
-
> Under the hood: an **82× average speedup** from the Rust native engine (UIA focus queries in 2 ms, SSE2-accelerated image diffing at 13–15×), with a transparent PowerShell fallback when the engine is absent. The npm launcher fetches only the GitHub Release tag matching the installed version and verifies the Windows runtime zip before extraction.
|
|
18
|
-
|
|
19
|
-
---
|
|
20
|
-
|
|
21
|
-
## Features
|
|
22
|
-
|
|
23
|
-
- **⚡ High-performance Rust Native Core** — The UIA bridge and image-diff engine are written in Rust (`napi-rs` + `windows-rs`) and loaded as a native `.node` addon. Direct COM calls from a dedicated MTA thread eliminate PowerShell process spawning — `getFocusedElement` completes in **2 ms** (160× faster), and `getUiElements` returns full trees in **~100 ms** with a batch BFS algorithm that minimizes cross-process RPC. Image-diff operations use **SSE2 SIMD** for 13–15× throughput. When the native engine is unavailable, every function transparently falls back to PowerShell — zero config required.
|
|
24
|
-
- **🎯 Set-of-Marks (SoM) visual fallback** — Games, RDP sessions, and non-accessible Electron apps return clickable elements even when UIA is completely blind. `screenshot(detail="text")` automatically detects UIA sparsity and activates a Hybrid Non-CDP pipeline: Rust-powered grayscale + bilinear upscale → Windows OCR → clustering → red bounding-box annotation with numbered badges (`[1]`, `[2]`…). Two parallel representations returned: a visual PNG for spatial orientation and a semantic `elements[]` list with `clickAt` coords — no CDP required.
|
|
25
|
-
- **🔁 One-call confirmation on visual-only targets** — On UIA-blind targets (Electron, PWAs, games, custom canvases, RDP windows), `desktop_act` can fold the post-action confirmation into its own response: an optional `roiCapture` carrying a PNG crop of *just the region that changed* plus a lease-less preview of the controls now visible there. The agent confirms what its click did and finds the next target without a separate `desktop_state` + `screenshot`. On visual-only targets it is **on by default** for a visible change (`returnCapture:"on-change"`); pass `returnCapture:"never"` to suppress it, or `"always"` to force it. Never attached on structured targets (browser/CDP, UIA-rich native), where `desktop_state` is cheaper and exact — so those responses are unchanged.
|
|
26
|
-
- **🔐 Key Locker — the terminal autofills your SSH / sudo passwords** — Save a credential once into the locker's own secure dialog (stored encrypted on your machine with Windows DPAPI; never shown to the assistant), then run `ssh` / `sudo` in a console opened by `key_locker(action='launch_console')` — the password is filled in automatically when the hidden prompt appears, with a per-fill confirmation prompt by default. See [Key Locker](#key-locker-terminal-credential-autofill).
|
|
27
|
-
- **LLM-native design** — Built around how LLMs think, not how humans click. `run_macro` batches multiple operations into a single API call; `diffMode` sends only the windows that changed since the last frame. Minimal tokens, minimal round-trips.
|
|
28
|
-
- **Reactive Perception Graph** — Register a `lensId` for a window or browser tab, pass it to action tools, and get guard-checked `post.perception` feedback after each action. It reduces repeated `screenshot` / `desktop_state` calls and prevents wrong-window typing or stale-coordinate clicks.
|
|
29
|
-
- **Full CJK support** — Uses Win32 `GetWindowTextW` for window titles, avoiding nut-js garbling. IME bypass input supported for Japanese/Chinese/Korean environments.
|
|
30
|
-
- **3-tier token reduction** — `detail="image"` (~443 tok) / `detail="text"` (~100–300 tok) / `diffMode=true` (~160 tok). Send pixels only when you actually need to see them.
|
|
31
|
-
- **1:1 coordinate mode** — `dotByDot=true` captures at native resolution (WebP). Image pixel = screen coordinate — no scale math needed. With `origin`+`scale` passed to `mouse_click`, the server converts coords for you — eliminating off-by-one / scale bugs.
|
|
32
|
-
- **Browser capture data reduction** — `grayscale=true` (~50% size), `dotByDotMaxDimension=1280` (auto-scaled with coord preservation), and `windowTitle + region` sub-crops help exclude browser chrome and other irrelevant pixels. Typical reduction for heavy captures: 50–70%.
|
|
33
|
-
- **Chromium smart fallback** — `detail="text"` on Chrome/Edge/Brave auto-skips UIA (prohibitively slow there) and runs Windows OCR. `hints.chromiumGuard` + `hints.ocrFallbackFired` flag the path taken.
|
|
34
|
-
- **UIA element extraction** — `detail="text"` returns button names and `clickAt` coords as JSON. Claude can click the right element without ever looking at a screenshot.
|
|
35
|
-
- **Auto-dock CLI** — `window_dock(action='dock')` snaps any window to a screen corner with always-on-top. Set `DESKTOP_TOUCH_DOCK_TITLE='@parent'` to auto-dock the terminal hosting Claude on MCP startup — the process-tree walker finds the right window regardless of title.
|
|
36
|
-
- **Emergency stop (Failsafe)** — Move the mouse to the **top-left corner (within 10px of 0,0)** to immediately terminate the MCP server.
|
|
37
|
-
|
|
38
|
-
---
|
|
39
|
-
|
|
40
|
-
## Requirements
|
|
41
|
-
|
|
42
|
-
| | |
|
|
43
|
-
|---|---|
|
|
44
|
-
| OS | Windows 10 / 11 (64-bit) |
|
|
45
|
-
| Node.js | v20+ recommended (tested on v22+) |
|
|
46
|
-
| PowerShell | 5.1+ (bundled with Windows) — used only as fallback when the Rust native engine is unavailable |
|
|
47
|
-
| Claude CLI | `claude` command must be available |
|
|
48
|
-
|
|
49
|
-
> **Note:** nut-js native bindings require the Visual C++ Redistributable.
|
|
50
|
-
> Download from [Microsoft](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist) if not already installed.
|
|
51
|
-
|
|
52
|
-
> **Note (Key Locker):** The credential helper Key Locker uses is an unsigned executable, so on
|
|
53
|
-
> some machines Windows SmartScreen or antivirus may show an "unknown publisher" warning the
|
|
54
|
-
> first time it runs. This is expected — the helper ships with desktop-touch-mcp and runs locally
|
|
55
|
-
> on your machine; you can allow it to proceed. Code signing is planned for a future release.
|
|
56
|
-
|
|
57
|
-
---
|
|
58
|
-
|
|
59
|
-
## Installation
|
|
60
|
-
|
|
61
|
-
```bash
|
|
62
|
-
npx -y @harusame64/desktop-touch-mcp
|
|
63
|
-
```
|
|
64
|
-
|
|
65
|
-
The npm launcher resolves runtime strictly by npm package version. For package `X.Y.Z`, it fetches only GitHub Release tag `vX.Y.Z`, downloads `desktop-touch-mcp-windows.zip`, verifies its SHA256 digest, and only then expands it under `%USERPROFILE%\.desktop-touch-mcp`. Verified cached releases are reused on later runs.
|
|
66
|
-
|
|
67
|
-
Set `DESKTOP_TOUCH_MCP_HOME` to override the cache root directory.
|
|
68
|
-
|
|
69
|
-
> **On a shared or CI network?** The first run reads the GitHub Releases API to
|
|
70
|
-
> locate the runtime zip. The anonymous limit is 60 requests/hour per IP, which a
|
|
71
|
-
> shared public address (CI runners, office NAT) can exhaust before your download
|
|
72
|
-
> even starts. Set `GITHUB_TOKEN` (or `GH_TOKEN`) in the environment and the
|
|
73
|
-
> launcher authenticates the request, raising the limit to 5,000 requests/hour.
|
|
74
|
-
> No token is needed on an ordinary home connection.
|
|
75
|
-
|
|
76
|
-
> **Running the launcher from a source checkout?** A source build's
|
|
77
|
-
> `bin/launcher.js` carries a placeholder integrity hash (`sha256: "PENDING"`)
|
|
78
|
-
> instead of a finalized one. Rather than download and run an unverified runtime,
|
|
79
|
-
> the launcher fails closed — this guard stops an accidentally published or
|
|
80
|
-
> unfinalized launcher from silently starting unverified code. Published npm
|
|
81
|
-
> releases always ship a real SHA256, so end users never see this. If you are
|
|
82
|
-
> intentionally running the launcher from source, set
|
|
83
|
-
> `DESKTOP_TOUCH_MCP_ALLOW_UNVERIFIED=1` to skip integrity verification
|
|
84
|
-
> (development only).
|
|
85
|
-
|
|
86
|
-
### Register with Claude CLI
|
|
87
|
-
|
|
88
|
-
Add to `~/.claude.json` under `mcpServers`:
|
|
89
|
-
|
|
90
|
-
```json
|
|
91
|
-
{
|
|
92
|
-
"mcpServers": {
|
|
93
|
-
"desktop-touch": {
|
|
94
|
-
"type": "stdio",
|
|
95
|
-
"command": "npx",
|
|
96
|
-
"args": ["-y", "@harusame64/desktop-touch-mcp"]
|
|
97
|
-
}
|
|
98
|
-
}
|
|
99
|
-
}
|
|
100
|
-
```
|
|
101
|
-
|
|
102
|
-
**No system prompt needed.** The command reference is automatically injected into Claude via the MCP `initialize` response's `instructions` field.
|
|
103
|
-
|
|
104
|
-
### Register with other clients (HTTP mode)
|
|
105
|
-
|
|
106
|
-
Clients that require an HTTP endpoint (GPT Desktop, VS Code Copilot, Cursor, etc.) can use the built-in Streamable HTTP transport:
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
npx -y @harusame64/desktop-touch-mcp --http
|
|
110
|
-
# or with a custom port:
|
|
111
|
-
npx -y @harusame64/desktop-touch-mcp --http --port 8080
|
|
112
|
-
```
|
|
113
|
-
|
|
114
|
-
The server starts at `http://127.0.0.1:23847/mcp` (localhost only). Register the URL in your MCP client settings. A health check is available at `http://127.0.0.1:<port>/health`.
|
|
115
|
-
|
|
116
|
-
In HTTP mode the system tray icon shows the active URL and provides quick-copy and open-in-browser shortcuts.
|
|
117
|
-
|
|
118
|
-
### Development install
|
|
119
|
-
|
|
120
|
-
```bash
|
|
121
|
-
git clone https://github.com/Harusame64/desktop-touch-mcp.git
|
|
122
|
-
cd desktop-touch-mcp
|
|
123
|
-
npm install
|
|
124
|
-
```
|
|
125
|
-
|
|
126
|
-
Build after install:
|
|
127
|
-
|
|
128
|
-
```bash
|
|
129
|
-
npm run build
|
|
130
|
-
```
|
|
131
|
-
|
|
132
|
-
For a local checkout, register the built server directly:
|
|
133
|
-
|
|
134
|
-
```json
|
|
135
|
-
{
|
|
136
|
-
"mcpServers": {
|
|
137
|
-
"desktop-touch": {
|
|
138
|
-
"type": "stdio",
|
|
139
|
-
"command": "node",
|
|
140
|
-
"args": ["D:/path/to/desktop-touch-mcp/dist/index.js"]
|
|
141
|
-
}
|
|
142
|
-
}
|
|
143
|
-
}
|
|
144
|
-
```
|
|
145
|
-
|
|
146
|
-
> **Note:** Replace `D:/path/to/desktop-touch-mcp` with the actual path where you cloned this repository.
|
|
147
|
-
|
|
148
|
-
---
|
|
149
|
-
|
|
150
|
-
## Tools (32 Optimized Tools)
|
|
151
|
-
|
|
152
|
-
> 📖 **Full Reference**: [`docs/system-overview.md`](docs/system-overview.md) — Exhaustive guide on parameters, return schemas, and coordinate math.
|
|
153
|
-
|
|
154
|
-
### 🌐 World-Graph V2 (Primary Path)
|
|
155
|
-
| Tool | Description |
|
|
156
|
-
|---|---|
|
|
157
|
-
| `desktop_discover` | Observe the desktop. Returns interactive entities with leases (UIA, CDP, Terminal, Visual SoM). |
|
|
158
|
-
| `desktop_act` | Perform actions (click, type, drag, select) on entities via lease validation. Returns semantic diffs — plus an optional `roiCapture` (changed-region PNG + next-target preview) on visual-only targets. |
|
|
159
|
-
|
|
160
|
-
### 👁️ Observation & State
|
|
161
|
-
| Tool | Description |
|
|
162
|
-
|---|---|
|
|
163
|
-
| `desktop_state` | Lightweight check of focus, active window, cursor, and Auto-Perception attention signal. |
|
|
164
|
-
| `screenshot` | Multi-mode capture: `detail='text'` (UIA/OCR), `diffMode` (P-frame), `dotByDot` (1:1), and `background`. Returns a cheap `screenshot://by-ref/{id}` link to the saved image instead of inlining pixels every time. |
|
|
165
|
-
| `screenshot_query` / `screenshot_gc` | Inspect and prune the on-disk screenshot cache behind the by-ref links: `screenshot_query` lists saved captures without re-reading pixels; `screenshot_gc` reclaims space by retention policy (dry-run by default). |
|
|
166
|
-
| `workspace_snapshot` | Instant session orientation: all window thumbnails + UI summaries in one call. |
|
|
167
|
-
| `server_status` | Diagnostic check for native engine health and feature activation. |
|
|
168
|
-
|
|
169
|
-
### ⌨️ Input & Control
|
|
170
|
-
| Tool | Description |
|
|
171
|
-
|---|---|
|
|
172
|
-
| `keyboard` | Send keyboard input. Supports background input (WM_CHAR) and IME-safe clipboard bypass. |
|
|
173
|
-
| `mouse_click` / `mouse_drag` | Precision coordinate-based interaction with homing and force-focus protection. |
|
|
174
|
-
| `scroll` | Multi-strategy: `raw` (notches), `to_element`, `smart` (virtual lists), and `capture` (stitch). |
|
|
175
|
-
| `click_element` | Legacy UIA-based click by name/ID (fallback when entities are unavailable). |
|
|
176
|
-
|
|
177
|
-
### 🌐 Browser CDP (Chrome/Edge/Brave)
|
|
178
|
-
| Tool | Description |
|
|
179
|
-
|---|---|
|
|
180
|
-
| `browser_open` / `browser_navigate` | Idempotent debug-mode launch and reliable navigation. |
|
|
181
|
-
| `browser_click` / `browser_fill` / `browser_form` | High-level DOM interaction stable across repaints and framework re-renders. |
|
|
182
|
-
| `browser_eval` | Deep inspection via `js` (scripting), `dom` (HTML), and `appState` (SPA data extraction). |
|
|
183
|
-
| `browser_overview` / `browser_search` / `browser_locate` | Semantic discovery, grep-like DOM search, and pixel-accurate coordinate lookup. |
|
|
184
|
-
|
|
185
|
-
### 🛠️ Utilities & Workflow
|
|
186
|
-
| Tool | Description |
|
|
187
|
-
|---|---|
|
|
188
|
-
| `terminal` | Unified command execution: `run` (send + wait + read), `read` (OCR/UIA), and `send`. `run` completion modes: `quiet`, `pattern`, and `exit` (waits for the command to finish + returns its exit code — see [Terminal command completion](#terminal-command-completion-until)). |
|
|
189
|
-
| `wait_until` | Efficient server-side polling for window, focus, text, or URL state changes. |
|
|
190
|
-
| `window_dock` / `focus_window` | Window management: `pin` (always-on-top), `unpin`, `dock` (corner snap), and `focus`. |
|
|
191
|
-
| `workspace_launch` | Launch apps and auto-detect new HWNDs (supports localized titles). |
|
|
192
|
-
| `run_macro` | Batch up to 50 operations into a single round-trip for maximum efficiency. |
|
|
193
|
-
| `clipboard` / `notification_show` | System-level text exchange and user alerts. |
|
|
194
|
-
| `key_locker` | Manage credentials the terminal autofills for you (SSH key passphrases, sudo / login passwords). Secrets are entered once into the locker's own secure dialog and stored encrypted on this machine (Windows DPAPI); they are never shown to the assistant. `action='launch_console'` opens an autofill-capable console (returns a `paneId` to drive `ssh`/`sudo` into via `terminal`); `save` / `list` / `forget` / `set_policy` / `status` manage bindings. Autofill only fires in a console opened by `launch_console`. Disable with `DESKTOP_TOUCH_DISABLE_KEY_LOCKER=1`. |
|
|
195
|
-
|
|
196
|
-
### 📊 Office (Excel)
|
|
197
|
-
| Tool | Description |
|
|
198
|
-
|---|---|
|
|
199
|
-
| `excel` | Author and run Excel VBA macros via COM. `action='run_vba'` writes a macro into a managed Trusted Location and runs it; `action='check_access_vbom'` is a read-only preflight. Runs VBA where formula-only tools cannot. One-time setup: `node scripts/enable-access-vbom.mjs`. |
|
|
200
|
-
|
|
201
|
-
---
|
|
202
|
-
|
|
203
|
-
## Standard workflow (v1.0.0)
|
|
204
|
-
|
|
205
|
-
The v2 World-Graph surface (`desktop_discover` / `desktop_act`) is the recommended dispatch path. The four-call shape works for native apps, browsers, and terminals identically.
|
|
206
|
-
|
|
207
|
-
```
|
|
208
|
-
desktop_state → orient: focused window/element, modal, attention signal
|
|
209
|
-
desktop_discover → find actionable entities (returns lease + windows[])
|
|
210
|
-
desktop_act(lease, …) → act on entity (returns attention + post.perception)
|
|
211
|
-
desktop_state → confirm the world changed as expected
|
|
212
|
-
```
|
|
213
|
-
|
|
214
|
-
Clicking — priority order:
|
|
215
|
-
|
|
216
|
-
```
|
|
217
|
-
browser_click(selector) → Chrome / Edge (CDP, stable across repaints)
|
|
218
|
-
desktop_act(lease, action='click') → native / dialog / visual (entity-based; use after desktop_discover)
|
|
219
|
-
click_element(name | automationId) → native UIA fallback if desktop_act returns ok:false
|
|
220
|
-
mouse_click(x, y, origin?, scale?) → pixel last resort; origin+scale from dotByDot screenshots only
|
|
221
|
-
```
|
|
222
|
-
|
|
223
|
-
Recovery hints — read `response.attention` after every observation and `response.warnings[]` on `desktop_discover` / `desktop_act`. Common reasons:
|
|
224
|
-
|
|
225
|
-
- `lease_expired` / `lease_generation_mismatch` / `lease_digest_mismatch` / `entity_not_found` → re-call `desktop_discover`
|
|
226
|
-
- `modal_blocking` → `response.blockingElement` (when present) names the blocking modal; dismiss via `click_element(name=blockingElement.name)` then retry
|
|
227
|
-
- `entity_outside_viewport` → `scroll(action='to_element' | 'raw')`, then re-call `desktop_discover`
|
|
228
|
-
- `executor_failed` → fall back to `click_element` / `mouse_click` / `browser_click`
|
|
229
|
-
|
|
230
|
-
Lease lifecycle:
|
|
231
|
-
|
|
232
|
-
- Each `desktop_discover` response carries `softExpiresAtMs` (≈ 60 % of the TTL window). Past that timestamp the LLM should consider re-calling `desktop_discover` even though the lease is still technically valid — `lease.expiresAtMs` is the only correctness wall.
|
|
233
|
-
- TTL adapts to `view` mode (`action`/`explore`/`debug`), entity count, and response payload size. Cap is 60 s.
|
|
234
|
-
- Set `DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1` to fall back to the v1 tool surface (`get_windows` / `get_ui_elements` / `set_element_value`) for troubleshooting only — V2 is the recommended default.
|
|
235
|
-
|
|
236
|
-
---
|
|
237
|
-
|
|
238
|
-
## Terminal command completion (`until`)
|
|
239
|
-
|
|
240
|
-
`terminal(action='run')` sends a command, waits for it to complete, and reads the
|
|
241
|
-
output in one call. How it decides "complete" is controlled by `until`:
|
|
242
|
-
|
|
243
|
-
| Mode | Waits for | Best for |
|
|
244
|
-
|---|---|---|
|
|
245
|
-
| `quiet` (default) | output to fall silent for `quietMs` | short interactive commands |
|
|
246
|
-
| `pattern` | a string/regex you expect in the output | long commands with a known final marker |
|
|
247
|
-
| `exit` | the command to actually **finish** | when you need completion or the exit code |
|
|
248
|
-
|
|
249
|
-
> **Anchoring caveat (#384):** a command whose final line has no trailing newline
|
|
250
|
-
> glues the marker to the next prompt with no line boundary (`printf X` →
|
|
251
|
-
> `Xuser@host:~$`), so an end-anchored `pattern` (`X\s*\n` / `X$`) can never bind.
|
|
252
|
-
> For *completion* use `mode:'exit'`; for *content* matching use a bare marker
|
|
253
|
-
> (no `\n`/`$`). `mode:'pattern'` also accepts an optional `quietMs` settle
|
|
254
|
-
> fallback: `until:{mode:'pattern', pattern, quietMs:1000}` completes with
|
|
255
|
-
> `reason:'quiet'` (no `matchedPattern`) once output is stable for that long
|
|
256
|
-
> without a match — instead of hanging until `timeoutMs`. It is opt-in (omit
|
|
257
|
-
> `quietMs` to keep waiting for the pattern; long commands with mid-run silent
|
|
258
|
-
> gaps are unaffected).
|
|
259
|
-
|
|
260
|
-
### `until:{mode:'exit'}` — real completion + exit code
|
|
261
|
-
|
|
262
|
-
The heuristic modes can misfire on the common "append a sentinel" idiom
|
|
263
|
-
(`some-task; echo DONE` matched by `DONE`): the sentinel also shows up in the
|
|
264
|
-
**echoed command line**, and for multi-line commands there is no reliable way to
|
|
265
|
-
tell that echo apart from real output. `mode:'exit'` removes the guesswork — the
|
|
266
|
-
server appends its own completion marker whose *printed* form differs from its
|
|
267
|
-
*typed* form, so it never matches the echoed command (even for multi-line input),
|
|
268
|
-
and it returns the real process exit code:
|
|
269
|
-
|
|
270
|
-
```js
|
|
271
|
-
terminal({
|
|
272
|
-
action: 'run',
|
|
273
|
-
windowTitle: 'pwsh',
|
|
274
|
-
input: 'npm run build',
|
|
275
|
-
until: { mode: 'exit', shell: 'powershell' },
|
|
276
|
-
})
|
|
277
|
-
// → completion: { reason: 'exited', exitCode: 0, elapsedMs: … }
|
|
278
|
-
// output: just the command's real output (the injected marker is stripped)
|
|
279
|
-
```
|
|
280
|
-
|
|
281
|
-
- **Pass `shell` explicitly** (`'bash'` or `'powershell'`). `shell:'auto'` detects
|
|
282
|
-
the shell from the terminal window, but it cannot see a shell running *inside*
|
|
283
|
-
SSH or WSL — the window still looks like its local host — so for remote/nested
|
|
284
|
-
sessions pass the remote side's shell (`auto` otherwise warns and may pick the
|
|
285
|
-
outer shell). A window whose process is genuinely unidentifiable (e.g. Windows
|
|
286
|
-
Terminal) returns `ExitModeShellAmbiguous`.
|
|
287
|
-
- **First-class shells:** `bash` and `powershell`. `cmd.exe` is not supported yet
|
|
288
|
-
(`ExitModeShellUnsupported`).
|
|
289
|
-
- **Unsafe input is rejected up front** (`ExitModeUnsafeInput`) rather than
|
|
290
|
-
hanging: a command ending mid-construct (unterminated quote, here-doc, `$(…)`,
|
|
291
|
-
a trailing `\` or PowerShell backtick).
|
|
292
|
-
- Exit mode controls its own delivery, so delivery-shaping `sendOptions`
|
|
293
|
-
(`method` / `preferClipboard` / `pressEnter` / `chunkSize` / `pasteKey`) are
|
|
294
|
-
rejected with `InvalidArgs`; focus options remain accepted.
|
|
295
|
-
|
|
296
|
-
---
|
|
297
|
-
|
|
298
|
-
## Key Locker (terminal credential autofill)
|
|
299
|
-
|
|
300
|
-
Running `ssh user@host` or `sudo …` normally stops at a hidden password prompt an
|
|
301
|
-
assistant can't safely type into. Key Locker stores your SSH key passphrases and
|
|
302
|
-
sudo / login passwords encrypted on your machine (Windows DPAPI, current user) and
|
|
303
|
-
fills them in automatically when a bound command reaches its prompt. The secret is
|
|
304
|
-
typed once into the locker's own secure dialog — it is never shown to the assistant
|
|
305
|
-
and never travels through the MCP channel.
|
|
306
|
-
|
|
307
|
-
```js
|
|
308
|
-
// 1. Save the credential once — opens a secure dialog on your desktop
|
|
309
|
-
key_locker({ action:'save', uri:'ssh://user@host:22' })
|
|
310
|
-
|
|
311
|
-
// 2. Open an autofill-capable console (returns its paneId)
|
|
312
|
-
key_locker({ action:'launch_console' }) // → { paneId:'12345678', windowTitle:'…' }
|
|
313
|
-
|
|
314
|
-
// 3. Run the command through that pane — the password is filled at the prompt
|
|
315
|
-
terminal({ action:'send', paneId:'12345678', input:'ssh user@host' })
|
|
316
|
-
```
|
|
317
|
-
|
|
318
|
-
- **Autofill only fires in a console opened by `launch_console`** — a pre-existing
|
|
319
|
-
terminal is never autofilled. The console is a classic visible Windows console,
|
|
320
|
-
so you can watch it and take over at any prompt yourself.
|
|
321
|
-
- **Every autofill asks you to confirm by default**; opt a binding out with
|
|
322
|
-
`set_policy`. `list` / `status` / `forget` manage saved credentials.
|
|
323
|
-
- `terminal` `read` / `send` accept `paneId` as an alternative to `windowTitle` —
|
|
324
|
-
it targets that exact window even after an `ssh` login renames its title.
|
|
325
|
-
- Supported binding URIs: `ssh://user@host:22`, `sudo://host/user`,
|
|
326
|
-
`https-cred://host`, and SSH key passphrases (`sshkey:SHA256:…`). An `ssh`
|
|
327
|
-
save needs the host key already in `known_hosts` (connect to the host once first).
|
|
328
|
-
- Windows only. Disable the whole feature with `DESKTOP_TOUCH_DISABLE_KEY_LOCKER=1`.
|
|
329
|
-
The secure dialog is an unsigned helper executable — Windows SmartScreen may show
|
|
330
|
-
an "unknown publisher" warning on first run (see the note under
|
|
331
|
-
[Requirements](#requirements)).
|
|
332
|
-
|
|
333
|
-
---
|
|
334
|
-
|
|
335
|
-
## Browser CDP automation
|
|
336
|
-
|
|
337
|
-
For web automation, connect Chrome or Edge with the remote debugging port enabled — no Selenium or Playwright needed.
|
|
338
|
-
|
|
339
|
-
```bash
|
|
340
|
-
# Launch Chrome in CDP mode
|
|
341
|
-
chrome.exe --remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp
|
|
342
|
-
```
|
|
343
|
-
|
|
344
|
-
```
|
|
345
|
-
browser_open({launch:{}}) → spawn-if-needed Chrome in debug mode + list tabs (idempotent)
|
|
346
|
-
browser_open() → connect-only (fail if no CDP endpoint live)
|
|
347
|
-
browser_locate({selector:"#submit"}) → CSS selector → physical screen coords
|
|
348
|
-
browser_click({selector:"#submit"}) → find + click in one step (auto-focuses browser)
|
|
349
|
-
browser_eval({action:"js", expression:"document.title"}) → evaluate JS, returns result
|
|
350
|
-
browser_eval({action:"dom", selector:"#main", maxLength:5000}) → outerHTML, truncated to maxLength chars
|
|
351
|
-
browser_eval({action:"appState"}) → one-shot SPA state (Next/Nuxt/Remix/Apollo/GitHub react-app/Redux SSR)
|
|
352
|
-
browser_fill({selector:"#email", value:"user@example.com"}) → fill React/Vue/Svelte controlled input (state-safe)
|
|
353
|
-
browser_overview() → links/buttons/inputs + ARIA toggles + viewportPosition per element
|
|
354
|
-
browser_search({by:"text", pattern:"..."}) → grep DOM with confidence ranking
|
|
355
|
-
browser_navigate({url:"https://example.com"}) → navigate via CDP (no address bar interaction)
|
|
356
|
-
```
|
|
357
|
-
|
|
358
|
-
For chained calls in the same tab, pass `includeContext:false` to omit the activeTab/readyState annotation (~150 tok/call saved). Boolean / object params accept the LLM-friendly string spellings (`"true"`, `"{}"`).
|
|
359
|
-
|
|
360
|
-
Coordinates returned by `browser_locate` account for the browser chrome (tab strip + address bar height) and `devicePixelRatio`, so they can be passed directly to `mouse_click` without any scaling.
|
|
361
|
-
|
|
362
|
-
**Recommended web workflow:**
|
|
363
|
-
```
|
|
364
|
-
browser_open({launch:{}}) → browser_eval({action:"dom"}) → browser_locate(selector) → browser_click(selector)
|
|
365
|
-
```
|
|
366
|
-
|
|
367
|
-
---
|
|
368
|
-
|
|
369
|
-
## Auto-dock CLI on startup
|
|
370
|
-
|
|
371
|
-
Keep Claude CLI visible while operating other apps full-screen. Set env vars in your MCP config and the docked window auto-snaps into place every MCP startup.
|
|
372
|
-
|
|
373
|
-
```json
|
|
374
|
-
{
|
|
375
|
-
"mcpServers": {
|
|
376
|
-
"desktop-touch": {
|
|
377
|
-
"type": "stdio",
|
|
378
|
-
"command": "npx",
|
|
379
|
-
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
380
|
-
"env": {
|
|
381
|
-
"DESKTOP_TOUCH_DOCK_TITLE": "@parent",
|
|
382
|
-
"DESKTOP_TOUCH_DOCK_CORNER": "bottom-right",
|
|
383
|
-
"DESKTOP_TOUCH_DOCK_WIDTH": "480",
|
|
384
|
-
"DESKTOP_TOUCH_DOCK_HEIGHT": "360",
|
|
385
|
-
"DESKTOP_TOUCH_DOCK_PIN": "true"
|
|
386
|
-
}
|
|
387
|
-
}
|
|
388
|
-
}
|
|
389
|
-
}
|
|
390
|
-
```
|
|
391
|
-
|
|
392
|
-
| Env var | Default | Notes |
|
|
393
|
-
|---|---|---|
|
|
394
|
-
| `DESKTOP_TOUCH_DOCK_TITLE` | *(unset = off)* | `@parent` walks the MCP process tree to find the hosting terminal — immune to title / branch / project changes. Or use a literal substring. |
|
|
395
|
-
| `DESKTOP_TOUCH_DOCK_CORNER` | `bottom-right` | `top-left` / `top-right` / `bottom-left` / `bottom-right` |
|
|
396
|
-
| `DESKTOP_TOUCH_DOCK_WIDTH` / `HEIGHT` | `480` / `360` | px (`"480"`) or ratio of work area (`"25%"`) — 4K/8K auto-adapts |
|
|
397
|
-
| `DESKTOP_TOUCH_DOCK_PIN` | `true` | Always-on-top toggle |
|
|
398
|
-
| `DESKTOP_TOUCH_DOCK_MONITOR` | primary | Monitor id from `desktop_state({includeScreen:true})` |
|
|
399
|
-
| `DESKTOP_TOUCH_DOCK_SCALE_DPI` | `false` | If true, multiply px values by `dpi / 96` (opt-in per-monitor scaling) |
|
|
400
|
-
| `DESKTOP_TOUCH_DOCK_MARGIN` | `8` | Screen-edge padding (px) |
|
|
401
|
-
| `DESKTOP_TOUCH_DOCK_TIMEOUT_MS` | `5000` | Max wait for the target window to appear |
|
|
402
|
-
|
|
403
|
-
> **Input routing gotcha:** when a pinned window is active (e.g. Claude CLI), `keyboard(action='type')` / `keyboard(action='press')` send keys to it, **not** the app you wanted to type into. Always call `focus_window(title=...)` before keyboard operations, then verify `isActive=true` via `screenshot(detail='meta')`.
|
|
404
|
-
|
|
405
|
-
### Screenshot cache (by-ref storage)
|
|
406
|
-
|
|
407
|
-
`screenshot` and the other visual results return a cheap `screenshot://by-ref/{id}` link to an image saved on disk instead of inlining the pixels every time, so routine look-act-confirm loops cost far fewer tokens. The cache bounds itself automatically and `screenshot_query` / `screenshot_gc` let you inspect and prune it. Tune the storage with:
|
|
408
|
-
|
|
409
|
-
| Env var | Default | Notes |
|
|
410
|
-
|---|---|---|
|
|
411
|
-
| `DESKTOP_TOUCH_SCREENSHOTS_DIR` | *(per-user cache dir)* | Pin the cache to a specific folder. If the default folder can't be created or written (e.g. corporate policy blocking new folders under your profile), the server auto-probes this → the runtime dir → an OS temp folder and uses the first writable one instead of giving up on the cache. |
|
|
412
|
-
| `DESKTOP_TOUCH_SCREENSHOT_MAX_COUNT` | `200` | Keep at most this many captures in the cache. |
|
|
413
|
-
| `DESKTOP_TOUCH_SCREENSHOT_MAX_BYTES` | `256 MiB` | Cap the total cache size on disk. |
|
|
414
|
-
| `DESKTOP_TOUCH_SCREENSHOT_MAX_AGE_MS` | *(off)* | Drop captures older than this many milliseconds (opt-in). |
|
|
415
|
-
| `DESKTOP_TOUCH_SCREENSHOT_AUTOPRUNE` | `on` | Auto-trim the cache as new captures are saved. Set `0` to disable. |
|
|
416
|
-
| `DESKTOP_TOUCH_SCREENSHOT_MIN_EVICT_AGE_MS` | `60000` | Never auto-evict a capture younger than this (ms), so a by-ref link you were just handed survives long enough to open even when another AI/process on the same PC is also capturing. `0` disables. |
|
|
417
|
-
|
|
418
|
-
### Auto Perception (always-on)
|
|
419
|
-
|
|
420
|
-
Phase 4 privatizes the explicit `perception_*` tool family — the v0.12 Auto
|
|
421
|
-
Perception layer attaches an `attention` signal to every `desktop_state` and
|
|
422
|
-
`desktop_act` response automatically. Action tools also auto-guard when given
|
|
423
|
-
a `windowTitle`. There is no longer a need to register / read / forget lenses
|
|
424
|
-
manually.
|
|
425
|
-
|
|
426
|
-
```
|
|
427
|
-
# desktop_state always returns the attention signal
|
|
428
|
-
desktop_state() → {focusedWindow, focusedElement, modal, attention:"ok", ...}
|
|
429
|
-
|
|
430
|
-
# Action tools auto-guard when windowTitle is given:
|
|
431
|
-
keyboard({action:"type", text:"hello", windowTitle:"Notepad"})
|
|
432
|
-
→ post.perception:{status:"ok"} // unsafe input blocked if guards fail
|
|
433
|
-
|
|
434
|
-
# When attention is dirty / stale / settling, refresh with desktop_state:
|
|
435
|
-
desktop_state() // re-evaluates attention via Auto Perception
|
|
436
|
-
```
|
|
437
|
-
|
|
438
|
-
For advanced pinned-target workflows, the `lensId` parameter remains on action
|
|
439
|
-
tools (`keyboard`, `mouse_click`, `mouse_drag`, `click_element`,
|
|
440
|
-
`browser_click`, `browser_navigate`, `browser_eval`, `desktop_act`). Omit
|
|
441
|
-
`lensId` for the normal Auto Perception path. The underlying registry, hot
|
|
442
|
-
target cache, and sensor loop are unchanged; only the explicit
|
|
443
|
-
`perception_register / perception_read / perception_forget / perception_list`
|
|
444
|
-
tools were retired.
|
|
445
|
-
|
|
446
|
-
---
|
|
447
|
-
|
|
448
|
-
## Mouse homing correction
|
|
449
|
-
|
|
450
|
-
When Claude calls `screenshot(detail='text')` to read coordinates and then `mouse_click` seconds later, the target window may have moved. The homing system corrects this automatically.
|
|
451
|
-
|
|
452
|
-
| Tier | How to enable | Latency | What it does |
|
|
453
|
-
|------|--------------|---------|--------------|
|
|
454
|
-
| 1 | Always-on (if cache exists) | <1ms | Applies (dx, dy) offset when window moved |
|
|
455
|
-
| 2 | Pass `windowTitle` hint | ~100ms | Auto-focuses window if it went behind another |
|
|
456
|
-
| 3 | Pass `elementName`/`elementId` + `windowTitle` | 1–3s | UIA re-query for fresh coords on resize |
|
|
457
|
-
|
|
458
|
-
```
|
|
459
|
-
# Tier 1 only (automatic)
|
|
460
|
-
mouse_click(x=500, y=300)
|
|
461
|
-
|
|
462
|
-
# Tier 1 + 2: also bring window to front if hidden
|
|
463
|
-
mouse_click(x=500, y=300, windowTitle="Notepad")
|
|
464
|
-
|
|
465
|
-
# Tier 1 + 2 + 3: also re-query UIA if window resized
|
|
466
|
-
mouse_click(x=500, y=300, windowTitle="Notepad", elementName="Save")
|
|
467
|
-
|
|
468
|
-
# Traction control OFF — no correction
|
|
469
|
-
mouse_click(x=500, y=300, homing=false)
|
|
470
|
-
```
|
|
471
|
-
|
|
472
|
-
The `homing` parameter is available on `mouse_click`, `mouse_drag`, and `scroll`. The cache is updated automatically on every `screenshot()`, `desktop_discover()`, `focus_window()`, and `workspace_snapshot()` call.
|
|
473
|
-
|
|
474
|
-
### `mouse_click` image-local coords (origin + scale)
|
|
475
|
-
|
|
476
|
-
When you take a `dotByDot` screenshot with `dotByDotMaxDimension`, the response prints the `origin` and `scale` values. Instead of computing screen coords manually, copy them into `mouse_click`:
|
|
477
|
-
|
|
478
|
-
```
|
|
479
|
-
# Screenshot response:
|
|
480
|
-
# origin: (0, 120) | scale: 0.6667
|
|
481
|
-
# To click image pixel (ix, iy): mouse_click(x=ix, y=iy, origin={x:0, y:120}, scale=0.6667)
|
|
482
|
-
|
|
483
|
-
mouse_click(x=640, y=300, origin={x:0, y:120}, scale=0.6667, windowTitle="Chrome")
|
|
484
|
-
# Server converts: screen = (0 + 640/0.6667, 120 + 300/0.6667) = (960, 570)
|
|
485
|
-
```
|
|
486
|
-
|
|
487
|
-
This eliminates a whole class of off-by-one and scale bugs. Without origin/scale, `x`/`y` remain absolute screen pixels (unchanged behavior).
|
|
488
|
-
|
|
489
|
-
---
|
|
490
|
-
|
|
491
|
-
## `screenshot` key parameters
|
|
492
|
-
|
|
493
|
-
```
|
|
494
|
-
detail="image" — PNG/WebP pixels (default)
|
|
495
|
-
detail="text" — UIA element JSON + clickAt coords (no image, ~100–300 tok)
|
|
496
|
-
detail="meta" — Title + region only (cheapest, ~20 tok/window)
|
|
497
|
-
dotByDot=true — 1:1 WebP; image_px + origin = screen_px
|
|
498
|
-
dotByDotMaxDimension=N — cap longest edge (response includes scale for coord math)
|
|
499
|
-
grayscale=true — ~50% smaller for text-heavy captures (code/AWS console)
|
|
500
|
-
region={x,y,w,h} — with windowTitle: window-local coords (exclude browser chrome)
|
|
501
|
-
without: virtual screen coords
|
|
502
|
-
diffMode=true — I-frame first call, P-frame (changed windows only) after (~160 tok)
|
|
503
|
-
ocrFallback="auto" — detail='text' auto-fires Windows OCR on uiaSparse or empty
|
|
504
|
-
```
|
|
505
|
-
|
|
506
|
-
**Recommended Chrome combo** (50–70% data reduction):
|
|
507
|
-
```
|
|
508
|
-
screenshot(windowTitle="Chrome",
|
|
509
|
-
dotByDot=true, dotByDotMaxDimension=1280, grayscale=true,
|
|
510
|
-
region={x:0, y:120, width:1920, height:900}) # skip browser chrome
|
|
511
|
-
```
|
|
512
|
-
|
|
513
|
-
**Recommended workflow:**
|
|
514
|
-
```
|
|
515
|
-
workspace_snapshot() → full orientation (resets diff buffer)
|
|
516
|
-
screenshot(detail="text", windowTitle=X) → get actionable[].clickAt coords
|
|
517
|
-
mouse_click(x, y) → click directly, no math needed
|
|
518
|
-
screenshot(diffMode=true) → check only what changed (~160 tok)
|
|
519
|
-
```
|
|
520
|
-
|
|
521
|
-
---
|
|
522
|
-
|
|
523
|
-
## Security
|
|
524
|
-
|
|
525
|
-
### Emergency stop (Failsafe)
|
|
526
|
-
|
|
527
|
-
**Move the mouse to the top-left corner of the screen (within 10px of 0,0) to immediately terminate the MCP server.**
|
|
528
|
-
|
|
529
|
-
- **Per-tool check**: `checkFailsafe()` runs before every tool handler
|
|
530
|
-
- **Background monitor**: 500ms polling as a backup for long-running operations
|
|
531
|
-
- Trigger radius: 10px
|
|
532
|
-
|
|
533
|
-
### Blocked operations
|
|
534
|
-
|
|
535
|
-
**`workspace_launch` blocklist:**
|
|
536
|
-
`cmd.exe`, `powershell.exe`, `pwsh.exe`, `wscript.exe`, `cscript.exe`, `mshta.exe`, `regsvr32.exe`, `rundll32.exe`, `msiexec.exe`, `bash.exe`, `wsl.exe` are blocked.
|
|
537
|
-
Script extensions (`.bat`, `.ps1`, `.vbs`, etc.) are rejected. Arguments containing `;`, `&`, `|`, `` ` ``, `$(`, `${` are also rejected.
|
|
538
|
-
|
|
539
|
-
**`keyboard(action='press')` blocklist:**
|
|
540
|
-
`Win+R` (Run dialog), `Win+X` (admin menu), `Win+S` (search), `Win+L` (lock screen) are blocked.
|
|
541
|
-
|
|
542
|
-
### PowerShell injection protection
|
|
543
|
-
|
|
544
|
-
All `-like` patterns in the UIA bridge PowerShell fallback path are sanitized with `escapeLike()`, which escapes wildcard characters (`*`, `?`, `[`, `]`) before they reach PowerShell. When the Rust native engine is active, PowerShell is not invoked for UIA operations.
|
|
545
|
-
|
|
546
|
-
### Allowlist for `workspace_launch`
|
|
547
|
-
|
|
548
|
-
Shell interpreters are blocked by default. To allow specific executables, create an allowlist file:
|
|
549
|
-
|
|
550
|
-
**File locations (searched in order):**
|
|
551
|
-
1. Path in `DESKTOP_TOUCH_ALLOWLIST` environment variable
|
|
552
|
-
2. `~/.claude/desktop-touch-allowlist.json`
|
|
553
|
-
3. `desktop-touch-allowlist.json` in the server's working directory
|
|
554
|
-
|
|
555
|
-
**Format:**
|
|
556
|
-
```json
|
|
557
|
-
{
|
|
558
|
-
"allowedExecutables": [
|
|
559
|
-
"pwsh.exe",
|
|
560
|
-
"C:\\Tools\\myapp.exe"
|
|
561
|
-
]
|
|
562
|
-
}
|
|
563
|
-
```
|
|
564
|
-
|
|
565
|
-
Changes take effect immediately — no restart needed.
|
|
566
|
-
|
|
567
|
-
---
|
|
568
|
-
|
|
569
|
-
## Mouse movement speed
|
|
570
|
-
|
|
571
|
-
All mouse tools (`mouse_click`, `mouse_drag`, `scroll`) accept an optional `speed` parameter:
|
|
572
|
-
|
|
573
|
-
| Value | Behavior |
|
|
574
|
-
|---|---|
|
|
575
|
-
| Omitted | Uses the configured default (see below) |
|
|
576
|
-
| `0` | Instant teleport — `setPosition()`, no animation |
|
|
577
|
-
| `1–N` | Animated movement at N px/sec |
|
|
578
|
-
|
|
579
|
-
**Default speed** is 1500 px/sec. Change it permanently via the `DESKTOP_TOUCH_MOUSE_SPEED` environment variable:
|
|
580
|
-
|
|
581
|
-
```json
|
|
582
|
-
{
|
|
583
|
-
"mcpServers": {
|
|
584
|
-
"desktop-touch": {
|
|
585
|
-
"type": "stdio",
|
|
586
|
-
"command": "npx",
|
|
587
|
-
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
588
|
-
"env": {
|
|
589
|
-
"DESKTOP_TOUCH_MOUSE_SPEED": "3000"
|
|
590
|
-
}
|
|
591
|
-
}
|
|
592
|
-
}
|
|
593
|
-
}
|
|
594
|
-
```
|
|
595
|
-
|
|
596
|
-
Common values: `0` = teleport, `1500` = default gentle, `3000` = fast, `5000` = very fast.
|
|
597
|
-
|
|
598
|
-
---
|
|
599
|
-
|
|
600
|
-
## Force-Focus (AttachThreadInput)
|
|
601
|
-
|
|
602
|
-
Windows foreground-stealing protection can prevent `SetForegroundWindow` from succeeding when another window (such as a pinned Claude CLI) is in the foreground. This causes subsequent keystrokes or clicks to land in the wrong window — a silent failure.
|
|
603
|
-
|
|
604
|
-
`mouse_click`, `keyboard(action='type')`, `keyboard(action='press')`, and `terminal(action='send')` all accept a `forceFocus` parameter that bypasses this protection using `AttachThreadInput`:
|
|
605
|
-
|
|
606
|
-
```json
|
|
607
|
-
{
|
|
608
|
-
"name": "mouse_click",
|
|
609
|
-
"arguments": {
|
|
610
|
-
"x": 500,
|
|
611
|
-
"y": 300,
|
|
612
|
-
"windowTitle": "Google Chrome",
|
|
613
|
-
"forceFocus": true
|
|
614
|
-
}
|
|
615
|
-
}
|
|
616
|
-
```
|
|
617
|
-
|
|
618
|
-
If the force attempt is refused despite `AttachThreadInput`, the response is `ok:false` with `code: "ForegroundRestricted"` (issue #202 unification — same shape as `focus_window`, `keyboard`, `terminal_send`, `mouse_click`). The action itself is **suppressed** so the keystrokes / click never land on the wrong window. Recover via `focus_window`'s auto-escalate ladder before retrying. The legacy `hints.warnings: ["ForceFocusRefused"]` shape is no longer emitted.
|
|
619
|
-
|
|
620
|
-
**Global default via environment variable:**
|
|
621
|
-
|
|
622
|
-
```json
|
|
623
|
-
{
|
|
624
|
-
"mcpServers": {
|
|
625
|
-
"desktop-touch": {
|
|
626
|
-
"env": {
|
|
627
|
-
"DESKTOP_TOUCH_FORCE_FOCUS": "1"
|
|
628
|
-
}
|
|
629
|
-
}
|
|
630
|
-
}
|
|
631
|
-
}
|
|
632
|
-
```
|
|
633
|
-
|
|
634
|
-
Setting `DESKTOP_TOUCH_FORCE_FOCUS=1` makes `forceFocus: true` the default for all four tools without changing each call.
|
|
635
|
-
|
|
636
|
-
**Known tradeoffs:**
|
|
637
|
-
|
|
638
|
-
- During the ~10ms `AttachThreadInput` window, key state and mouse capture are shared between the two threads. In rapid macro sequences this can cause a race condition (rare in practice).
|
|
639
|
-
- Disable `forceFocus` (or unset the env var) when the user is manually operating another app to avoid unexpected focus shifts.
|
|
640
|
-
|
|
641
|
-
---
|
|
642
|
-
|
|
643
|
-
## Auto Guard
|
|
644
|
-
|
|
645
|
-
Action tools (`mouse_click`, `mouse_drag`, `keyboard(action='type'/'press')`, `click_element`, `desktop_act`, `browser_click`, `browser_navigate`) automatically guard each action when you pass `windowTitle` / `tabId`:
|
|
646
|
-
|
|
647
|
-
- Verifies target window identity (process restart / HWND replacement detected)
|
|
648
|
-
- Confirms click coordinates are inside the target window rect
|
|
649
|
-
- Returns `post.perception.status` on every response — including failures — so the LLM can recover without a screenshot
|
|
650
|
-
|
|
651
|
-
**Disabling auto guard** — set `DESKTOP_TOUCH_AUTO_GUARD=0` to restore v0.11.12 behavior (no auto guard):
|
|
652
|
-
|
|
653
|
-
```json
|
|
654
|
-
{
|
|
655
|
-
"mcpServers": {
|
|
656
|
-
"desktop-touch": {
|
|
657
|
-
"type": "stdio",
|
|
658
|
-
"command": "npx",
|
|
659
|
-
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
660
|
-
"env": {
|
|
661
|
-
"DESKTOP_TOUCH_AUTO_GUARD": "0"
|
|
662
|
-
}
|
|
663
|
-
}
|
|
664
|
-
}
|
|
665
|
-
}
|
|
666
|
-
```
|
|
667
|
-
|
|
668
|
-
When auto guard is enabled (default), `post.perception.status` will be one of:
|
|
669
|
-
|
|
670
|
-
| Status | Meaning |
|
|
671
|
-
|---|---|
|
|
672
|
-
| `ok` | Guard passed — target verified |
|
|
673
|
-
| `unguarded` | `windowTitle` not provided; action ran without guard |
|
|
674
|
-
| `target_not_found` | No window matched the given title |
|
|
675
|
-
| `ambiguous_target` | Multiple windows matched; use a more specific title |
|
|
676
|
-
| `identity_changed` | Window was replaced (process restart / HWND change) |
|
|
677
|
-
| `unsafe_coordinates` | Click coordinates are outside the target window rect |
|
|
678
|
-
| `needs_escalation` | Use `browser_click` or specify `windowTitle` |
|
|
679
|
-
|
|
680
|
-
When `unsafe_coordinates` or `identity_changed` is returned, the response may include a `suggestedFix.fixId`. Pass that `fixId` to the relevant tool call to approve the recovery:
|
|
681
|
-
|
|
682
|
-
```json
|
|
683
|
-
{ "name": "mouse_click", "arguments": { "fixId": "fix-..." } }
|
|
684
|
-
{ "name": "keyboard(action='type')", "arguments": { "fixId": "fix-...", "text": "hello" } }
|
|
685
|
-
{ "name": "click_element", "arguments": { "fixId": "fix-..." } }
|
|
686
|
-
{ "name": "browser_click", "arguments": { "fixId": "fix-..." } }
|
|
687
|
-
```
|
|
688
|
-
|
|
689
|
-
The fix is one-shot and expires in 15 seconds. The server revalidates the target process identity before executing.
|
|
690
|
-
|
|
691
|
-
---
|
|
692
|
-
|
|
693
|
-
## Advanced response options
|
|
694
|
-
|
|
695
|
-
### browser_eval Structured Mode
|
|
696
|
-
|
|
697
|
-
Pass `withPerception: true` to receive a structured JSON response with `post.perception` instead of raw text:
|
|
698
|
-
|
|
699
|
-
```json
|
|
700
|
-
{ "name": "browser_eval", "arguments": { "expression": "document.title", "withPerception": true } }
|
|
701
|
-
```
|
|
702
|
-
|
|
703
|
-
Returns `{ ok: true, result: "...", post: { perception: { status: "ok", ... } } }`.
|
|
704
|
-
|
|
705
|
-
### mouse_drag Cross-Window Guard
|
|
706
|
-
|
|
707
|
-
`mouse_drag` now guards both start and end coordinates. Drags that cross window boundaries (or reach the desktop wallpaper) are blocked by default. To allow intentional cross-window or range-selection drags:
|
|
708
|
-
|
|
709
|
-
```json
|
|
710
|
-
{ "name": "mouse_drag", "arguments": { "startX": 100, "startY": 100, "endX": 900, "endY": 900, "allowCrossWindowDrag": true } }
|
|
711
|
-
```
|
|
712
|
-
|
|
713
|
-
---
|
|
714
|
-
|
|
715
|
-
## Performance (v0.15 — Rust Native Engine)
|
|
716
|
-
|
|
717
|
-
The Rust native engine (`@harusame64/desktop-touch-engine`) replaces PowerShell process spawning with direct COM calls over a persistent MTA thread. It loads automatically as a `.node` addon — no configuration needed.
|
|
718
|
-
|
|
719
|
-
### UIA Benchmark (vs PowerShell baseline)
|
|
720
|
-
|
|
721
|
-
| Function | Rust Native | PowerShell | Speedup |
|
|
722
|
-
|---|---|---|---|
|
|
723
|
-
| `getFocusedElement` | **2.2 ms** | 366 ms | **163.9×** |
|
|
724
|
-
| `getUiElements` (Explorer, ~60 elements) | **106.5 ms** | 346 ms | **3.3×** |
|
|
725
|
-
| **Weighted average** | | | **~82×** |
|
|
726
|
-
|
|
727
|
-
### Image Diff Benchmark (SSE2 SIMD)
|
|
728
|
-
|
|
729
|
-
| Function | Rust (SSE2) | TypeScript | Speedup |
|
|
730
|
-
|---|---|---|---|
|
|
731
|
-
| `computeChangeFraction` (1920×1080) | **0.26 ms** | 3.8 ms | **~15×** |
|
|
732
|
-
| `dHash` (perceptual hash) | **0.09 ms** | 1.2 ms | **~13×** |
|
|
733
|
-
|
|
734
|
-
### Architecture
|
|
735
|
-
|
|
736
|
-
```
|
|
737
|
-
Claude CLI / MCP Client
|
|
738
|
-
│ stdio or HTTP (MCP protocol)
|
|
739
|
-
▼
|
|
740
|
-
desktop-touch-mcp (TypeScript)
|
|
741
|
-
│
|
|
742
|
-
├── Rust Native Engine (.node addon) ← NEW in v0.15
|
|
743
|
-
│ ├── UIA: 13 functions via napi-rs + windows-rs 0.62
|
|
744
|
-
│ │ └── Dedicated COM thread (MTA) + batch BFS algorithm
|
|
745
|
-
│ └── Image: SSE2 SIMD pixel diff + perceptual hashing
|
|
746
|
-
│
|
|
747
|
-
└── PowerShell Fallback (automatic)
|
|
748
|
-
└── Activates transparently if .node is unavailable
|
|
749
|
-
```
|
|
750
|
-
|
|
751
|
-
### Why `getUiElements` is 3.3× (not 160×)
|
|
752
|
-
|
|
753
|
-
The 160× speedup on `getFocusedElement` comes from eliminating PowerShell process startup (~200 ms) and .NET assembly loading. For `getUiElements`, the bottleneck shifts to the **UIA provider** inside the target application (e.g., Explorer) — it must enumerate its UI tree regardless of who asks. The Rust engine uses a **batch BFS algorithm** (`FindAllBuildCache` + `TreeScope_Children`) that minimizes cross-process RPC calls and supports `maxElements` early exit, making it dramatically faster on large trees (VS Code, browsers with 1000+ elements).
|
|
754
|
-
|
|
755
|
-
---
|
|
756
|
-
|
|
757
|
-
## UI Operating Layer (V2)
|
|
758
|
-
|
|
759
|
-
> **Status: Default ON since v0.17.** `desktop_discover` and `desktop_act` are available out of the box.
|
|
760
|
-
|
|
761
|
-
V2 introduces two new tools that replace coordinate-based clicking with entity-based interaction:
|
|
762
|
-
|
|
763
|
-
| Tool | Description |
|
|
764
|
-
|---|---|
|
|
765
|
-
| `desktop_discover` | Observe a window or browser tab. Returns interactive entities with leases — no raw screen coordinates. Supports UIA (native), CDP (browser), terminal, and visual GPU lanes. |
|
|
766
|
-
| `desktop_act` | Interact with an entity returned by `desktop_discover`. Validates the lease before executing. Returns a semantic diff (`entity_disappeared`, `modal_appeared`, `focus_shifted`, …). On visual-only targets a successful act can bundle a `roiCapture` (a PNG crop of the changed region + a lease-less next-target preview) so you confirm the result and find the next target in one call — controlled by `returnCapture` (`on-change`, the default on a visible change; `never` to suppress; `always` to force). |
|
|
767
|
-
|
|
768
|
-
### Clicking — priority order
|
|
769
|
-
|
|
770
|
-
When multiple tools could perform the same click, prefer them in this order:
|
|
771
|
-
|
|
772
|
-
1. `browser_click(selector)` — Chrome / Edge over CDP (stable across repaints)
|
|
773
|
-
2. `desktop_act(lease)` — native windows, dialogs, visual-only targets (entity-based; use after `desktop_discover`)
|
|
774
|
-
3. `click_element(name | automationId)` — native UIA fallback when `desktop_act` returns `ok:false`
|
|
775
|
-
4. `mouse_click(x, y)` — pixel-level last resort (`origin` + `scale` from `dotByDot` screenshots only)
|
|
776
|
-
|
|
777
|
-
### Disabling V2 (kill switch)
|
|
778
|
-
|
|
779
|
-
To hide `desktop_discover` / `desktop_act` from the tool catalog, add the disable flag and restart:
|
|
780
|
-
|
|
781
|
-
```json
|
|
782
|
-
{
|
|
783
|
-
"mcpServers": {
|
|
784
|
-
"desktop-touch": {
|
|
785
|
-
"type": "stdio",
|
|
786
|
-
"command": "npx",
|
|
787
|
-
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
788
|
-
"env": {
|
|
789
|
-
"DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2": "1"
|
|
790
|
-
}
|
|
791
|
-
}
|
|
792
|
-
}
|
|
793
|
-
}
|
|
794
|
-
```
|
|
795
|
-
|
|
796
|
-
All V1 tools continue to work without interruption — no reinstall required. Remove the env entry and restart to re-enable.
|
|
797
|
-
|
|
798
|
-
Flag semantics (exact-match: only the literal string `"1"` counts):
|
|
799
|
-
|
|
800
|
-
| `DISABLE_FUKUWARAI_V2` | V2 state |
|
|
801
|
-
|---|---|
|
|
802
|
-
| unset / not `"1"` | **ON** (default) |
|
|
803
|
-
| `"1"` | **OFF** (kill switch) |
|
|
804
|
-
|
|
805
|
-
### Removed: `DESKTOP_TOUCH_ENABLE_FUKUWARAI_V2`
|
|
806
|
-
|
|
807
|
-
This was the opt-in switch in v0.16.x. V2 is on by default since v0.17, so the flag no longer has any effect and is safe to delete from your config. To turn V2 off, set `DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1`.
|
|
808
|
-
|
|
809
|
-
### Recovery when V2 fails
|
|
810
|
-
|
|
811
|
-
If `desktop_act` returns `ok: false`, read `reason` and follow the built-in recovery hints in the tool description. Common paths:
|
|
812
|
-
|
|
813
|
-
- `lease_expired` / `*_mismatch` / `entity_not_found` → re-call `desktop_discover`
|
|
814
|
-
- `modal_blocking` → `response.blockingElement` (when present) carries `{ name, role, automationId? }`; dismiss with `click_element(name=blockingElement.name)`, then retry
|
|
815
|
-
- `entity_outside_viewport` → `scroll` / `scroll(action='to_element')`, then re-call `desktop_discover`
|
|
816
|
-
- `executor_failed` → fall back to `click_element` / `mouse_click` / `browser_click`
|
|
817
|
-
|
|
818
|
-
For `desktop_discover` warnings (`visual_provider_unavailable`, `visual_provider_warming`, `cdp_provider_failed`, …), the coordinate-based tools (`screenshot(detail='text')`, `click_element`, `mouse_click`, `terminal`, …) remain available as an escape hatch.
|
|
819
|
-
|
|
820
|
-
---
|
|
821
|
-
|
|
822
|
-
## Known limitations
|
|
823
|
-
|
|
824
|
-
| Limitation | Detail | Workaround |
|
|
825
|
-
|---|---|---|
|
|
826
|
-
| Games / video players may return black or hang in PrintWindow capture | DirectX fullscreen apps may not redraw under `PW_RENDERFULLCONTENT`. Window-targeted `screenshot(detail='image')` already falls back to BitBlt automatically when PrintWindow returns no data or an all-black + zero-variance frame, but DirectX surfaces that hang the call don't surface as fallback. | Retry with `screenshot({mode:'background', fullContent:false})` to switch to the legacy PrintWindow flag; if still black, the BitBlt fallback path (default `mode='normal'`) will at least return the on-screen rect — `hints.captureFallbackReason` will say `printwindow-all-black` |
|
|
827
|
-
| UIA call overhead | ~2 ms (focus) / ~100 ms (tree) via Rust native engine; ~300 ms via PowerShell fallback | Rust engine loads automatically; `workspace_snapshot` uses a 2 s timeout internally |
|
|
828
|
-
| Chrome / WinUI3 UIA elements are empty | Chromium exposes only limited UIA | `screenshot(detail='text')` auto-detects Chromium and falls back to Windows OCR (`hints.chromiumGuard=true`). For richer DOM access use `browser_open` + `browser_locate` |
|
|
829
|
-
| Chromium title-regex misses when sites rewrite `document.title` | Guard relies on the ` - Google Chrome` suffix being present; some sites push it off the end of a long title | Title is treated as plain Chrome (UIA runs). OCR path is still reachable via `ocrFallback='always'` or when UIA returns `<5` elements (`uiaSparse`) |
|
|
830
|
-
| `browser_*` CDP tools need Chrome launched with `--remote-debugging-port` | If Chrome is already running on the default profile without the flag, `browser_open` fails. The CDP E2E suite (`tests/e2e/browser-cdp.test.ts`) will also fail in that state | Close Chrome first, then `browser_open({launch:{}})` will relaunch it in debug mode, or start Chrome manually with `--remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp` |
|
|
831
|
-
| Layer buffer TTL | Buffer auto-clears after 90s of inactivity → next `diffMode` becomes an I-frame | After long waits, call `workspace_snapshot` to explicitly reset the buffer |
|
|
832
|
-
| `keyboard(action='type')` / `keyboard(action='press')` follow focus | When `window_dock(action='dock')(pin=true)` keeps another window on top (e.g. Claude CLI), keystrokes may be absorbed by that window | Call `focus_window(title=...)` first and verify `isActive=true` via `screenshot(detail='meta')` before sending keys |
|
|
833
|
-
| `keyboard(action='type')` em-dash / smart quotes in Chrome/Edge | Non-ASCII punctuation (em-dash `—`, en-dash `–`, smart quotes `"" ''`) can be intercepted as keyboard accelerators, shifting focus to the address bar | Always use `use_clipboard=true` when the text contains such characters |
|
|
834
|
-
| `browser_eval(action='js')` on React / Vue / Svelte inputs | Setting `element.value = ...` or dispatching synthetic events does not update the framework's internal state | Use `browser_fill(selector, value)` — it uses native prototype setter + InputEvent which does update React/Vue/Svelte state |
|
|
835
|
-
|
|
836
|
-
---
|
|
837
|
-
|
|
838
|
-
## Token cost reference
|
|
839
|
-
|
|
840
|
-
| Mode | Tokens | Use case |
|
|
841
|
-
|---|---|---|
|
|
842
|
-
| `screenshot` (768px PNG) | ~443 tok | General visual check |
|
|
843
|
-
| `screenshot(dotByDot=true)` window | ~800 tok | Precise clicking (no coordinate math) |
|
|
844
|
-
| `screenshot(diffMode=true)` | ~160 tok | Post-action diff |
|
|
845
|
-
| `screenshot(detail="text")` | ~100–300 tok | UI interaction (no image) |
|
|
846
|
-
| `workspace_snapshot` | ~2000 tok | Full session orientation |
|
|
847
|
-
|
|
848
|
-
---
|
|
849
|
-
|
|
850
|
-
## 🚀
|
|
851
|
-
|
|
852
|
-
This project just passed **
|
|
853
|
-
tried an experimental desktop-automation MCP server, filed issues, opened PRs,
|
|
854
|
-
and shared what broke. Every bug report made the next release better.
|
|
855
|
-
Thank you for building with me!
|
|
856
|
-
|
|
857
|
-
---
|
|
858
|
-
|
|
859
|
-
## License
|
|
860
|
-
|
|
861
|
-
MIT
|
|
1
|
+
# desktop-touch-mcp
|
|
2
|
+
|
|
3
|
+
[](https://glama.ai/mcp/servers/Harusame64/desktop-touch-mcp)
|
|
4
|
+
|
|
5
|
+
[日本語](README.ja.md)
|
|
6
|
+
|
|
7
|
+
> **Computer-use MCP server for Windows.** Lets Claude, Cursor, or any MCP client see and operate your Windows 10/11 desktop — screenshots, UI Automation, Chrome CDP, keyboard / mouse, terminal — with **semantic discover-then-act targeting** that avoids pixel-coordinate guessing, and **per-action perception guards** that catch wrong-window typing before it happens.
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
npx -y @harusame64/desktop-touch-mcp
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
32 tools, native Rust engine (UIA in 2 ms), zero-config PowerShell fallback, full CJK support, MIT licensed. Add the snippet above to your Claude / Cursor / VS Code Copilot config and Claude can drive Notepad, Excel, Chrome, Windows Terminal, and any other app on your machine.
|
|
14
|
+
|
|
15
|
+
> **Why this over pixel-clicking?** Two ideas run through every tool: **discover-then-act** — `desktop_discover` returns interactive entities with short-lived leases instead of raw coordinates, so `desktop_act` operates on *what* you mean, not *where* it was — and **per-action perception guards** that verify the target window's identity and bounds before input lands, catching wrong-window typing and stale-coordinate clicks before they happen.
|
|
16
|
+
>
|
|
17
|
+
> Under the hood: an **82× average speedup** from the Rust native engine (UIA focus queries in 2 ms, SSE2-accelerated image diffing at 13–15×), with a transparent PowerShell fallback when the engine is absent. The npm launcher fetches only the GitHub Release tag matching the installed version and verifies the Windows runtime zip before extraction.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
## Features
|
|
22
|
+
|
|
23
|
+
- **⚡ High-performance Rust Native Core** — The UIA bridge and image-diff engine are written in Rust (`napi-rs` + `windows-rs`) and loaded as a native `.node` addon. Direct COM calls from a dedicated MTA thread eliminate PowerShell process spawning — `getFocusedElement` completes in **2 ms** (160× faster), and `getUiElements` returns full trees in **~100 ms** with a batch BFS algorithm that minimizes cross-process RPC. Image-diff operations use **SSE2 SIMD** for 13–15× throughput. When the native engine is unavailable, every function transparently falls back to PowerShell — zero config required.
|
|
24
|
+
- **🎯 Set-of-Marks (SoM) visual fallback** — Games, RDP sessions, and non-accessible Electron apps return clickable elements even when UIA is completely blind. `screenshot(detail="text")` automatically detects UIA sparsity and activates a Hybrid Non-CDP pipeline: Rust-powered grayscale + bilinear upscale → Windows OCR → clustering → red bounding-box annotation with numbered badges (`[1]`, `[2]`…). Two parallel representations returned: a visual PNG for spatial orientation and a semantic `elements[]` list with `clickAt` coords — no CDP required.
|
|
25
|
+
- **🔁 One-call confirmation on visual-only targets** — On UIA-blind targets (Electron, PWAs, games, custom canvases, RDP windows), `desktop_act` can fold the post-action confirmation into its own response: an optional `roiCapture` carrying a PNG crop of *just the region that changed* plus a lease-less preview of the controls now visible there. The agent confirms what its click did and finds the next target without a separate `desktop_state` + `screenshot`. On visual-only targets it is **on by default** for a visible change (`returnCapture:"on-change"`); pass `returnCapture:"never"` to suppress it, or `"always"` to force it. Never attached on structured targets (browser/CDP, UIA-rich native), where `desktop_state` is cheaper and exact — so those responses are unchanged.
|
|
26
|
+
- **🔐 Key Locker — the terminal autofills your SSH / sudo passwords** — Save a credential once into the locker's own secure dialog (stored encrypted on your machine with Windows DPAPI; never shown to the assistant), then run `ssh` / `sudo` in a console opened by `key_locker(action='launch_console')` — the password is filled in automatically when the hidden prompt appears, with a per-fill confirmation prompt by default. See [Key Locker](#key-locker-terminal-credential-autofill).
|
|
27
|
+
- **LLM-native design** — Built around how LLMs think, not how humans click. `run_macro` batches multiple operations into a single API call; `diffMode` sends only the windows that changed since the last frame. Minimal tokens, minimal round-trips.
|
|
28
|
+
- **Reactive Perception Graph** — Register a `lensId` for a window or browser tab, pass it to action tools, and get guard-checked `post.perception` feedback after each action. It reduces repeated `screenshot` / `desktop_state` calls and prevents wrong-window typing or stale-coordinate clicks.
|
|
29
|
+
- **Full CJK support** — Uses Win32 `GetWindowTextW` for window titles, avoiding nut-js garbling. IME bypass input supported for Japanese/Chinese/Korean environments.
|
|
30
|
+
- **3-tier token reduction** — `detail="image"` (~443 tok) / `detail="text"` (~100–300 tok) / `diffMode=true` (~160 tok). Send pixels only when you actually need to see them.
|
|
31
|
+
- **1:1 coordinate mode** — `dotByDot=true` captures at native resolution (WebP). Image pixel = screen coordinate — no scale math needed. With `origin`+`scale` passed to `mouse_click`, the server converts coords for you — eliminating off-by-one / scale bugs.
|
|
32
|
+
- **Browser capture data reduction** — `grayscale=true` (~50% size), `dotByDotMaxDimension=1280` (auto-scaled with coord preservation), and `windowTitle + region` sub-crops help exclude browser chrome and other irrelevant pixels. Typical reduction for heavy captures: 50–70%.
|
|
33
|
+
- **Chromium smart fallback** — `detail="text"` on Chrome/Edge/Brave auto-skips UIA (prohibitively slow there) and runs Windows OCR. `hints.chromiumGuard` + `hints.ocrFallbackFired` flag the path taken.
|
|
34
|
+
- **UIA element extraction** — `detail="text"` returns button names and `clickAt` coords as JSON. Claude can click the right element without ever looking at a screenshot.
|
|
35
|
+
- **Auto-dock CLI** — `window_dock(action='dock')` snaps any window to a screen corner with always-on-top. Set `DESKTOP_TOUCH_DOCK_TITLE='@parent'` to auto-dock the terminal hosting Claude on MCP startup — the process-tree walker finds the right window regardless of title.
|
|
36
|
+
- **Emergency stop (Failsafe)** — Move the mouse to the **top-left corner (within 10px of 0,0)** to immediately terminate the MCP server.
|
|
37
|
+
|
|
38
|
+
---
|
|
39
|
+
|
|
40
|
+
## Requirements
|
|
41
|
+
|
|
42
|
+
| | |
|
|
43
|
+
|---|---|
|
|
44
|
+
| OS | Windows 10 / 11 (64-bit) |
|
|
45
|
+
| Node.js | v20+ recommended (tested on v22+) |
|
|
46
|
+
| PowerShell | 5.1+ (bundled with Windows) — used only as fallback when the Rust native engine is unavailable |
|
|
47
|
+
| Claude CLI | `claude` command must be available |
|
|
48
|
+
|
|
49
|
+
> **Note:** nut-js native bindings require the Visual C++ Redistributable.
|
|
50
|
+
> Download from [Microsoft](https://learn.microsoft.com/en-us/cpp/windows/latest-supported-vc-redist) if not already installed.
|
|
51
|
+
|
|
52
|
+
> **Note (Key Locker):** The credential helper Key Locker uses is an unsigned executable, so on
|
|
53
|
+
> some machines Windows SmartScreen or antivirus may show an "unknown publisher" warning the
|
|
54
|
+
> first time it runs. This is expected — the helper ships with desktop-touch-mcp and runs locally
|
|
55
|
+
> on your machine; you can allow it to proceed. Code signing is planned for a future release.
|
|
56
|
+
|
|
57
|
+
---
|
|
58
|
+
|
|
59
|
+
## Installation
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
npx -y @harusame64/desktop-touch-mcp
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
The npm launcher resolves runtime strictly by npm package version. For package `X.Y.Z`, it fetches only GitHub Release tag `vX.Y.Z`, downloads `desktop-touch-mcp-windows.zip`, verifies its SHA256 digest, and only then expands it under `%USERPROFILE%\.desktop-touch-mcp`. Verified cached releases are reused on later runs.
|
|
66
|
+
|
|
67
|
+
Set `DESKTOP_TOUCH_MCP_HOME` to override the cache root directory.
|
|
68
|
+
|
|
69
|
+
> **On a shared or CI network?** The first run reads the GitHub Releases API to
|
|
70
|
+
> locate the runtime zip. The anonymous limit is 60 requests/hour per IP, which a
|
|
71
|
+
> shared public address (CI runners, office NAT) can exhaust before your download
|
|
72
|
+
> even starts. Set `GITHUB_TOKEN` (or `GH_TOKEN`) in the environment and the
|
|
73
|
+
> launcher authenticates the request, raising the limit to 5,000 requests/hour.
|
|
74
|
+
> No token is needed on an ordinary home connection.
|
|
75
|
+
|
|
76
|
+
> **Running the launcher from a source checkout?** A source build's
|
|
77
|
+
> `bin/launcher.js` carries a placeholder integrity hash (`sha256: "PENDING"`)
|
|
78
|
+
> instead of a finalized one. Rather than download and run an unverified runtime,
|
|
79
|
+
> the launcher fails closed — this guard stops an accidentally published or
|
|
80
|
+
> unfinalized launcher from silently starting unverified code. Published npm
|
|
81
|
+
> releases always ship a real SHA256, so end users never see this. If you are
|
|
82
|
+
> intentionally running the launcher from source, set
|
|
83
|
+
> `DESKTOP_TOUCH_MCP_ALLOW_UNVERIFIED=1` to skip integrity verification
|
|
84
|
+
> (development only).
|
|
85
|
+
|
|
86
|
+
### Register with Claude CLI
|
|
87
|
+
|
|
88
|
+
Add to `~/.claude.json` under `mcpServers`:
|
|
89
|
+
|
|
90
|
+
```json
|
|
91
|
+
{
|
|
92
|
+
"mcpServers": {
|
|
93
|
+
"desktop-touch": {
|
|
94
|
+
"type": "stdio",
|
|
95
|
+
"command": "npx",
|
|
96
|
+
"args": ["-y", "@harusame64/desktop-touch-mcp"]
|
|
97
|
+
}
|
|
98
|
+
}
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
**No system prompt needed.** The command reference is automatically injected into Claude via the MCP `initialize` response's `instructions` field.
|
|
103
|
+
|
|
104
|
+
### Register with other clients (HTTP mode)
|
|
105
|
+
|
|
106
|
+
Clients that require an HTTP endpoint (GPT Desktop, VS Code Copilot, Cursor, etc.) can use the built-in Streamable HTTP transport:
|
|
107
|
+
|
|
108
|
+
```bash
|
|
109
|
+
npx -y @harusame64/desktop-touch-mcp --http
|
|
110
|
+
# or with a custom port:
|
|
111
|
+
npx -y @harusame64/desktop-touch-mcp --http --port 8080
|
|
112
|
+
```
|
|
113
|
+
|
|
114
|
+
The server starts at `http://127.0.0.1:23847/mcp` (localhost only). Register the URL in your MCP client settings. A health check is available at `http://127.0.0.1:<port>/health`.
|
|
115
|
+
|
|
116
|
+
In HTTP mode the system tray icon shows the active URL and provides quick-copy and open-in-browser shortcuts.
|
|
117
|
+
|
|
118
|
+
### Development install
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
git clone https://github.com/Harusame64/desktop-touch-mcp.git
|
|
122
|
+
cd desktop-touch-mcp
|
|
123
|
+
npm install
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Build after install:
|
|
127
|
+
|
|
128
|
+
```bash
|
|
129
|
+
npm run build
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
For a local checkout, register the built server directly:
|
|
133
|
+
|
|
134
|
+
```json
|
|
135
|
+
{
|
|
136
|
+
"mcpServers": {
|
|
137
|
+
"desktop-touch": {
|
|
138
|
+
"type": "stdio",
|
|
139
|
+
"command": "node",
|
|
140
|
+
"args": ["D:/path/to/desktop-touch-mcp/dist/index.js"]
|
|
141
|
+
}
|
|
142
|
+
}
|
|
143
|
+
}
|
|
144
|
+
```
|
|
145
|
+
|
|
146
|
+
> **Note:** Replace `D:/path/to/desktop-touch-mcp` with the actual path where you cloned this repository.
|
|
147
|
+
|
|
148
|
+
---
|
|
149
|
+
|
|
150
|
+
## Tools (32 Optimized Tools)
|
|
151
|
+
|
|
152
|
+
> 📖 **Full Reference**: [`docs/system-overview.md`](docs/system-overview.md) — Exhaustive guide on parameters, return schemas, and coordinate math.
|
|
153
|
+
|
|
154
|
+
### 🌐 World-Graph V2 (Primary Path)
|
|
155
|
+
| Tool | Description |
|
|
156
|
+
|---|---|
|
|
157
|
+
| `desktop_discover` | Observe the desktop. Returns interactive entities with leases (UIA, CDP, Terminal, Visual SoM). |
|
|
158
|
+
| `desktop_act` | Perform actions (click, type, drag, select) on entities via lease validation. Returns semantic diffs — plus an optional `roiCapture` (changed-region PNG + next-target preview) on visual-only targets. |
|
|
159
|
+
|
|
160
|
+
### 👁️ Observation & State
|
|
161
|
+
| Tool | Description |
|
|
162
|
+
|---|---|
|
|
163
|
+
| `desktop_state` | Lightweight check of focus, active window, cursor, and Auto-Perception attention signal. |
|
|
164
|
+
| `screenshot` | Multi-mode capture: `detail='text'` (UIA/OCR), `diffMode` (P-frame), `dotByDot` (1:1), and `background`. Returns a cheap `screenshot://by-ref/{id}` link to the saved image instead of inlining pixels every time. |
|
|
165
|
+
| `screenshot_query` / `screenshot_gc` | Inspect and prune the on-disk screenshot cache behind the by-ref links: `screenshot_query` lists saved captures without re-reading pixels; `screenshot_gc` reclaims space by retention policy (dry-run by default). |
|
|
166
|
+
| `workspace_snapshot` | Instant session orientation: all window thumbnails + UI summaries in one call. |
|
|
167
|
+
| `server_status` | Diagnostic check for native engine health and feature activation. |
|
|
168
|
+
|
|
169
|
+
### ⌨️ Input & Control
|
|
170
|
+
| Tool | Description |
|
|
171
|
+
|---|---|
|
|
172
|
+
| `keyboard` | Send keyboard input. Supports background input (WM_CHAR) and IME-safe clipboard bypass. |
|
|
173
|
+
| `mouse_click` / `mouse_drag` | Precision coordinate-based interaction with homing and force-focus protection. |
|
|
174
|
+
| `scroll` | Multi-strategy: `raw` (notches), `to_element`, `smart` (virtual lists), and `capture` (stitch). |
|
|
175
|
+
| `click_element` | Legacy UIA-based click by name/ID (fallback when entities are unavailable). |
|
|
176
|
+
|
|
177
|
+
### 🌐 Browser CDP (Chrome/Edge/Brave)
|
|
178
|
+
| Tool | Description |
|
|
179
|
+
|---|---|
|
|
180
|
+
| `browser_open` / `browser_navigate` | Idempotent debug-mode launch and reliable navigation. |
|
|
181
|
+
| `browser_click` / `browser_fill` / `browser_form` | High-level DOM interaction stable across repaints and framework re-renders. |
|
|
182
|
+
| `browser_eval` | Deep inspection via `js` (scripting), `dom` (HTML), and `appState` (SPA data extraction). |
|
|
183
|
+
| `browser_overview` / `browser_search` / `browser_locate` | Semantic discovery, grep-like DOM search, and pixel-accurate coordinate lookup. |
|
|
184
|
+
|
|
185
|
+
### 🛠️ Utilities & Workflow
|
|
186
|
+
| Tool | Description |
|
|
187
|
+
|---|---|
|
|
188
|
+
| `terminal` | Unified command execution: `run` (send + wait + read), `read` (OCR/UIA), and `send`. `run` completion modes: `quiet`, `pattern`, and `exit` (waits for the command to finish + returns its exit code — see [Terminal command completion](#terminal-command-completion-until)). |
|
|
189
|
+
| `wait_until` | Efficient server-side polling for window, focus, text, or URL state changes. |
|
|
190
|
+
| `window_dock` / `focus_window` | Window management: `pin` (always-on-top), `unpin`, `dock` (corner snap), and `focus`. |
|
|
191
|
+
| `workspace_launch` | Launch apps and auto-detect new HWNDs (supports localized titles). |
|
|
192
|
+
| `run_macro` | Batch up to 50 operations into a single round-trip for maximum efficiency. |
|
|
193
|
+
| `clipboard` / `notification_show` | System-level text exchange and user alerts. |
|
|
194
|
+
| `key_locker` | Manage credentials the terminal autofills for you (SSH key passphrases, sudo / login passwords). Secrets are entered once into the locker's own secure dialog and stored encrypted on this machine (Windows DPAPI); they are never shown to the assistant. `action='launch_console'` opens an autofill-capable console (returns a `paneId` to drive `ssh`/`sudo` into via `terminal`); `save` / `list` / `forget` / `set_policy` / `status` manage bindings. Autofill only fires in a console opened by `launch_console`. Disable with `DESKTOP_TOUCH_DISABLE_KEY_LOCKER=1`. |
|
|
195
|
+
|
|
196
|
+
### 📊 Office (Excel)
|
|
197
|
+
| Tool | Description |
|
|
198
|
+
|---|---|
|
|
199
|
+
| `excel` | Author and run Excel VBA macros via COM. `action='run_vba'` writes a macro into a managed Trusted Location and runs it; `action='check_access_vbom'` is a read-only preflight. Runs VBA where formula-only tools cannot. One-time setup: `node scripts/enable-access-vbom.mjs`. |
|
|
200
|
+
|
|
201
|
+
---
|
|
202
|
+
|
|
203
|
+
## Standard workflow (v1.0.0)
|
|
204
|
+
|
|
205
|
+
The v2 World-Graph surface (`desktop_discover` / `desktop_act`) is the recommended dispatch path. The four-call shape works for native apps, browsers, and terminals identically.
|
|
206
|
+
|
|
207
|
+
```
|
|
208
|
+
desktop_state → orient: focused window/element, modal, attention signal
|
|
209
|
+
desktop_discover → find actionable entities (returns lease + windows[])
|
|
210
|
+
desktop_act(lease, …) → act on entity (returns attention + post.perception)
|
|
211
|
+
desktop_state → confirm the world changed as expected
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
Clicking — priority order:
|
|
215
|
+
|
|
216
|
+
```
|
|
217
|
+
browser_click(selector) → Chrome / Edge (CDP, stable across repaints)
|
|
218
|
+
desktop_act(lease, action='click') → native / dialog / visual (entity-based; use after desktop_discover)
|
|
219
|
+
click_element(name | automationId) → native UIA fallback if desktop_act returns ok:false
|
|
220
|
+
mouse_click(x, y, origin?, scale?) → pixel last resort; origin+scale from dotByDot screenshots only
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Recovery hints — read `response.attention` after every observation and `response.warnings[]` on `desktop_discover` / `desktop_act`. Common reasons:
|
|
224
|
+
|
|
225
|
+
- `lease_expired` / `lease_generation_mismatch` / `lease_digest_mismatch` / `entity_not_found` → re-call `desktop_discover`
|
|
226
|
+
- `modal_blocking` → `response.blockingElement` (when present) names the blocking modal; dismiss via `click_element(name=blockingElement.name)` then retry
|
|
227
|
+
- `entity_outside_viewport` → `scroll(action='to_element' | 'raw')`, then re-call `desktop_discover`
|
|
228
|
+
- `executor_failed` → fall back to `click_element` / `mouse_click` / `browser_click`
|
|
229
|
+
|
|
230
|
+
Lease lifecycle:
|
|
231
|
+
|
|
232
|
+
- Each `desktop_discover` response carries `softExpiresAtMs` (≈ 60 % of the TTL window). Past that timestamp the LLM should consider re-calling `desktop_discover` even though the lease is still technically valid — `lease.expiresAtMs` is the only correctness wall.
|
|
233
|
+
- TTL adapts to `view` mode (`action`/`explore`/`debug`), entity count, and response payload size. Cap is 60 s.
|
|
234
|
+
- Set `DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1` to fall back to the v1 tool surface (`get_windows` / `get_ui_elements` / `set_element_value`) for troubleshooting only — V2 is the recommended default.
|
|
235
|
+
|
|
236
|
+
---
|
|
237
|
+
|
|
238
|
+
## Terminal command completion (`until`)
|
|
239
|
+
|
|
240
|
+
`terminal(action='run')` sends a command, waits for it to complete, and reads the
|
|
241
|
+
output in one call. How it decides "complete" is controlled by `until`:
|
|
242
|
+
|
|
243
|
+
| Mode | Waits for | Best for |
|
|
244
|
+
|---|---|---|
|
|
245
|
+
| `quiet` (default) | output to fall silent for `quietMs` | short interactive commands |
|
|
246
|
+
| `pattern` | a string/regex you expect in the output | long commands with a known final marker |
|
|
247
|
+
| `exit` | the command to actually **finish** | when you need completion or the exit code |
|
|
248
|
+
|
|
249
|
+
> **Anchoring caveat (#384):** a command whose final line has no trailing newline
|
|
250
|
+
> glues the marker to the next prompt with no line boundary (`printf X` →
|
|
251
|
+
> `Xuser@host:~$`), so an end-anchored `pattern` (`X\s*\n` / `X$`) can never bind.
|
|
252
|
+
> For *completion* use `mode:'exit'`; for *content* matching use a bare marker
|
|
253
|
+
> (no `\n`/`$`). `mode:'pattern'` also accepts an optional `quietMs` settle
|
|
254
|
+
> fallback: `until:{mode:'pattern', pattern, quietMs:1000}` completes with
|
|
255
|
+
> `reason:'quiet'` (no `matchedPattern`) once output is stable for that long
|
|
256
|
+
> without a match — instead of hanging until `timeoutMs`. It is opt-in (omit
|
|
257
|
+
> `quietMs` to keep waiting for the pattern; long commands with mid-run silent
|
|
258
|
+
> gaps are unaffected).
|
|
259
|
+
|
|
260
|
+
### `until:{mode:'exit'}` — real completion + exit code
|
|
261
|
+
|
|
262
|
+
The heuristic modes can misfire on the common "append a sentinel" idiom
|
|
263
|
+
(`some-task; echo DONE` matched by `DONE`): the sentinel also shows up in the
|
|
264
|
+
**echoed command line**, and for multi-line commands there is no reliable way to
|
|
265
|
+
tell that echo apart from real output. `mode:'exit'` removes the guesswork — the
|
|
266
|
+
server appends its own completion marker whose *printed* form differs from its
|
|
267
|
+
*typed* form, so it never matches the echoed command (even for multi-line input),
|
|
268
|
+
and it returns the real process exit code:
|
|
269
|
+
|
|
270
|
+
```js
|
|
271
|
+
terminal({
|
|
272
|
+
action: 'run',
|
|
273
|
+
windowTitle: 'pwsh',
|
|
274
|
+
input: 'npm run build',
|
|
275
|
+
until: { mode: 'exit', shell: 'powershell' },
|
|
276
|
+
})
|
|
277
|
+
// → completion: { reason: 'exited', exitCode: 0, elapsedMs: … }
|
|
278
|
+
// output: just the command's real output (the injected marker is stripped)
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
- **Pass `shell` explicitly** (`'bash'` or `'powershell'`). `shell:'auto'` detects
|
|
282
|
+
the shell from the terminal window, but it cannot see a shell running *inside*
|
|
283
|
+
SSH or WSL — the window still looks like its local host — so for remote/nested
|
|
284
|
+
sessions pass the remote side's shell (`auto` otherwise warns and may pick the
|
|
285
|
+
outer shell). A window whose process is genuinely unidentifiable (e.g. Windows
|
|
286
|
+
Terminal) returns `ExitModeShellAmbiguous`.
|
|
287
|
+
- **First-class shells:** `bash` and `powershell`. `cmd.exe` is not supported yet
|
|
288
|
+
(`ExitModeShellUnsupported`).
|
|
289
|
+
- **Unsafe input is rejected up front** (`ExitModeUnsafeInput`) rather than
|
|
290
|
+
hanging: a command ending mid-construct (unterminated quote, here-doc, `$(…)`,
|
|
291
|
+
a trailing `\` or PowerShell backtick).
|
|
292
|
+
- Exit mode controls its own delivery, so delivery-shaping `sendOptions`
|
|
293
|
+
(`method` / `preferClipboard` / `pressEnter` / `chunkSize` / `pasteKey`) are
|
|
294
|
+
rejected with `InvalidArgs`; focus options remain accepted.
|
|
295
|
+
|
|
296
|
+
---
|
|
297
|
+
|
|
298
|
+
## Key Locker (terminal credential autofill)
|
|
299
|
+
|
|
300
|
+
Running `ssh user@host` or `sudo …` normally stops at a hidden password prompt an
|
|
301
|
+
assistant can't safely type into. Key Locker stores your SSH key passphrases and
|
|
302
|
+
sudo / login passwords encrypted on your machine (Windows DPAPI, current user) and
|
|
303
|
+
fills them in automatically when a bound command reaches its prompt. The secret is
|
|
304
|
+
typed once into the locker's own secure dialog — it is never shown to the assistant
|
|
305
|
+
and never travels through the MCP channel.
|
|
306
|
+
|
|
307
|
+
```js
|
|
308
|
+
// 1. Save the credential once — opens a secure dialog on your desktop
|
|
309
|
+
key_locker({ action:'save', uri:'ssh://user@host:22' })
|
|
310
|
+
|
|
311
|
+
// 2. Open an autofill-capable console (returns its paneId)
|
|
312
|
+
key_locker({ action:'launch_console' }) // → { paneId:'12345678', windowTitle:'…' }
|
|
313
|
+
|
|
314
|
+
// 3. Run the command through that pane — the password is filled at the prompt
|
|
315
|
+
terminal({ action:'send', paneId:'12345678', input:'ssh user@host' })
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
- **Autofill only fires in a console opened by `launch_console`** — a pre-existing
|
|
319
|
+
terminal is never autofilled. The console is a classic visible Windows console,
|
|
320
|
+
so you can watch it and take over at any prompt yourself.
|
|
321
|
+
- **Every autofill asks you to confirm by default**; opt a binding out with
|
|
322
|
+
`set_policy`. `list` / `status` / `forget` manage saved credentials.
|
|
323
|
+
- `terminal` `read` / `send` accept `paneId` as an alternative to `windowTitle` —
|
|
324
|
+
it targets that exact window even after an `ssh` login renames its title.
|
|
325
|
+
- Supported binding URIs: `ssh://user@host:22`, `sudo://host/user`,
|
|
326
|
+
`https-cred://host`, and SSH key passphrases (`sshkey:SHA256:…`). An `ssh`
|
|
327
|
+
save needs the host key already in `known_hosts` (connect to the host once first).
|
|
328
|
+
- Windows only. Disable the whole feature with `DESKTOP_TOUCH_DISABLE_KEY_LOCKER=1`.
|
|
329
|
+
The secure dialog is an unsigned helper executable — Windows SmartScreen may show
|
|
330
|
+
an "unknown publisher" warning on first run (see the note under
|
|
331
|
+
[Requirements](#requirements)).
|
|
332
|
+
|
|
333
|
+
---
|
|
334
|
+
|
|
335
|
+
## Browser CDP automation
|
|
336
|
+
|
|
337
|
+
For web automation, connect Chrome or Edge with the remote debugging port enabled — no Selenium or Playwright needed.
|
|
338
|
+
|
|
339
|
+
```bash
|
|
340
|
+
# Launch Chrome in CDP mode
|
|
341
|
+
chrome.exe --remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
```
|
|
345
|
+
browser_open({launch:{}}) → spawn-if-needed Chrome in debug mode + list tabs (idempotent)
|
|
346
|
+
browser_open() → connect-only (fail if no CDP endpoint live)
|
|
347
|
+
browser_locate({selector:"#submit"}) → CSS selector → physical screen coords
|
|
348
|
+
browser_click({selector:"#submit"}) → find + click in one step (auto-focuses browser)
|
|
349
|
+
browser_eval({action:"js", expression:"document.title"}) → evaluate JS, returns result
|
|
350
|
+
browser_eval({action:"dom", selector:"#main", maxLength:5000}) → outerHTML, truncated to maxLength chars
|
|
351
|
+
browser_eval({action:"appState"}) → one-shot SPA state (Next/Nuxt/Remix/Apollo/GitHub react-app/Redux SSR)
|
|
352
|
+
browser_fill({selector:"#email", value:"user@example.com"}) → fill React/Vue/Svelte controlled input (state-safe)
|
|
353
|
+
browser_overview() → links/buttons/inputs + ARIA toggles + viewportPosition per element
|
|
354
|
+
browser_search({by:"text", pattern:"..."}) → grep DOM with confidence ranking
|
|
355
|
+
browser_navigate({url:"https://example.com"}) → navigate via CDP (no address bar interaction)
|
|
356
|
+
```
|
|
357
|
+
|
|
358
|
+
For chained calls in the same tab, pass `includeContext:false` to omit the activeTab/readyState annotation (~150 tok/call saved). Boolean / object params accept the LLM-friendly string spellings (`"true"`, `"{}"`).
|
|
359
|
+
|
|
360
|
+
Coordinates returned by `browser_locate` account for the browser chrome (tab strip + address bar height) and `devicePixelRatio`, so they can be passed directly to `mouse_click` without any scaling.
|
|
361
|
+
|
|
362
|
+
**Recommended web workflow:**
|
|
363
|
+
```
|
|
364
|
+
browser_open({launch:{}}) → browser_eval({action:"dom"}) → browser_locate(selector) → browser_click(selector)
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
---
|
|
368
|
+
|
|
369
|
+
## Auto-dock CLI on startup
|
|
370
|
+
|
|
371
|
+
Keep Claude CLI visible while operating other apps full-screen. Set env vars in your MCP config and the docked window auto-snaps into place every MCP startup.
|
|
372
|
+
|
|
373
|
+
```json
|
|
374
|
+
{
|
|
375
|
+
"mcpServers": {
|
|
376
|
+
"desktop-touch": {
|
|
377
|
+
"type": "stdio",
|
|
378
|
+
"command": "npx",
|
|
379
|
+
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
380
|
+
"env": {
|
|
381
|
+
"DESKTOP_TOUCH_DOCK_TITLE": "@parent",
|
|
382
|
+
"DESKTOP_TOUCH_DOCK_CORNER": "bottom-right",
|
|
383
|
+
"DESKTOP_TOUCH_DOCK_WIDTH": "480",
|
|
384
|
+
"DESKTOP_TOUCH_DOCK_HEIGHT": "360",
|
|
385
|
+
"DESKTOP_TOUCH_DOCK_PIN": "true"
|
|
386
|
+
}
|
|
387
|
+
}
|
|
388
|
+
}
|
|
389
|
+
}
|
|
390
|
+
```
|
|
391
|
+
|
|
392
|
+
| Env var | Default | Notes |
|
|
393
|
+
|---|---|---|
|
|
394
|
+
| `DESKTOP_TOUCH_DOCK_TITLE` | *(unset = off)* | `@parent` walks the MCP process tree to find the hosting terminal — immune to title / branch / project changes. Or use a literal substring. |
|
|
395
|
+
| `DESKTOP_TOUCH_DOCK_CORNER` | `bottom-right` | `top-left` / `top-right` / `bottom-left` / `bottom-right` |
|
|
396
|
+
| `DESKTOP_TOUCH_DOCK_WIDTH` / `HEIGHT` | `480` / `360` | px (`"480"`) or ratio of work area (`"25%"`) — 4K/8K auto-adapts |
|
|
397
|
+
| `DESKTOP_TOUCH_DOCK_PIN` | `true` | Always-on-top toggle |
|
|
398
|
+
| `DESKTOP_TOUCH_DOCK_MONITOR` | primary | Monitor id from `desktop_state({includeScreen:true})` |
|
|
399
|
+
| `DESKTOP_TOUCH_DOCK_SCALE_DPI` | `false` | If true, multiply px values by `dpi / 96` (opt-in per-monitor scaling) |
|
|
400
|
+
| `DESKTOP_TOUCH_DOCK_MARGIN` | `8` | Screen-edge padding (px) |
|
|
401
|
+
| `DESKTOP_TOUCH_DOCK_TIMEOUT_MS` | `5000` | Max wait for the target window to appear |
|
|
402
|
+
|
|
403
|
+
> **Input routing gotcha:** when a pinned window is active (e.g. Claude CLI), `keyboard(action='type')` / `keyboard(action='press')` send keys to it, **not** the app you wanted to type into. Always call `focus_window(title=...)` before keyboard operations, then verify `isActive=true` via `screenshot(detail='meta')`.
|
|
404
|
+
|
|
405
|
+
### Screenshot cache (by-ref storage)
|
|
406
|
+
|
|
407
|
+
`screenshot` and the other visual results return a cheap `screenshot://by-ref/{id}` link to an image saved on disk instead of inlining the pixels every time, so routine look-act-confirm loops cost far fewer tokens. The cache bounds itself automatically and `screenshot_query` / `screenshot_gc` let you inspect and prune it. Tune the storage with:
|
|
408
|
+
|
|
409
|
+
| Env var | Default | Notes |
|
|
410
|
+
|---|---|---|
|
|
411
|
+
| `DESKTOP_TOUCH_SCREENSHOTS_DIR` | *(per-user cache dir)* | Pin the cache to a specific folder. If the default folder can't be created or written (e.g. corporate policy blocking new folders under your profile), the server auto-probes this → the runtime dir → an OS temp folder and uses the first writable one instead of giving up on the cache. |
|
|
412
|
+
| `DESKTOP_TOUCH_SCREENSHOT_MAX_COUNT` | `200` | Keep at most this many captures in the cache. |
|
|
413
|
+
| `DESKTOP_TOUCH_SCREENSHOT_MAX_BYTES` | `256 MiB` | Cap the total cache size on disk. |
|
|
414
|
+
| `DESKTOP_TOUCH_SCREENSHOT_MAX_AGE_MS` | *(off)* | Drop captures older than this many milliseconds (opt-in). |
|
|
415
|
+
| `DESKTOP_TOUCH_SCREENSHOT_AUTOPRUNE` | `on` | Auto-trim the cache as new captures are saved. Set `0` to disable. |
|
|
416
|
+
| `DESKTOP_TOUCH_SCREENSHOT_MIN_EVICT_AGE_MS` | `60000` | Never auto-evict a capture younger than this (ms), so a by-ref link you were just handed survives long enough to open even when another AI/process on the same PC is also capturing. `0` disables. |
|
|
417
|
+
|
|
418
|
+
### Auto Perception (always-on)
|
|
419
|
+
|
|
420
|
+
Phase 4 privatizes the explicit `perception_*` tool family — the v0.12 Auto
|
|
421
|
+
Perception layer attaches an `attention` signal to every `desktop_state` and
|
|
422
|
+
`desktop_act` response automatically. Action tools also auto-guard when given
|
|
423
|
+
a `windowTitle`. There is no longer a need to register / read / forget lenses
|
|
424
|
+
manually.
|
|
425
|
+
|
|
426
|
+
```
|
|
427
|
+
# desktop_state always returns the attention signal
|
|
428
|
+
desktop_state() → {focusedWindow, focusedElement, modal, attention:"ok", ...}
|
|
429
|
+
|
|
430
|
+
# Action tools auto-guard when windowTitle is given:
|
|
431
|
+
keyboard({action:"type", text:"hello", windowTitle:"Notepad"})
|
|
432
|
+
→ post.perception:{status:"ok"} // unsafe input blocked if guards fail
|
|
433
|
+
|
|
434
|
+
# When attention is dirty / stale / settling, refresh with desktop_state:
|
|
435
|
+
desktop_state() // re-evaluates attention via Auto Perception
|
|
436
|
+
```
|
|
437
|
+
|
|
438
|
+
For advanced pinned-target workflows, the `lensId` parameter remains on action
|
|
439
|
+
tools (`keyboard`, `mouse_click`, `mouse_drag`, `click_element`,
|
|
440
|
+
`browser_click`, `browser_navigate`, `browser_eval`, `desktop_act`). Omit
|
|
441
|
+
`lensId` for the normal Auto Perception path. The underlying registry, hot
|
|
442
|
+
target cache, and sensor loop are unchanged; only the explicit
|
|
443
|
+
`perception_register / perception_read / perception_forget / perception_list`
|
|
444
|
+
tools were retired.
|
|
445
|
+
|
|
446
|
+
---
|
|
447
|
+
|
|
448
|
+
## Mouse homing correction
|
|
449
|
+
|
|
450
|
+
When Claude calls `screenshot(detail='text')` to read coordinates and then `mouse_click` seconds later, the target window may have moved. The homing system corrects this automatically.
|
|
451
|
+
|
|
452
|
+
| Tier | How to enable | Latency | What it does |
|
|
453
|
+
|------|--------------|---------|--------------|
|
|
454
|
+
| 1 | Always-on (if cache exists) | <1ms | Applies (dx, dy) offset when window moved |
|
|
455
|
+
| 2 | Pass `windowTitle` hint | ~100ms | Auto-focuses window if it went behind another |
|
|
456
|
+
| 3 | Pass `elementName`/`elementId` + `windowTitle` | 1–3s | UIA re-query for fresh coords on resize |
|
|
457
|
+
|
|
458
|
+
```
|
|
459
|
+
# Tier 1 only (automatic)
|
|
460
|
+
mouse_click(x=500, y=300)
|
|
461
|
+
|
|
462
|
+
# Tier 1 + 2: also bring window to front if hidden
|
|
463
|
+
mouse_click(x=500, y=300, windowTitle="Notepad")
|
|
464
|
+
|
|
465
|
+
# Tier 1 + 2 + 3: also re-query UIA if window resized
|
|
466
|
+
mouse_click(x=500, y=300, windowTitle="Notepad", elementName="Save")
|
|
467
|
+
|
|
468
|
+
# Traction control OFF — no correction
|
|
469
|
+
mouse_click(x=500, y=300, homing=false)
|
|
470
|
+
```
|
|
471
|
+
|
|
472
|
+
The `homing` parameter is available on `mouse_click`, `mouse_drag`, and `scroll`. The cache is updated automatically on every `screenshot()`, `desktop_discover()`, `focus_window()`, and `workspace_snapshot()` call.
|
|
473
|
+
|
|
474
|
+
### `mouse_click` image-local coords (origin + scale)
|
|
475
|
+
|
|
476
|
+
When you take a `dotByDot` screenshot with `dotByDotMaxDimension`, the response prints the `origin` and `scale` values. Instead of computing screen coords manually, copy them into `mouse_click`:
|
|
477
|
+
|
|
478
|
+
```
|
|
479
|
+
# Screenshot response:
|
|
480
|
+
# origin: (0, 120) | scale: 0.6667
|
|
481
|
+
# To click image pixel (ix, iy): mouse_click(x=ix, y=iy, origin={x:0, y:120}, scale=0.6667)
|
|
482
|
+
|
|
483
|
+
mouse_click(x=640, y=300, origin={x:0, y:120}, scale=0.6667, windowTitle="Chrome")
|
|
484
|
+
# Server converts: screen = (0 + 640/0.6667, 120 + 300/0.6667) = (960, 570)
|
|
485
|
+
```
|
|
486
|
+
|
|
487
|
+
This eliminates a whole class of off-by-one and scale bugs. Without origin/scale, `x`/`y` remain absolute screen pixels (unchanged behavior).
|
|
488
|
+
|
|
489
|
+
---
|
|
490
|
+
|
|
491
|
+
## `screenshot` key parameters
|
|
492
|
+
|
|
493
|
+
```
|
|
494
|
+
detail="image" — PNG/WebP pixels (default)
|
|
495
|
+
detail="text" — UIA element JSON + clickAt coords (no image, ~100–300 tok)
|
|
496
|
+
detail="meta" — Title + region only (cheapest, ~20 tok/window)
|
|
497
|
+
dotByDot=true — 1:1 WebP; image_px + origin = screen_px
|
|
498
|
+
dotByDotMaxDimension=N — cap longest edge (response includes scale for coord math)
|
|
499
|
+
grayscale=true — ~50% smaller for text-heavy captures (code/AWS console)
|
|
500
|
+
region={x,y,w,h} — with windowTitle: window-local coords (exclude browser chrome)
|
|
501
|
+
without: virtual screen coords
|
|
502
|
+
diffMode=true — I-frame first call, P-frame (changed windows only) after (~160 tok)
|
|
503
|
+
ocrFallback="auto" — detail='text' auto-fires Windows OCR on uiaSparse or empty
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
**Recommended Chrome combo** (50–70% data reduction):
|
|
507
|
+
```
|
|
508
|
+
screenshot(windowTitle="Chrome",
|
|
509
|
+
dotByDot=true, dotByDotMaxDimension=1280, grayscale=true,
|
|
510
|
+
region={x:0, y:120, width:1920, height:900}) # skip browser chrome
|
|
511
|
+
```
|
|
512
|
+
|
|
513
|
+
**Recommended workflow:**
|
|
514
|
+
```
|
|
515
|
+
workspace_snapshot() → full orientation (resets diff buffer)
|
|
516
|
+
screenshot(detail="text", windowTitle=X) → get actionable[].clickAt coords
|
|
517
|
+
mouse_click(x, y) → click directly, no math needed
|
|
518
|
+
screenshot(diffMode=true) → check only what changed (~160 tok)
|
|
519
|
+
```
|
|
520
|
+
|
|
521
|
+
---
|
|
522
|
+
|
|
523
|
+
## Security
|
|
524
|
+
|
|
525
|
+
### Emergency stop (Failsafe)
|
|
526
|
+
|
|
527
|
+
**Move the mouse to the top-left corner of the screen (within 10px of 0,0) to immediately terminate the MCP server.**
|
|
528
|
+
|
|
529
|
+
- **Per-tool check**: `checkFailsafe()` runs before every tool handler
|
|
530
|
+
- **Background monitor**: 500ms polling as a backup for long-running operations
|
|
531
|
+
- Trigger radius: 10px
|
|
532
|
+
|
|
533
|
+
### Blocked operations
|
|
534
|
+
|
|
535
|
+
**`workspace_launch` blocklist:**
|
|
536
|
+
`cmd.exe`, `powershell.exe`, `pwsh.exe`, `wscript.exe`, `cscript.exe`, `mshta.exe`, `regsvr32.exe`, `rundll32.exe`, `msiexec.exe`, `bash.exe`, `wsl.exe` are blocked.
|
|
537
|
+
Script extensions (`.bat`, `.ps1`, `.vbs`, etc.) are rejected. Arguments containing `;`, `&`, `|`, `` ` ``, `$(`, `${` are also rejected.
|
|
538
|
+
|
|
539
|
+
**`keyboard(action='press')` blocklist:**
|
|
540
|
+
`Win+R` (Run dialog), `Win+X` (admin menu), `Win+S` (search), `Win+L` (lock screen) are blocked.
|
|
541
|
+
|
|
542
|
+
### PowerShell injection protection
|
|
543
|
+
|
|
544
|
+
All `-like` patterns in the UIA bridge PowerShell fallback path are sanitized with `escapeLike()`, which escapes wildcard characters (`*`, `?`, `[`, `]`) before they reach PowerShell. When the Rust native engine is active, PowerShell is not invoked for UIA operations.
|
|
545
|
+
|
|
546
|
+
### Allowlist for `workspace_launch`
|
|
547
|
+
|
|
548
|
+
Shell interpreters are blocked by default. To allow specific executables, create an allowlist file:
|
|
549
|
+
|
|
550
|
+
**File locations (searched in order):**
|
|
551
|
+
1. Path in `DESKTOP_TOUCH_ALLOWLIST` environment variable
|
|
552
|
+
2. `~/.claude/desktop-touch-allowlist.json`
|
|
553
|
+
3. `desktop-touch-allowlist.json` in the server's working directory
|
|
554
|
+
|
|
555
|
+
**Format:**
|
|
556
|
+
```json
|
|
557
|
+
{
|
|
558
|
+
"allowedExecutables": [
|
|
559
|
+
"pwsh.exe",
|
|
560
|
+
"C:\\Tools\\myapp.exe"
|
|
561
|
+
]
|
|
562
|
+
}
|
|
563
|
+
```
|
|
564
|
+
|
|
565
|
+
Changes take effect immediately — no restart needed.
|
|
566
|
+
|
|
567
|
+
---
|
|
568
|
+
|
|
569
|
+
## Mouse movement speed
|
|
570
|
+
|
|
571
|
+
All mouse tools (`mouse_click`, `mouse_drag`, `scroll`) accept an optional `speed` parameter:
|
|
572
|
+
|
|
573
|
+
| Value | Behavior |
|
|
574
|
+
|---|---|
|
|
575
|
+
| Omitted | Uses the configured default (see below) |
|
|
576
|
+
| `0` | Instant teleport — `setPosition()`, no animation |
|
|
577
|
+
| `1–N` | Animated movement at N px/sec |
|
|
578
|
+
|
|
579
|
+
**Default speed** is 1500 px/sec. Change it permanently via the `DESKTOP_TOUCH_MOUSE_SPEED` environment variable:
|
|
580
|
+
|
|
581
|
+
```json
|
|
582
|
+
{
|
|
583
|
+
"mcpServers": {
|
|
584
|
+
"desktop-touch": {
|
|
585
|
+
"type": "stdio",
|
|
586
|
+
"command": "npx",
|
|
587
|
+
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
588
|
+
"env": {
|
|
589
|
+
"DESKTOP_TOUCH_MOUSE_SPEED": "3000"
|
|
590
|
+
}
|
|
591
|
+
}
|
|
592
|
+
}
|
|
593
|
+
}
|
|
594
|
+
```
|
|
595
|
+
|
|
596
|
+
Common values: `0` = teleport, `1500` = default gentle, `3000` = fast, `5000` = very fast.
|
|
597
|
+
|
|
598
|
+
---
|
|
599
|
+
|
|
600
|
+
## Force-Focus (AttachThreadInput)
|
|
601
|
+
|
|
602
|
+
Windows foreground-stealing protection can prevent `SetForegroundWindow` from succeeding when another window (such as a pinned Claude CLI) is in the foreground. This causes subsequent keystrokes or clicks to land in the wrong window — a silent failure.
|
|
603
|
+
|
|
604
|
+
`mouse_click`, `keyboard(action='type')`, `keyboard(action='press')`, and `terminal(action='send')` all accept a `forceFocus` parameter that bypasses this protection using `AttachThreadInput`:
|
|
605
|
+
|
|
606
|
+
```json
|
|
607
|
+
{
|
|
608
|
+
"name": "mouse_click",
|
|
609
|
+
"arguments": {
|
|
610
|
+
"x": 500,
|
|
611
|
+
"y": 300,
|
|
612
|
+
"windowTitle": "Google Chrome",
|
|
613
|
+
"forceFocus": true
|
|
614
|
+
}
|
|
615
|
+
}
|
|
616
|
+
```
|
|
617
|
+
|
|
618
|
+
If the force attempt is refused despite `AttachThreadInput`, the response is `ok:false` with `code: "ForegroundRestricted"` (issue #202 unification — same shape as `focus_window`, `keyboard`, `terminal_send`, `mouse_click`). The action itself is **suppressed** so the keystrokes / click never land on the wrong window. Recover via `focus_window`'s auto-escalate ladder before retrying. The legacy `hints.warnings: ["ForceFocusRefused"]` shape is no longer emitted.
|
|
619
|
+
|
|
620
|
+
**Global default via environment variable:**
|
|
621
|
+
|
|
622
|
+
```json
|
|
623
|
+
{
|
|
624
|
+
"mcpServers": {
|
|
625
|
+
"desktop-touch": {
|
|
626
|
+
"env": {
|
|
627
|
+
"DESKTOP_TOUCH_FORCE_FOCUS": "1"
|
|
628
|
+
}
|
|
629
|
+
}
|
|
630
|
+
}
|
|
631
|
+
}
|
|
632
|
+
```
|
|
633
|
+
|
|
634
|
+
Setting `DESKTOP_TOUCH_FORCE_FOCUS=1` makes `forceFocus: true` the default for all four tools without changing each call.
|
|
635
|
+
|
|
636
|
+
**Known tradeoffs:**
|
|
637
|
+
|
|
638
|
+
- During the ~10ms `AttachThreadInput` window, key state and mouse capture are shared between the two threads. In rapid macro sequences this can cause a race condition (rare in practice).
|
|
639
|
+
- Disable `forceFocus` (or unset the env var) when the user is manually operating another app to avoid unexpected focus shifts.
|
|
640
|
+
|
|
641
|
+
---
|
|
642
|
+
|
|
643
|
+
## Auto Guard
|
|
644
|
+
|
|
645
|
+
Action tools (`mouse_click`, `mouse_drag`, `keyboard(action='type'/'press')`, `click_element`, `desktop_act`, `browser_click`, `browser_navigate`) automatically guard each action when you pass `windowTitle` / `tabId`:
|
|
646
|
+
|
|
647
|
+
- Verifies target window identity (process restart / HWND replacement detected)
|
|
648
|
+
- Confirms click coordinates are inside the target window rect
|
|
649
|
+
- Returns `post.perception.status` on every response — including failures — so the LLM can recover without a screenshot
|
|
650
|
+
|
|
651
|
+
**Disabling auto guard** — set `DESKTOP_TOUCH_AUTO_GUARD=0` to restore v0.11.12 behavior (no auto guard):
|
|
652
|
+
|
|
653
|
+
```json
|
|
654
|
+
{
|
|
655
|
+
"mcpServers": {
|
|
656
|
+
"desktop-touch": {
|
|
657
|
+
"type": "stdio",
|
|
658
|
+
"command": "npx",
|
|
659
|
+
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
660
|
+
"env": {
|
|
661
|
+
"DESKTOP_TOUCH_AUTO_GUARD": "0"
|
|
662
|
+
}
|
|
663
|
+
}
|
|
664
|
+
}
|
|
665
|
+
}
|
|
666
|
+
```
|
|
667
|
+
|
|
668
|
+
When auto guard is enabled (default), `post.perception.status` will be one of:
|
|
669
|
+
|
|
670
|
+
| Status | Meaning |
|
|
671
|
+
|---|---|
|
|
672
|
+
| `ok` | Guard passed — target verified |
|
|
673
|
+
| `unguarded` | `windowTitle` not provided; action ran without guard |
|
|
674
|
+
| `target_not_found` | No window matched the given title |
|
|
675
|
+
| `ambiguous_target` | Multiple windows matched; use a more specific title |
|
|
676
|
+
| `identity_changed` | Window was replaced (process restart / HWND change) |
|
|
677
|
+
| `unsafe_coordinates` | Click coordinates are outside the target window rect |
|
|
678
|
+
| `needs_escalation` | Use `browser_click` or specify `windowTitle` |
|
|
679
|
+
|
|
680
|
+
When `unsafe_coordinates` or `identity_changed` is returned, the response may include a `suggestedFix.fixId`. Pass that `fixId` to the relevant tool call to approve the recovery:
|
|
681
|
+
|
|
682
|
+
```json
|
|
683
|
+
{ "name": "mouse_click", "arguments": { "fixId": "fix-..." } }
|
|
684
|
+
{ "name": "keyboard(action='type')", "arguments": { "fixId": "fix-...", "text": "hello" } }
|
|
685
|
+
{ "name": "click_element", "arguments": { "fixId": "fix-..." } }
|
|
686
|
+
{ "name": "browser_click", "arguments": { "fixId": "fix-..." } }
|
|
687
|
+
```
|
|
688
|
+
|
|
689
|
+
The fix is one-shot and expires in 15 seconds. The server revalidates the target process identity before executing.
|
|
690
|
+
|
|
691
|
+
---
|
|
692
|
+
|
|
693
|
+
## Advanced response options
|
|
694
|
+
|
|
695
|
+
### browser_eval Structured Mode
|
|
696
|
+
|
|
697
|
+
Pass `withPerception: true` to receive a structured JSON response with `post.perception` instead of raw text:
|
|
698
|
+
|
|
699
|
+
```json
|
|
700
|
+
{ "name": "browser_eval", "arguments": { "expression": "document.title", "withPerception": true } }
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
Returns `{ ok: true, result: "...", post: { perception: { status: "ok", ... } } }`.
|
|
704
|
+
|
|
705
|
+
### mouse_drag Cross-Window Guard
|
|
706
|
+
|
|
707
|
+
`mouse_drag` now guards both start and end coordinates. Drags that cross window boundaries (or reach the desktop wallpaper) are blocked by default. To allow intentional cross-window or range-selection drags:
|
|
708
|
+
|
|
709
|
+
```json
|
|
710
|
+
{ "name": "mouse_drag", "arguments": { "startX": 100, "startY": 100, "endX": 900, "endY": 900, "allowCrossWindowDrag": true } }
|
|
711
|
+
```
|
|
712
|
+
|
|
713
|
+
---
|
|
714
|
+
|
|
715
|
+
## Performance (v0.15 — Rust Native Engine)
|
|
716
|
+
|
|
717
|
+
The Rust native engine (`@harusame64/desktop-touch-engine`) replaces PowerShell process spawning with direct COM calls over a persistent MTA thread. It loads automatically as a `.node` addon — no configuration needed.
|
|
718
|
+
|
|
719
|
+
### UIA Benchmark (vs PowerShell baseline)
|
|
720
|
+
|
|
721
|
+
| Function | Rust Native | PowerShell | Speedup |
|
|
722
|
+
|---|---|---|---|
|
|
723
|
+
| `getFocusedElement` | **2.2 ms** | 366 ms | **163.9×** |
|
|
724
|
+
| `getUiElements` (Explorer, ~60 elements) | **106.5 ms** | 346 ms | **3.3×** |
|
|
725
|
+
| **Weighted average** | | | **~82×** |
|
|
726
|
+
|
|
727
|
+
### Image Diff Benchmark (SSE2 SIMD)
|
|
728
|
+
|
|
729
|
+
| Function | Rust (SSE2) | TypeScript | Speedup |
|
|
730
|
+
|---|---|---|---|
|
|
731
|
+
| `computeChangeFraction` (1920×1080) | **0.26 ms** | 3.8 ms | **~15×** |
|
|
732
|
+
| `dHash` (perceptual hash) | **0.09 ms** | 1.2 ms | **~13×** |
|
|
733
|
+
|
|
734
|
+
### Architecture
|
|
735
|
+
|
|
736
|
+
```
|
|
737
|
+
Claude CLI / MCP Client
|
|
738
|
+
│ stdio or HTTP (MCP protocol)
|
|
739
|
+
▼
|
|
740
|
+
desktop-touch-mcp (TypeScript)
|
|
741
|
+
│
|
|
742
|
+
├── Rust Native Engine (.node addon) ← NEW in v0.15
|
|
743
|
+
│ ├── UIA: 13 functions via napi-rs + windows-rs 0.62
|
|
744
|
+
│ │ └── Dedicated COM thread (MTA) + batch BFS algorithm
|
|
745
|
+
│ └── Image: SSE2 SIMD pixel diff + perceptual hashing
|
|
746
|
+
│
|
|
747
|
+
└── PowerShell Fallback (automatic)
|
|
748
|
+
└── Activates transparently if .node is unavailable
|
|
749
|
+
```
|
|
750
|
+
|
|
751
|
+
### Why `getUiElements` is 3.3× (not 160×)
|
|
752
|
+
|
|
753
|
+
The 160× speedup on `getFocusedElement` comes from eliminating PowerShell process startup (~200 ms) and .NET assembly loading. For `getUiElements`, the bottleneck shifts to the **UIA provider** inside the target application (e.g., Explorer) — it must enumerate its UI tree regardless of who asks. The Rust engine uses a **batch BFS algorithm** (`FindAllBuildCache` + `TreeScope_Children`) that minimizes cross-process RPC calls and supports `maxElements` early exit, making it dramatically faster on large trees (VS Code, browsers with 1000+ elements).
|
|
754
|
+
|
|
755
|
+
---
|
|
756
|
+
|
|
757
|
+
## UI Operating Layer (V2)
|
|
758
|
+
|
|
759
|
+
> **Status: Default ON since v0.17.** `desktop_discover` and `desktop_act` are available out of the box.
|
|
760
|
+
|
|
761
|
+
V2 introduces two new tools that replace coordinate-based clicking with entity-based interaction:
|
|
762
|
+
|
|
763
|
+
| Tool | Description |
|
|
764
|
+
|---|---|
|
|
765
|
+
| `desktop_discover` | Observe a window or browser tab. Returns interactive entities with leases — no raw screen coordinates. Supports UIA (native), CDP (browser), terminal, and visual GPU lanes. |
|
|
766
|
+
| `desktop_act` | Interact with an entity returned by `desktop_discover`. Validates the lease before executing. Returns a semantic diff (`entity_disappeared`, `modal_appeared`, `focus_shifted`, …). On visual-only targets a successful act can bundle a `roiCapture` (a PNG crop of the changed region + a lease-less next-target preview) so you confirm the result and find the next target in one call — controlled by `returnCapture` (`on-change`, the default on a visible change; `never` to suppress; `always` to force). |
|
|
767
|
+
|
|
768
|
+
### Clicking — priority order
|
|
769
|
+
|
|
770
|
+
When multiple tools could perform the same click, prefer them in this order:
|
|
771
|
+
|
|
772
|
+
1. `browser_click(selector)` — Chrome / Edge over CDP (stable across repaints)
|
|
773
|
+
2. `desktop_act(lease)` — native windows, dialogs, visual-only targets (entity-based; use after `desktop_discover`)
|
|
774
|
+
3. `click_element(name | automationId)` — native UIA fallback when `desktop_act` returns `ok:false`
|
|
775
|
+
4. `mouse_click(x, y)` — pixel-level last resort (`origin` + `scale` from `dotByDot` screenshots only)
|
|
776
|
+
|
|
777
|
+
### Disabling V2 (kill switch)
|
|
778
|
+
|
|
779
|
+
To hide `desktop_discover` / `desktop_act` from the tool catalog, add the disable flag and restart:
|
|
780
|
+
|
|
781
|
+
```json
|
|
782
|
+
{
|
|
783
|
+
"mcpServers": {
|
|
784
|
+
"desktop-touch": {
|
|
785
|
+
"type": "stdio",
|
|
786
|
+
"command": "npx",
|
|
787
|
+
"args": ["-y", "@harusame64/desktop-touch-mcp"],
|
|
788
|
+
"env": {
|
|
789
|
+
"DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2": "1"
|
|
790
|
+
}
|
|
791
|
+
}
|
|
792
|
+
}
|
|
793
|
+
}
|
|
794
|
+
```
|
|
795
|
+
|
|
796
|
+
All V1 tools continue to work without interruption — no reinstall required. Remove the env entry and restart to re-enable.
|
|
797
|
+
|
|
798
|
+
Flag semantics (exact-match: only the literal string `"1"` counts):
|
|
799
|
+
|
|
800
|
+
| `DISABLE_FUKUWARAI_V2` | V2 state |
|
|
801
|
+
|---|---|
|
|
802
|
+
| unset / not `"1"` | **ON** (default) |
|
|
803
|
+
| `"1"` | **OFF** (kill switch) |
|
|
804
|
+
|
|
805
|
+
### Removed: `DESKTOP_TOUCH_ENABLE_FUKUWARAI_V2`
|
|
806
|
+
|
|
807
|
+
This was the opt-in switch in v0.16.x. V2 is on by default since v0.17, so the flag no longer has any effect and is safe to delete from your config. To turn V2 off, set `DESKTOP_TOUCH_DISABLE_FUKUWARAI_V2=1`.
|
|
808
|
+
|
|
809
|
+
### Recovery when V2 fails
|
|
810
|
+
|
|
811
|
+
If `desktop_act` returns `ok: false`, read `reason` and follow the built-in recovery hints in the tool description. Common paths:
|
|
812
|
+
|
|
813
|
+
- `lease_expired` / `*_mismatch` / `entity_not_found` → re-call `desktop_discover`
|
|
814
|
+
- `modal_blocking` → `response.blockingElement` (when present) carries `{ name, role, automationId? }`; dismiss with `click_element(name=blockingElement.name)`, then retry
|
|
815
|
+
- `entity_outside_viewport` → `scroll` / `scroll(action='to_element')`, then re-call `desktop_discover`
|
|
816
|
+
- `executor_failed` → fall back to `click_element` / `mouse_click` / `browser_click`
|
|
817
|
+
|
|
818
|
+
For `desktop_discover` warnings (`visual_provider_unavailable`, `visual_provider_warming`, `cdp_provider_failed`, …), the coordinate-based tools (`screenshot(detail='text')`, `click_element`, `mouse_click`, `terminal`, …) remain available as an escape hatch.
|
|
819
|
+
|
|
820
|
+
---
|
|
821
|
+
|
|
822
|
+
## Known limitations
|
|
823
|
+
|
|
824
|
+
| Limitation | Detail | Workaround |
|
|
825
|
+
|---|---|---|
|
|
826
|
+
| Games / video players may return black or hang in PrintWindow capture | DirectX fullscreen apps may not redraw under `PW_RENDERFULLCONTENT`. Window-targeted `screenshot(detail='image')` already falls back to BitBlt automatically when PrintWindow returns no data or an all-black + zero-variance frame, but DirectX surfaces that hang the call don't surface as fallback. | Retry with `screenshot({mode:'background', fullContent:false})` to switch to the legacy PrintWindow flag; if still black, the BitBlt fallback path (default `mode='normal'`) will at least return the on-screen rect — `hints.captureFallbackReason` will say `printwindow-all-black` |
|
|
827
|
+
| UIA call overhead | ~2 ms (focus) / ~100 ms (tree) via Rust native engine; ~300 ms via PowerShell fallback | Rust engine loads automatically; `workspace_snapshot` uses a 2 s timeout internally |
|
|
828
|
+
| Chrome / WinUI3 UIA elements are empty | Chromium exposes only limited UIA | `screenshot(detail='text')` auto-detects Chromium and falls back to Windows OCR (`hints.chromiumGuard=true`). For richer DOM access use `browser_open` + `browser_locate` |
|
|
829
|
+
| Chromium title-regex misses when sites rewrite `document.title` | Guard relies on the ` - Google Chrome` suffix being present; some sites push it off the end of a long title | Title is treated as plain Chrome (UIA runs). OCR path is still reachable via `ocrFallback='always'` or when UIA returns `<5` elements (`uiaSparse`) |
|
|
830
|
+
| `browser_*` CDP tools need Chrome launched with `--remote-debugging-port` | If Chrome is already running on the default profile without the flag, `browser_open` fails. The CDP E2E suite (`tests/e2e/browser-cdp.test.ts`) will also fail in that state | Close Chrome first, then `browser_open({launch:{}})` will relaunch it in debug mode, or start Chrome manually with `--remote-debugging-port=9222 --user-data-dir=C:\tmp\cdp` |
|
|
831
|
+
| Layer buffer TTL | Buffer auto-clears after 90s of inactivity → next `diffMode` becomes an I-frame | After long waits, call `workspace_snapshot` to explicitly reset the buffer |
|
|
832
|
+
| `keyboard(action='type')` / `keyboard(action='press')` follow focus | When `window_dock(action='dock')(pin=true)` keeps another window on top (e.g. Claude CLI), keystrokes may be absorbed by that window | Call `focus_window(title=...)` first and verify `isActive=true` via `screenshot(detail='meta')` before sending keys |
|
|
833
|
+
| `keyboard(action='type')` em-dash / smart quotes in Chrome/Edge | Non-ASCII punctuation (em-dash `—`, en-dash `–`, smart quotes `"" ''`) can be intercepted as keyboard accelerators, shifting focus to the address bar | Always use `use_clipboard=true` when the text contains such characters |
|
|
834
|
+
| `browser_eval(action='js')` on React / Vue / Svelte inputs | Setting `element.value = ...` or dispatching synthetic events does not update the framework's internal state | Use `browser_fill(selector, value)` — it uses native prototype setter + InputEvent which does update React/Vue/Svelte state |
|
|
835
|
+
|
|
836
|
+
---
|
|
837
|
+
|
|
838
|
+
## Token cost reference
|
|
839
|
+
|
|
840
|
+
| Mode | Tokens | Use case |
|
|
841
|
+
|---|---|---|
|
|
842
|
+
| `screenshot` (768px PNG) | ~443 tok | General visual check |
|
|
843
|
+
| `screenshot(dotByDot=true)` window | ~800 tok | Precise clicking (no coordinate math) |
|
|
844
|
+
| `screenshot(diffMode=true)` | ~160 tok | Post-action diff |
|
|
845
|
+
| `screenshot(detail="text")` | ~100–300 tok | UI interaction (no image) |
|
|
846
|
+
| `workspace_snapshot` | ~2000 tok | Full session orientation |
|
|
847
|
+
|
|
848
|
+
---
|
|
849
|
+
|
|
850
|
+
## 🚀 5,000+ Downloads!
|
|
851
|
+
|
|
852
|
+
This project just passed **5,000+ downloads**. Huge thanks to everyone who
|
|
853
|
+
tried an experimental desktop-automation MCP server, filed issues, opened PRs,
|
|
854
|
+
and shared what broke. Every bug report made the next release better.
|
|
855
|
+
Thank you for building with me!
|
|
856
|
+
|
|
857
|
+
---
|
|
858
|
+
|
|
859
|
+
## License
|
|
860
|
+
|
|
861
|
+
MIT
|