ucu-mcp 0.5.1 → 0.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +17 -0
- package/README.md +7 -5
- package/package.json +1 -1
- package/skills/ucu-mcp/SKILL.md +113 -75
- package/skills/ucu-mcp/references/troubleshooting.md +24 -0
- package/skills/ucu-mcp/references/workflows.md +143 -80
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.5.2] - 2026-06-17
|
|
9
|
+
|
|
10
|
+
Agent Skill 重写为 CLI agent 优先(Claude Code / Codex / OpenCode 都是 stdio CLI 环境)。
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- **SKILL.md 重写**:Core Workflow 改为"决策循环"状态机视角(observe → decide → act → verify),强调 CLI agent 每次工具调用无状态、必须每次重新观察。新增"Reading click results"表(v0.5.1 的 method/verified 信号解读:axpress+true 放心走 / axpress+false 复核 / coordinate+false 必须重新观察)。工具选择改为 AX-first / vision-fallback / tray 三分法。Operating Rules 第一条改为"Re-observe before every action"。
|
|
15
|
+
- **workflows.md 重写**:所有 playbook 改为 CLI 可执行序列(每步读响应再决定下一步)。cc-switch playbook(#2)补充真实 gotcha:click_menu_bar_extra 打开的是原生应用菜单(About/Hide/Quit),"使用统计"在 WebView 设置窗口内;托盘菜单在独立调用间会自动关闭,必须用 captureAfter 单次捕获。新增 workflow #6(verify click 结果)和 #8(卡住时的诊断顺序)。
|
|
16
|
+
- **troubleshooting.md**:新增"Click result signals (v0.5.1+)"节,解释 method/verified 不是错误而是置信度信号。
|
|
17
|
+
- **README** Agent Skill 节:明确"为 CLI agent 编写"。
|
|
18
|
+
|
|
19
|
+
### 真机验证(按重写后的 skill workflow 重跑)
|
|
20
|
+
|
|
21
|
+
- ✅ doctor ready / focus_app(cc-switch) 建 tray target / click_menu_bar_extra 返回 method:axpress verified:true。
|
|
22
|
+
- ✅ 验证了 skill 的核心价值:workflow #2 的 gotcha 指引正确诊断出 cc-switch 托盘菜单只有 About/Hide/Quit(原生应用菜单),"使用统计"需开 WebView 设置窗口。
|
|
23
|
+
- ⚠️ 发现真实 CLI 痛点:cc-switch 托盘菜单在独立 ocr 调用间自动关闭(失焦即关),必须用 captureAfter 在单次 click_menu_bar_extra 调用里捕获——这正是 skill 新 workflow #6 强调的"单次调用捕获"模式。
|
|
24
|
+
|
|
8
25
|
## [0.5.1] - 2026-06-16
|
|
9
26
|
|
|
10
27
|
方向4b AXPress verify-then-fallback + 方向5 FocusStealSuppression @deprecated 文档化。
|
package/README.md
CHANGED
|
@@ -421,11 +421,13 @@ The same readiness report is also available as the MCP `doctor` tool.
|
|
|
421
421
|
|
|
422
422
|
## Agent Skill
|
|
423
423
|
|
|
424
|
-
UCU-MCP ships an installable **agent skill**
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
424
|
+
UCU-MCP ships an installable **agent skill** written for **CLI agents** (Claude
|
|
425
|
+
Code, Codex, OpenCode) that connect via stdio MCP and drive the desktop one
|
|
426
|
+
tool call at a time. It gives richer guidance than the embedded MCP
|
|
427
|
+
`instructions:` field: a decision loop (observe → decide → act → verify),
|
|
428
|
+
tool-selection rules (AX-first / vision-fallback / tray), click-result signal
|
|
429
|
+
interpretation (`method`/`verified`), task playbooks, and an error-recovery
|
|
430
|
+
reference.
|
|
429
431
|
|
|
430
432
|
Install it for your agent runtime with the [`skills` CLI](https://www.npmjs.com/package/skills):
|
|
431
433
|
|
package/package.json
CHANGED
package/skills/ucu-mcp/SKILL.md
CHANGED
|
@@ -5,96 +5,134 @@ description: >-
|
|
|
5
5
|
click, type, OCR, AX element tools, menu-bar tray support). Use when an
|
|
6
6
|
agent needs to automate macOS desktop apps over MCP — establishing target
|
|
7
7
|
context, reading screen state, interacting with UI elements, operating
|
|
8
|
-
menu-bar/tray apps, or recovering from AX/permission errors.
|
|
9
|
-
Code
|
|
8
|
+
menu-bar/tray apps, or recovering from AX/permission errors. Designed for
|
|
9
|
+
CLI agents (Claude Code, Codex, OpenCode) that connect via stdio MCP and
|
|
10
|
+
drive the desktop one tool call at a time.
|
|
10
11
|
---
|
|
11
12
|
|
|
12
13
|
# UCU-MCP
|
|
13
14
|
|
|
14
|
-
UCU-MCP is a
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
- npm package: `ucu-mcp`
|
|
20
|
-
- Run: `npx -y ucu-mcp` (stdio MCP server) or install globally via `npm i -g ucu-mcp`
|
|
21
|
-
|
|
22
|
-
## Core Workflow
|
|
23
|
-
|
|
24
|
-
1. **Check readiness** → `doctor` verifies Accessibility + Screen Recording
|
|
25
|
-
permissions and native helpers. If anything is missing, follow its guidance
|
|
26
|
-
(see [troubleshooting](references/troubleshooting.md)).
|
|
27
|
-
2. **Establish target context** → `list_apps` then `focus_app(name)` sets the
|
|
28
|
-
active window target. Subsequent AX tools operate against that target.
|
|
29
|
-
3. **Prefer AX over coordinates** → `find_element(text/role/value)` →
|
|
30
|
-
`click_element` / `type_in_element` / `set_value`. AX is precise and survives
|
|
31
|
-
layout shifts; coordinates are a last resort.
|
|
32
|
-
4. **When AX is opaque (Electron/Tauri/WebView)** → `screenshot` + `ocr` to
|
|
33
|
-
locate text by bounding box, then `click(x, y)` at the returned coordinates.
|
|
34
|
-
5. **When image content is not visible to you** (relayed/downgraded to a URL) →
|
|
35
|
-
`screenshot(describe: true)` or the standalone `describe_screen` tool to get a
|
|
36
|
-
structured text view (OCR blocks + AX tree + foreground window).
|
|
37
|
-
6. **Menu-bar/tray apps** (e.g. cc-switch) → `click_menu_bar_extra(app,
|
|
38
|
-
description/name/index)` opens the tray menu, then `find_element` inside it.
|
|
39
|
-
7. **Verify actions** → pass `captureAfter: true` on action tools, or call
|
|
40
|
-
`screenshot` / `get_window_state` afterwards.
|
|
41
|
-
8. **Recover from errors** → every error response carries a `hint` with the
|
|
42
|
-
next step. See the [error code table](references/troubleshooting.md).
|
|
43
|
-
|
|
44
|
-
Full tool inventory with parameters: [tool-reference](references/tool-reference.md).
|
|
45
|
-
Common task playbooks: [workflows](references/workflows.md).
|
|
15
|
+
UCU-MCP is a macOS computer-use MCP server for CLI agents. It exposes 26 tools
|
|
16
|
+
that let you see the screen and drive native apps through Accessibility (AX)
|
|
17
|
+
APIs, CGEvent input synthesis, Vision OCR, and ScreenCaptureKit screenshots.
|
|
18
|
+
Windows/Linux are explicit stubs.
|
|
46
19
|
|
|
47
|
-
|
|
20
|
+
- npm: `ucu-mcp` · run: `npx -y ucu-mcp` (stdio MCP server)
|
|
21
|
+
- You are a **CLI agent**: each tool call is one MCP request over stdio. There
|
|
22
|
+
is no persistent UI session between calls — **always re-observe state before
|
|
23
|
+
acting**, because the user or the app may have changed the screen since your
|
|
24
|
+
last call.
|
|
48
25
|
|
|
49
|
-
|
|
50
|
-
`set_value` whenever the AX tree exposes the target. Fall back to coordinates
|
|
51
|
-
only when AX returns nothing (Electron/WebView) or the control silently
|
|
52
|
-
swallows AX actions.
|
|
53
|
-
- **Observe before acting.** Call `screenshot` / `get_window_state` /
|
|
54
|
-
`describe_screen` before destructive or hard-to-reverse actions so you act on
|
|
55
|
-
current state, not assumptions.
|
|
56
|
-
- **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
|
|
57
|
-
retry — the element cache refetches equivalent AX nodes.
|
|
58
|
-
- **Tray apps need `click_menu_bar_extra`.** `focus_app` alone cannot reach
|
|
59
|
-
pure menu-bar (LSUIElement) apps; their status item is hosted by
|
|
60
|
-
`SystemUIServer` and is not in any app window's AX tree.
|
|
61
|
-
- **Dangerous actions are blocked.** Quit/logout/lock shortcuts (`cmd+q`,
|
|
62
|
-
`cmd+shift+q`, `cmd+l`, …), sensitive-window URLs, and suspicious injected
|
|
63
|
-
text are rejected by the safety guard. Choose a safer action or ask the user.
|
|
64
|
-
- **Sensitive fields are masked in `describe_screen`.** Password fields
|
|
65
|
-
(`AXSecureTextField`, or names matching `/password|secret|token/i`) appear as
|
|
66
|
-
`[REDACTED]` — never try to read or exfiltrate them.
|
|
67
|
-
- **macOS is locked → actions blocked.** The server refuses to synthesize input
|
|
68
|
-
while the screen is locked; wait for unlock or ask the user.
|
|
69
|
-
|
|
70
|
-
## MCP Config
|
|
71
|
-
|
|
72
|
-
Add UCU-MCP to your MCP client. Stdio transport, no arguments needed.
|
|
73
|
-
|
|
74
|
-
**Codex / generic TOML:**
|
|
75
|
-
|
|
76
|
-
```toml
|
|
77
|
-
[mcp_servers.ucu-mcp]
|
|
78
|
-
command = "npx"
|
|
79
|
-
args = ["-y", "ucu-mcp"]
|
|
80
|
-
```
|
|
26
|
+
## The Decision Loop (run this for every action)
|
|
81
27
|
|
|
82
|
-
|
|
28
|
+
Think in cycles of **observe → decide → act → verify**. Do not chain actions
|
|
29
|
+
blindly; the desktop is a moving target.
|
|
83
30
|
|
|
84
|
-
```bash
|
|
85
|
-
claude mcp add ucu-mcp -- npx -y ucu-mcp
|
|
86
31
|
```
|
|
32
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
33
|
+
│ OBSERVE: what's on screen / what's focused right now? │
|
|
34
|
+
│ screenshot{} · describe_screen{} · get_window_state{} │
|
|
35
|
+
├─────────────────────────────────────────────────────────────┤
|
|
36
|
+
│ DECIDE: AX-first, coordinates only as fallback │
|
|
37
|
+
│ AX tree exposes target? → find_element → element tools │
|
|
38
|
+
│ AX opaque (Electron/Tauri)? → ocr → click(x,y) │
|
|
39
|
+
│ Tray-only app? → click_menu_bar_extra │
|
|
40
|
+
├─────────────────────────────────────────────────────────────┤
|
|
41
|
+
│ ACT: click_element / type_in_element / set_value / click │
|
|
42
|
+
│ Pass captureAfter:true to get a screenshot in the reply │
|
|
43
|
+
├─────────────────────────────────────────────────────────────┤
|
|
44
|
+
│ VERIFY: did it work? (see "Reading click results" below) │
|
|
45
|
+
│ result.verified === true → proceed │
|
|
46
|
+
│ result.verified === false → screenshot/get_window_state │
|
|
47
|
+
│ result.method === "coordinate" → re-observe, may be off │
|
|
48
|
+
└─────────────────────────────────────────────────────────────┘
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
## Reading click results (v0.5.1+)
|
|
52
|
+
|
|
53
|
+
`click_element` and `click_menu_bar_extra` return a `result` object with
|
|
54
|
+
`method` and `verified` fields. **Read them every time** — they tell you
|
|
55
|
+
whether your click actually landed:
|
|
56
|
+
|
|
57
|
+
| `method` | `verified` | Meaning | What you do |
|
|
58
|
+
|---|---|---|---|
|
|
59
|
+
| `"axpress"` | `true` | AXPress changed observable state (value/focused/selected) | Proceed — high confidence it worked |
|
|
60
|
+
| `"axpress"` | `false` | AXPress ran but element had no observable state to verify (e.g. plain button) | Verify via `screenshot` or `get_window_state` |
|
|
61
|
+
| `"coordinate"` | `false` | AXPress was silently swallowed (Tauri/Electron) OR threw; fell back to coordinate click | **Always re-observe** — coordinate clicks can miss, or the app may need a second click |
|
|
62
|
+
|
|
63
|
+
A `warnings[]` array in the receipt explains the fallback. Never assume a
|
|
64
|
+
coordinate-fallback click succeeded without checking.
|
|
65
|
+
|
|
66
|
+
## Tool selection — AX vs vision vs tray
|
|
67
|
+
|
|
68
|
+
**AX-first** (precise, survives layout shifts):
|
|
69
|
+
`find_element` → `click_element` / `type_in_element` / `set_value`. Use when
|
|
70
|
+
the app exposes an AX tree (native macOS apps, most non-Electron apps).
|
|
71
|
+
|
|
72
|
+
**Vision fallback** (when AX is opaque — Electron/Tauri/WebView return an
|
|
73
|
+
empty `AXGroup` or `find_element` returns 0 with an "app is likely Electron"
|
|
74
|
+
hint):
|
|
75
|
+
`screenshot` → `ocr` → compute click point from the OCR block's bounding box
|
|
76
|
+
→ `click(x, y)` at `block.x + block.width/2, block.y + block.height/2`.
|
|
87
77
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
78
|
+
**Text-only fallback** (when you cannot see image content blocks — relay
|
|
79
|
+
downgrades them to URLs):
|
|
80
|
+
`describe_screen` or `screenshot({describe: true})` → structured text with OCR
|
|
81
|
+
blocks + AX tree. Password fields are masked to `[REDACTED]`.
|
|
82
|
+
|
|
83
|
+
**Tray apps** (menu-bar-only / LSUIElement apps like cc-switch — no window, no
|
|
84
|
+
AX tree entry):
|
|
85
|
+
`focus_app` (falls back to a tray target) → `click_menu_bar_extra` opens the
|
|
86
|
+
menu → `find_element` inside the menu, or `screenshot`+`ocr` if the menu is
|
|
87
|
+
also opaque.
|
|
88
|
+
|
|
89
|
+
## First-run setup
|
|
90
|
+
|
|
91
|
+
1. **Connect the server** to your CLI agent:
|
|
92
|
+
|
|
93
|
+
Codex / generic TOML (`.codex/config.toml` or equivalent):
|
|
94
|
+
```toml
|
|
95
|
+
[mcp_servers.ucu-mcp]
|
|
96
|
+
command = "npx"
|
|
97
|
+
args = ["-y", "ucu-mcp"]
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Claude Code CLI:
|
|
101
|
+
```bash
|
|
102
|
+
claude mcp add ucu-mcp -- npx -y ucu-mcp
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
2. **Grant macOS permissions** — Accessibility **and** Screen Recording must be
|
|
106
|
+
enabled for your terminal/client in System Settings → Privacy & Security.
|
|
107
|
+
**Restart the client after granting** (changes don't apply to running
|
|
108
|
+
processes).
|
|
109
|
+
|
|
110
|
+
3. **Verify** — call `doctor`. It reports per-permission status, native helper
|
|
111
|
+
health, and which process to authorize. Green = ready.
|
|
112
|
+
|
|
113
|
+
## Operating Rules
|
|
114
|
+
|
|
115
|
+
- **Re-observe before every action.** The screen changes between your calls.
|
|
116
|
+
A `focus_app` from 5 calls ago may be stale; a window may have closed.
|
|
117
|
+
- **AX-first, coordinates only as fallback.** AX clicks are precise and
|
|
118
|
+
verifiable; coordinate clicks can drift and are unverifiable.
|
|
119
|
+
- **`verified:false` means re-observe.** Never trust an unverifiable click
|
|
120
|
+
without a follow-up `screenshot` or `get_window_state`.
|
|
121
|
+
- **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
|
|
122
|
+
retry. `type_in_element` auto-refetches equivalent AX nodes.
|
|
123
|
+
- **Dangerous actions are blocked.** `cmd+q`, `cmd+shift+q`, `cmd+l`, `alt+f4`,
|
|
124
|
+
sensitive-window URLs, and suspicious injected text are rejected. Choose a
|
|
125
|
+
safer action or ask the user.
|
|
126
|
+
- **macOS locked → all input blocked.** Wait for unlock; there is no bypass.
|
|
127
|
+
- **Don't exfiltrate passwords.** `describe_screen` masks them to
|
|
128
|
+
`[REDACTED]`; respect that.
|
|
91
129
|
|
|
92
130
|
## References
|
|
93
131
|
|
|
94
132
|
- [tool-reference.md](references/tool-reference.md) — all 26 tools, parameters,
|
|
95
133
|
return shapes, and when to use each.
|
|
96
|
-
- [workflows.md](references/workflows.md) — playbooks
|
|
134
|
+
- [workflows.md](references/workflows.md) — CLI-executable playbooks (form
|
|
97
135
|
filling, tray apps, opaque Electron UIs, vision-degraded environments, stale
|
|
98
|
-
targets.
|
|
136
|
+
targets, click-result verification).
|
|
99
137
|
- [troubleshooting.md](references/troubleshooting.md) — error code table with
|
|
100
138
|
recovery steps, permission issues, AX-opacity workarounds, OCR failures.
|
|
@@ -30,6 +30,30 @@ to recovery steps (mirrors the runtime `recoveryHint`).
|
|
|
30
30
|
|
|
31
31
|
---
|
|
32
32
|
|
|
33
|
+
## Click result signals (v0.5.1+)
|
|
34
|
+
|
|
35
|
+
`click_element` and `click_menu_bar_extra` return `result.method` and
|
|
36
|
+
`result.verified`. These are not errors — they tell you how confident to be:
|
|
37
|
+
|
|
38
|
+
- **`method:"axpress", verified:true`** — AXPress changed observable state.
|
|
39
|
+
Proceed normally.
|
|
40
|
+
- **`method:"axpress", verified:false`** — AXPress ran but the element exposed
|
|
41
|
+
no observable state (plain button, no value/focused/selected). It *probably*
|
|
42
|
+
worked, but re-observe with `screenshot`/`get_window_state` to be sure. A
|
|
43
|
+
`warnings[]` entry explains this.
|
|
44
|
+
- **`method:"coordinate", verified:false`** — AXPress was silently swallowed
|
|
45
|
+
(Tauri/Electron custom controls) or threw; ucu-mcp fell back to a coordinate
|
|
46
|
+
click at the element's bounds center. **Coordinate clicks can miss** — always
|
|
47
|
+
re-observe. A `warnings[]` entry says "coordinate fallback was used".
|
|
48
|
+
|
|
49
|
+
This is not a bug — it's the server telling you it couldn't fully confirm the
|
|
50
|
+
click. The verify-then-fallback logic exists *because* Tauri/Electron controls
|
|
51
|
+
silently swallow AXPress without throwing.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
33
57
|
## Permission Issues
|
|
34
58
|
|
|
35
59
|
macOS requires two permissions for full functionality:
|
|
@@ -1,146 +1,209 @@
|
|
|
1
1
|
# Workflows
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
CLI-executable playbooks. Each is a sequence of tool calls you make one at a
|
|
4
|
+
time, reading each response before the next. Coordinates are screen-absolute
|
|
5
|
+
unless noted.
|
|
5
6
|
|
|
6
7
|
---
|
|
7
8
|
|
|
8
|
-
## 1. Fill a form field
|
|
9
|
+
## 1. Fill a form field (native app, AX-visible)
|
|
9
10
|
|
|
10
|
-
**Primary (AX):**
|
|
11
11
|
```
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
12
|
+
# 1. establish target
|
|
13
|
+
list_apps({})
|
|
14
|
+
focus_app({ app: "Safari" })
|
|
15
|
+
|
|
16
|
+
# 2. locate the field via AX
|
|
17
|
+
find_element({ text: "Email", role: "AXTextField", app: "Safari" })
|
|
18
|
+
# → response.result.results[0].id e.g. "Safari/w0/42"
|
|
19
|
+
|
|
20
|
+
# 3. type into it
|
|
15
21
|
type_in_element({ elementId: "Safari/w0/42", text: "user@example.com" })
|
|
16
|
-
```
|
|
17
22
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
set_value({ elementId: "...", value: "option" })
|
|
23
|
+
# 4. verify (captureAfter returns a screenshot in the same reply)
|
|
24
|
+
type_in_element({ elementId: "Safari/w0/42", text: "...", captureAfter: true })
|
|
21
25
|
```
|
|
22
26
|
|
|
23
|
-
**
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
type_text({ text: "user@example.com" })
|
|
29
|
-
```
|
|
27
|
+
**If `find_element` returns 0** with an "app is likely Electron" hint → switch
|
|
28
|
+
to workflow #3 (vision fallback).
|
|
29
|
+
|
|
30
|
+
**If `type_in_element` throws TARGET_STALE** → `focus_app("Safari")` then retry
|
|
31
|
+
(it auto-refetches equivalent AX nodes).
|
|
30
32
|
|
|
31
33
|
---
|
|
32
34
|
|
|
33
35
|
## 2. Operate a menu-bar / tray app (e.g. cc-switch)
|
|
34
36
|
|
|
35
|
-
Tray apps
|
|
36
|
-
|
|
37
|
+
Tray-only apps (LSUIElement) have no window; their status item is hosted by
|
|
38
|
+
`SystemUIServer`. `focus_app` alone returns `WINDOW_NOT_FOUND` unless ucu-mcp
|
|
39
|
+
finds a tray status item and falls back to a tray target.
|
|
37
40
|
|
|
38
41
|
```
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
# menu is now open — find items inside it:
|
|
42
|
-
find_element({ text: "使用统计", app: "cc-switch" })
|
|
43
|
-
→ elementId
|
|
44
|
-
click_element({ elementId })
|
|
45
|
-
```
|
|
42
|
+
# 1. establish tray target (returns windowId:"tray" on success)
|
|
43
|
+
focus_app({ app: "cc-switch" })
|
|
46
44
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
45
|
+
# 2. open the tray menu — check result.verified!
|
|
46
|
+
click_menu_bar_extra({ app: "cc-switch", name: "switch" })
|
|
47
|
+
# → result: { clicked: true, method: "axpress", verified: true|false }
|
|
48
|
+
|
|
49
|
+
# 3a. IF the menu exposes AX items, find and click them:
|
|
50
|
+
find_element({ text: "Settings", app: "cc-switch" })
|
|
51
|
+
click_element({ elementId: "..." })
|
|
52
|
+
|
|
53
|
+
# 3b. IF find_element returns 0 (menu is also opaque), use vision:
|
|
50
54
|
screenshot({})
|
|
51
|
-
ocr({})
|
|
52
|
-
|
|
55
|
+
ocr({})
|
|
56
|
+
# → find the menu item text in ocr.blocks, compute center, click:
|
|
57
|
+
click({ x: block.x + block.width/2, y: block.y + block.height/2 })
|
|
53
58
|
```
|
|
54
59
|
|
|
60
|
+
**Known gotcha (cc-switch and similar Tauri tray apps):** `click_menu_bar_extra`
|
|
61
|
+
may open the app's **native application menu** (About / Hide / Quit) rather than
|
|
62
|
+
a custom tray popup. If OCR shows only About/Hide/Quit, the app's real settings
|
|
63
|
+
live in a **WebView window** — you need to open that window first (via a menu
|
|
64
|
+
item, or `focus_app` once a window exists), then drive it with workflow #3
|
|
65
|
+
(Electron-opaque). Don't keep clicking the tray expecting a custom menu.
|
|
66
|
+
|
|
67
|
+
**If `click_menu_bar_extra` returns `method:"coordinate"`** (AXPress was
|
|
68
|
+
swallowed) → re-observe with `screenshot` to confirm the menu actually opened
|
|
69
|
+
before searching its contents.
|
|
70
|
+
|
|
55
71
|
---
|
|
56
72
|
|
|
57
|
-
## 3. Electron / WebView opaque UI
|
|
73
|
+
## 3. Electron / Tauri / WebView opaque UI
|
|
58
74
|
|
|
59
|
-
Electron/Tauri apps
|
|
60
|
-
|
|
75
|
+
Electron/Tauri apps render UI in a composited layer AX cannot introspect.
|
|
76
|
+
`find_element` returns 0; `get_window_state` shows a near-empty `AXGroup`;
|
|
77
|
+
`list_windows` emits an Electron hint.
|
|
61
78
|
|
|
62
79
|
```
|
|
80
|
+
# 1. confirm opacity (optional — the hint in the error tells you)
|
|
63
81
|
find_element({ text: "Submit" })
|
|
64
|
-
|
|
82
|
+
# → 0 results, hint: "...likely Electron... screenshot → ocr → click(x,y)"
|
|
65
83
|
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
84
|
+
# 2. see the screen via OCR (text + bounding boxes)
|
|
85
|
+
screenshot({}) # for yourself, if you can see images
|
|
86
|
+
ocr({}) # → blocks[].text with {x, y, width, height}
|
|
87
|
+
|
|
88
|
+
# 3. locate target text, compute click center, click by coordinate
|
|
89
|
+
# (OCR coordinates are screen-absolute)
|
|
69
90
|
click({ x: block.x + block.width/2, y: block.y + block.height/2 })
|
|
91
|
+
|
|
92
|
+
# 4. verify the coordinate click landed (coordinate clicks are unverifiable)
|
|
93
|
+
screenshot({}) # or describe_screen({}) if you can't see images
|
|
70
94
|
```
|
|
71
95
|
|
|
72
|
-
For
|
|
73
|
-
|
|
96
|
+
For multi-step interaction with a known-opaque app, call `describe_screen` once
|
|
97
|
+
to plan, then drive by coordinates. Keep re-OCR-ing between steps — WebView
|
|
98
|
+
layouts shift.
|
|
74
99
|
|
|
75
100
|
---
|
|
76
101
|
|
|
77
|
-
## 4. Vision-degraded environment (image content
|
|
102
|
+
## 4. Vision-degraded environment (you can't see image content)
|
|
78
103
|
|
|
79
|
-
When
|
|
80
|
-
|
|
104
|
+
When `screenshot` image blocks are downgraded to URLs you cannot fetch, switch
|
|
105
|
+
to text-based screen reading:
|
|
81
106
|
|
|
82
107
|
```
|
|
83
108
|
describe_screen({ ocr: true, includeAx: true })
|
|
84
|
-
|
|
109
|
+
# → { capturedAt, screen, foregroundWindow, ocr:{blocks, status}, ax:{elements, status}, errors }
|
|
85
110
|
|
|
86
|
-
#
|
|
87
|
-
|
|
88
|
-
→ [image block, text description block]
|
|
111
|
+
# OCR blocks give you text + screen coordinates — drive by click(x,y)
|
|
112
|
+
# AX elements give you a tree you can find_element/click_element on
|
|
89
113
|
```
|
|
90
114
|
|
|
91
|
-
`describe_screen` never throws — OCR and AX each try/catch independently
|
|
92
|
-
|
|
93
|
-
|
|
115
|
+
`describe_screen` **never throws** — OCR and AX each try/catch independently.
|
|
116
|
+
Check `errors[]` to see what was skipped/failed, and `ocr.status` / `ax.status`
|
|
117
|
+
(`"ok"` / `"skipped"` / `"failed"`). If OCR failed, fall back to AX-only
|
|
118
|
+
(`includeAx: true, ocr: false`); if AX failed, fall back to OCR-only.
|
|
119
|
+
|
|
120
|
+
`screenshot({ describe: true })` returns both an image (for clients that can
|
|
121
|
+
see it) **and** a text description block — use this when you're unsure whether
|
|
122
|
+
your client renders images.
|
|
94
123
|
|
|
95
124
|
---
|
|
96
125
|
|
|
97
126
|
## 5. Recover from TARGET_STALE
|
|
98
127
|
|
|
99
|
-
The active window target
|
|
100
|
-
|
|
128
|
+
The active window target goes stale when the window closes, the app restarts,
|
|
129
|
+
or the pid changes. AX tools throw `TARGET_STALE` (receipt includes a `hint`).
|
|
101
130
|
|
|
102
131
|
```
|
|
103
|
-
#
|
|
104
|
-
focus_app("Safari")
|
|
105
|
-
|
|
106
|
-
|
|
132
|
+
# 1. re-establish the target
|
|
133
|
+
focus_app({ app: "Safari" })
|
|
134
|
+
|
|
135
|
+
# 2. retry — element cache refetches equivalent AX nodes
|
|
136
|
+
find_element({ text: "Save" })
|
|
137
|
+
click_element({ elementId: "..." })
|
|
107
138
|
```
|
|
108
139
|
|
|
109
|
-
`type_in_element`
|
|
110
|
-
|
|
140
|
+
`type_in_element` auto-refetches an equivalent AX node if the `elementId` is
|
|
141
|
+
stale, so a single retry often succeeds without re-running `focus_app`.
|
|
111
142
|
|
|
112
143
|
---
|
|
113
144
|
|
|
114
|
-
## 6. Verify
|
|
145
|
+
## 6. Verify a click succeeded (v0.5.1+)
|
|
115
146
|
|
|
116
|
-
|
|
117
|
-
|
|
147
|
+
Every `click_element` / `click_menu_bar_extra` response includes `result.method`
|
|
148
|
+
and `result.verified`. Use them to decide whether to trust the click:
|
|
118
149
|
|
|
119
150
|
```
|
|
120
|
-
click_element({ elementId
|
|
121
|
-
#
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
#
|
|
126
|
-
|
|
151
|
+
click_element({ elementId: "btn1" })
|
|
152
|
+
# → result: { clicked: true, method: "axpress", verified: true }
|
|
153
|
+
# ✓ AXPress changed observable state — proceed with confidence
|
|
154
|
+
|
|
155
|
+
# OR
|
|
156
|
+
# → result: { clicked: true, method: "coordinate", verified: false }
|
|
157
|
+
# warnings: ["AXPress produced no observable state change...coordinate fallback..."]
|
|
158
|
+
# ⚠ MUST re-observe — coordinate clicks can miss or hit the wrong spot:
|
|
159
|
+
screenshot({}) # see where you are now
|
|
160
|
+
# or
|
|
161
|
+
get_window_state({}) # check AX state changed as expected
|
|
162
|
+
# or
|
|
163
|
+
wait_for_element({ text: "Success", until: "appear", timeout: 3000 })
|
|
127
164
|
```
|
|
128
165
|
|
|
166
|
+
**Rule: `verified:false` always triggers a follow-up observation.** Do not chain
|
|
167
|
+
another action on top of an unverifiable click.
|
|
168
|
+
|
|
129
169
|
---
|
|
130
170
|
|
|
131
|
-
## 7. Multi-step task with
|
|
171
|
+
## 7. Multi-step task with full loop
|
|
172
|
+
|
|
173
|
+
A realistic CLI sequence with observe → decide → act → verify at each step:
|
|
132
174
|
|
|
133
175
|
```
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
#
|
|
141
|
-
|
|
176
|
+
# setup
|
|
177
|
+
doctor({}) # permissions + helpers green?
|
|
178
|
+
list_apps({})
|
|
179
|
+
focus_app({ app: "Notes" })
|
|
180
|
+
|
|
181
|
+
# step 1: find and click "New Note"
|
|
182
|
+
find_element({ text: "New Note" }) # → id "Notes/w0/3"
|
|
183
|
+
click_element({ elementId: "Notes/w0/3", captureAfter: true })
|
|
184
|
+
# read result.verified; if false → screenshot to confirm a new note opened
|
|
185
|
+
|
|
186
|
+
# step 2: if ELEMENT_NOT_FOUND (UI shifted), refetch
|
|
187
|
+
find_element({ text: "New Note" }) # → id2 (may differ)
|
|
142
188
|
click_element({ elementId: id2 })
|
|
143
189
|
|
|
144
|
-
|
|
145
|
-
|
|
190
|
+
# step 3: type into the new note body
|
|
191
|
+
find_element({ role: "AXTextArea" }) # → bodyId
|
|
192
|
+
type_in_element({ elementId: bodyId, text: "Hello", captureAfter: true })
|
|
193
|
+
|
|
194
|
+
# step 4: confirm
|
|
195
|
+
screenshot({}) # or describe_screen({}) if vision-degraded
|
|
146
196
|
```
|
|
197
|
+
|
|
198
|
+
---
|
|
199
|
+
|
|
200
|
+
## 8. When you're stuck — diagnostic order
|
|
201
|
+
|
|
202
|
+
If tools keep failing and you don't know why:
|
|
203
|
+
|
|
204
|
+
1. `doctor({})` — permissions/helpers still green? (Screen may have re-locked.)
|
|
205
|
+
2. `list_apps({})` + `list_windows({})` — is the target app/window still there?
|
|
206
|
+
3. `screenshot({})` or `describe_screen({})` — what's *actually* on screen right
|
|
207
|
+
now? (You may be looking at a different app than you think.)
|
|
208
|
+
4. Read the `hint` in the last error response — it names the recovery step.
|
|
209
|
+
5. Check [troubleshooting.md](troubleshooting.md) for the error code.
|