ucu-mcp 0.5.1 → 0.5.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,23 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [0.5.2] - 2026-06-17
9
+
10
+ Agent Skill 重写为 CLI agent 优先(Claude Code / Codex / OpenCode 都是 stdio CLI 环境)。
11
+
12
+ ### Changed
13
+
14
+ - **SKILL.md 重写**:Core Workflow 改为"决策循环"状态机视角(observe → decide → act → verify),强调 CLI agent 每次工具调用无状态、必须每次重新观察。新增"Reading click results"表(v0.5.1 的 method/verified 信号解读:axpress+true 放心走 / axpress+false 复核 / coordinate+false 必须重新观察)。工具选择改为 AX-first / vision-fallback / tray 三分法。Operating Rules 第一条改为"Re-observe before every action"。
15
+ - **workflows.md 重写**:所有 playbook 改为 CLI 可执行序列(每步读响应再决定下一步)。cc-switch playbook(#2)补充真实 gotcha:click_menu_bar_extra 打开的是原生应用菜单(About/Hide/Quit),"使用统计"在 WebView 设置窗口内;托盘菜单在独立调用间会自动关闭,必须用 captureAfter 单次捕获。新增 workflow #6(verify click 结果)和 #8(卡住时的诊断顺序)。
16
+ - **troubleshooting.md**:新增"Click result signals (v0.5.1+)"节,解释 method/verified 不是错误而是置信度信号。
17
+ - **README** Agent Skill 节:明确"为 CLI agent 编写"。
18
+
19
+ ### 真机验证(按重写后的 skill workflow 重跑)
20
+
21
+ - ✅ doctor ready / focus_app(cc-switch) 建 tray target / click_menu_bar_extra 返回 method:axpress verified:true。
22
+ - ✅ 验证了 skill 的核心价值:workflow #2 的 gotcha 指引正确诊断出 cc-switch 托盘菜单只有 About/Hide/Quit(原生应用菜单),"使用统计"需开 WebView 设置窗口。
23
+ - ⚠️ 发现真实 CLI 痛点:cc-switch 托盘菜单在独立 ocr 调用间自动关闭(失焦即关),必须用 captureAfter 在单次 click_menu_bar_extra 调用里捕获——这正是 skill 新 workflow #6 强调的"单次调用捕获"模式。
24
+
8
25
  ## [0.5.1] - 2026-06-16
9
26
 
10
27
  方向4b AXPress verify-then-fallback + 方向5 FocusStealSuppression @deprecated 文档化。
package/README.md CHANGED
@@ -421,11 +421,13 @@ The same readiness report is also available as the MCP `doctor` tool.
421
421
 
422
422
  ## Agent Skill
423
423
 
424
- UCU-MCP ships an installable **agent skill** that gives Codex, Claude Code, and
425
- other agent runtimes richer guidance than the embedded MCP `instructions:` field
426
- alone structured tool-selection rules, task playbooks, and an error-recovery
427
- reference. The skill lives at [`skills/ucu-mcp/`](skills/ucu-mcp/SKILL.md) and
428
- is included in the npm tarball.
424
+ UCU-MCP ships an installable **agent skill** written for **CLI agents** (Claude
425
+ Code, Codex, OpenCode) that connect via stdio MCP and drive the desktop one
426
+ tool call at a time. It gives richer guidance than the embedded MCP
427
+ `instructions:` field: a decision loop (observe → decide → act → verify),
428
+ tool-selection rules (AX-first / vision-fallback / tray), click-result signal
429
+ interpretation (`method`/`verified`), task playbooks, and an error-recovery
430
+ reference.
429
431
 
430
432
  Install it for your agent runtime with the [`skills` CLI](https://www.npmjs.com/package/skills):
431
433
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "ucu-mcp",
3
- "version": "0.5.1",
3
+ "version": "0.5.2",
4
4
  "description": "MCP server for Universal Computer Use — desktop automation for AI agents via Model Context Protocol",
5
5
  "type": "module",
6
6
  "bin": {
@@ -5,96 +5,134 @@ description: >-
5
5
  click, type, OCR, AX element tools, menu-bar tray support). Use when an
6
6
  agent needs to automate macOS desktop apps over MCP — establishing target
7
7
  context, reading screen state, interacting with UI elements, operating
8
- menu-bar/tray apps, or recovering from AX/permission errors. Covers Claude
9
- Code CLI/Desktop, Codex, OpenCode, and other MCP clients.
8
+ menu-bar/tray apps, or recovering from AX/permission errors. Designed for
9
+ CLI agents (Claude Code, Codex, OpenCode) that connect via stdio MCP and
10
+ drive the desktop one tool call at a time.
10
11
  ---
11
12
 
12
13
  # UCU-MCP
13
14
 
14
- UCU-MCP is a cross-client computer-use MCP server for macOS (Windows/Linux are
15
- explicit stubs). It exposes 26 tools that let an agent see the screen and drive
16
- native apps through a combination of Accessibility (AX) APIs, CGEvent input
17
- synthesis, Vision OCR, and ScreenCaptureKit screenshots.
18
-
19
- - npm package: `ucu-mcp`
20
- - Run: `npx -y ucu-mcp` (stdio MCP server) or install globally via `npm i -g ucu-mcp`
21
-
22
- ## Core Workflow
23
-
24
- 1. **Check readiness** → `doctor` verifies Accessibility + Screen Recording
25
- permissions and native helpers. If anything is missing, follow its guidance
26
- (see [troubleshooting](references/troubleshooting.md)).
27
- 2. **Establish target context** → `list_apps` then `focus_app(name)` sets the
28
- active window target. Subsequent AX tools operate against that target.
29
- 3. **Prefer AX over coordinates** → `find_element(text/role/value)` →
30
- `click_element` / `type_in_element` / `set_value`. AX is precise and survives
31
- layout shifts; coordinates are a last resort.
32
- 4. **When AX is opaque (Electron/Tauri/WebView)** → `screenshot` + `ocr` to
33
- locate text by bounding box, then `click(x, y)` at the returned coordinates.
34
- 5. **When image content is not visible to you** (relayed/downgraded to a URL) →
35
- `screenshot(describe: true)` or the standalone `describe_screen` tool to get a
36
- structured text view (OCR blocks + AX tree + foreground window).
37
- 6. **Menu-bar/tray apps** (e.g. cc-switch) → `click_menu_bar_extra(app,
38
- description/name/index)` opens the tray menu, then `find_element` inside it.
39
- 7. **Verify actions** → pass `captureAfter: true` on action tools, or call
40
- `screenshot` / `get_window_state` afterwards.
41
- 8. **Recover from errors** → every error response carries a `hint` with the
42
- next step. See the [error code table](references/troubleshooting.md).
43
-
44
- Full tool inventory with parameters: [tool-reference](references/tool-reference.md).
45
- Common task playbooks: [workflows](references/workflows.md).
15
+ UCU-MCP is a macOS computer-use MCP server for CLI agents. It exposes 26 tools
16
+ that let you see the screen and drive native apps through Accessibility (AX)
17
+ APIs, CGEvent input synthesis, Vision OCR, and ScreenCaptureKit screenshots.
18
+ Windows/Linux are explicit stubs.
46
19
 
47
- ## Operating Rules
20
+ - npm: `ucu-mcp` · run: `npx -y ucu-mcp` (stdio MCP server)
21
+ - You are a **CLI agent**: each tool call is one MCP request over stdio. There
22
+ is no persistent UI session between calls — **always re-observe state before
23
+ acting**, because the user or the app may have changed the screen since your
24
+ last call.
48
25
 
49
- - **AX-first.** Use `find_element` `click_element` / `type_in_element` /
50
- `set_value` whenever the AX tree exposes the target. Fall back to coordinates
51
- only when AX returns nothing (Electron/WebView) or the control silently
52
- swallows AX actions.
53
- - **Observe before acting.** Call `screenshot` / `get_window_state` /
54
- `describe_screen` before destructive or hard-to-reverse actions so you act on
55
- current state, not assumptions.
56
- - **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
57
- retry — the element cache refetches equivalent AX nodes.
58
- - **Tray apps need `click_menu_bar_extra`.** `focus_app` alone cannot reach
59
- pure menu-bar (LSUIElement) apps; their status item is hosted by
60
- `SystemUIServer` and is not in any app window's AX tree.
61
- - **Dangerous actions are blocked.** Quit/logout/lock shortcuts (`cmd+q`,
62
- `cmd+shift+q`, `cmd+l`, …), sensitive-window URLs, and suspicious injected
63
- text are rejected by the safety guard. Choose a safer action or ask the user.
64
- - **Sensitive fields are masked in `describe_screen`.** Password fields
65
- (`AXSecureTextField`, or names matching `/password|secret|token/i`) appear as
66
- `[REDACTED]` — never try to read or exfiltrate them.
67
- - **macOS is locked → actions blocked.** The server refuses to synthesize input
68
- while the screen is locked; wait for unlock or ask the user.
69
-
70
- ## MCP Config
71
-
72
- Add UCU-MCP to your MCP client. Stdio transport, no arguments needed.
73
-
74
- **Codex / generic TOML:**
75
-
76
- ```toml
77
- [mcp_servers.ucu-mcp]
78
- command = "npx"
79
- args = ["-y", "ucu-mcp"]
80
- ```
26
+ ## The Decision Loop (run this for every action)
81
27
 
82
- **Claude Code CLI / Desktop** add via `claude mcp add`:
28
+ Think in cycles of **observe decide act verify**. Do not chain actions
29
+ blindly; the desktop is a moving target.
83
30
 
84
- ```bash
85
- claude mcp add ucu-mcp -- npx -y ucu-mcp
86
31
  ```
32
+ ┌─────────────────────────────────────────────────────────────┐
33
+ │ OBSERVE: what's on screen / what's focused right now? │
34
+ │ screenshot{} · describe_screen{} · get_window_state{} │
35
+ ├─────────────────────────────────────────────────────────────┤
36
+ │ DECIDE: AX-first, coordinates only as fallback │
37
+ │ AX tree exposes target? → find_element → element tools │
38
+ │ AX opaque (Electron/Tauri)? → ocr → click(x,y) │
39
+ │ Tray-only app? → click_menu_bar_extra │
40
+ ├─────────────────────────────────────────────────────────────┤
41
+ │ ACT: click_element / type_in_element / set_value / click │
42
+ │ Pass captureAfter:true to get a screenshot in the reply │
43
+ ├─────────────────────────────────────────────────────────────┤
44
+ │ VERIFY: did it work? (see "Reading click results" below) │
45
+ │ result.verified === true → proceed │
46
+ │ result.verified === false → screenshot/get_window_state │
47
+ │ result.method === "coordinate" → re-observe, may be off │
48
+ └─────────────────────────────────────────────────────────────┘
49
+ ```
50
+
51
+ ## Reading click results (v0.5.1+)
52
+
53
+ `click_element` and `click_menu_bar_extra` return a `result` object with
54
+ `method` and `verified` fields. **Read them every time** — they tell you
55
+ whether your click actually landed:
56
+
57
+ | `method` | `verified` | Meaning | What you do |
58
+ |---|---|---|---|
59
+ | `"axpress"` | `true` | AXPress changed observable state (value/focused/selected) | Proceed — high confidence it worked |
60
+ | `"axpress"` | `false` | AXPress ran but element had no observable state to verify (e.g. plain button) | Verify via `screenshot` or `get_window_state` |
61
+ | `"coordinate"` | `false` | AXPress was silently swallowed (Tauri/Electron) OR threw; fell back to coordinate click | **Always re-observe** — coordinate clicks can miss, or the app may need a second click |
62
+
63
+ A `warnings[]` array in the receipt explains the fallback. Never assume a
64
+ coordinate-fallback click succeeded without checking.
65
+
66
+ ## Tool selection — AX vs vision vs tray
67
+
68
+ **AX-first** (precise, survives layout shifts):
69
+ `find_element` → `click_element` / `type_in_element` / `set_value`. Use when
70
+ the app exposes an AX tree (native macOS apps, most non-Electron apps).
71
+
72
+ **Vision fallback** (when AX is opaque — Electron/Tauri/WebView return an
73
+ empty `AXGroup` or `find_element` returns 0 with an "app is likely Electron"
74
+ hint):
75
+ `screenshot` → `ocr` → compute click point from the OCR block's bounding box
76
+ → `click(x, y)` at `block.x + block.width/2, block.y + block.height/2`.
87
77
 
88
- Run `ucu-mcp doctor` once after first connect to verify macOS permissions
89
- (System Settings Privacy & Security → Accessibility **and** Screen Recording
90
- must be granted to the launching terminal/client).
78
+ **Text-only fallback** (when you cannot see image content blocks — relay
79
+ downgrades them to URLs):
80
+ `describe_screen` or `screenshot({describe: true})` structured text with OCR
81
+ blocks + AX tree. Password fields are masked to `[REDACTED]`.
82
+
83
+ **Tray apps** (menu-bar-only / LSUIElement apps like cc-switch — no window, no
84
+ AX tree entry):
85
+ `focus_app` (falls back to a tray target) → `click_menu_bar_extra` opens the
86
+ menu → `find_element` inside the menu, or `screenshot`+`ocr` if the menu is
87
+ also opaque.
88
+
89
+ ## First-run setup
90
+
91
+ 1. **Connect the server** to your CLI agent:
92
+
93
+ Codex / generic TOML (`.codex/config.toml` or equivalent):
94
+ ```toml
95
+ [mcp_servers.ucu-mcp]
96
+ command = "npx"
97
+ args = ["-y", "ucu-mcp"]
98
+ ```
99
+
100
+ Claude Code CLI:
101
+ ```bash
102
+ claude mcp add ucu-mcp -- npx -y ucu-mcp
103
+ ```
104
+
105
+ 2. **Grant macOS permissions** — Accessibility **and** Screen Recording must be
106
+ enabled for your terminal/client in System Settings → Privacy & Security.
107
+ **Restart the client after granting** (changes don't apply to running
108
+ processes).
109
+
110
+ 3. **Verify** — call `doctor`. It reports per-permission status, native helper
111
+ health, and which process to authorize. Green = ready.
112
+
113
+ ## Operating Rules
114
+
115
+ - **Re-observe before every action.** The screen changes between your calls.
116
+ A `focus_app` from 5 calls ago may be stale; a window may have closed.
117
+ - **AX-first, coordinates only as fallback.** AX clicks are precise and
118
+ verifiable; coordinate clicks can drift and are unverifiable.
119
+ - **`verified:false` means re-observe.** Never trust an unverifiable click
120
+ without a follow-up `screenshot` or `get_window_state`.
121
+ - **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
122
+ retry. `type_in_element` auto-refetches equivalent AX nodes.
123
+ - **Dangerous actions are blocked.** `cmd+q`, `cmd+shift+q`, `cmd+l`, `alt+f4`,
124
+ sensitive-window URLs, and suspicious injected text are rejected. Choose a
125
+ safer action or ask the user.
126
+ - **macOS locked → all input blocked.** Wait for unlock; there is no bypass.
127
+ - **Don't exfiltrate passwords.** `describe_screen` masks them to
128
+ `[REDACTED]`; respect that.
91
129
 
92
130
  ## References
93
131
 
94
132
  - [tool-reference.md](references/tool-reference.md) — all 26 tools, parameters,
95
133
  return shapes, and when to use each.
96
- - [workflows.md](references/workflows.md) — playbooks for common tasks: form
134
+ - [workflows.md](references/workflows.md) — CLI-executable playbooks (form
97
135
  filling, tray apps, opaque Electron UIs, vision-degraded environments, stale
98
- targets.
136
+ targets, click-result verification).
99
137
  - [troubleshooting.md](references/troubleshooting.md) — error code table with
100
138
  recovery steps, permission issues, AX-opacity workarounds, OCR failures.
@@ -30,6 +30,30 @@ to recovery steps (mirrors the runtime `recoveryHint`).
30
30
 
31
31
  ---
32
32
 
33
+ ## Click result signals (v0.5.1+)
34
+
35
+ `click_element` and `click_menu_bar_extra` return `result.method` and
36
+ `result.verified`. These are not errors — they tell you how confident to be:
37
+
38
+ - **`method:"axpress", verified:true`** — AXPress changed observable state.
39
+ Proceed normally.
40
+ - **`method:"axpress", verified:false`** — AXPress ran but the element exposed
41
+ no observable state (plain button, no value/focused/selected). It *probably*
42
+ worked, but re-observe with `screenshot`/`get_window_state` to be sure. A
43
+ `warnings[]` entry explains this.
44
+ - **`method:"coordinate", verified:false`** — AXPress was silently swallowed
45
+ (Tauri/Electron custom controls) or threw; ucu-mcp fell back to a coordinate
46
+ click at the element's bounds center. **Coordinate clicks can miss** — always
47
+ re-observe. A `warnings[]` entry says "coordinate fallback was used".
48
+
49
+ This is not a bug — it's the server telling you it couldn't fully confirm the
50
+ click. The verify-then-fallback logic exists *because* Tauri/Electron controls
51
+ silently swallow AXPress without throwing.
52
+
53
+ ---
54
+
55
+ ---
56
+
33
57
  ## Permission Issues
34
58
 
35
59
  macOS requires two permissions for full functionality:
@@ -1,146 +1,209 @@
1
1
  # Workflows
2
2
 
3
- Common task playbooks. Each shows the preferred tool sequence and the fallback
4
- path when the primary path is blocked.
3
+ CLI-executable playbooks. Each is a sequence of tool calls you make one at a
4
+ time, reading each response before the next. Coordinates are screen-absolute
5
+ unless noted.
5
6
 
6
7
  ---
7
8
 
8
- ## 1. Fill a form field
9
+ ## 1. Fill a form field (native app, AX-visible)
9
10
 
10
- **Primary (AX):**
11
11
  ```
12
- focus_app("Safari")
13
- find_element({ text: "Email", role: "AXTextField" })
14
- elementId "Safari/w0/42"
12
+ # 1. establish target
13
+ list_apps({})
14
+ focus_app({ app: "Safari" })
15
+
16
+ # 2. locate the field via AX
17
+ find_element({ text: "Email", role: "AXTextField", app: "Safari" })
18
+ # → response.result.results[0].id e.g. "Safari/w0/42"
19
+
20
+ # 3. type into it
15
21
  type_in_element({ elementId: "Safari/w0/42", text: "user@example.com" })
16
- ```
17
22
 
18
- **Fallback (AX value set, for non-text controls):**
19
- ```
20
- set_value({ elementId: "...", value: "option" })
23
+ # 4. verify (captureAfter returns a screenshot in the same reply)
24
+ type_in_element({ elementId: "Safari/w0/42", text: "...", captureAfter: true })
21
25
  ```
22
26
 
23
- **Fallback (coordinates, when AX is opaque):**
24
- ```
25
- screenshot({})
26
- ocr({}) → blocks[].text === "Email"{x, y, width, height}
27
- click({ x: block.x + block.width/2, y: block.y + block.height/2 })
28
- type_text({ text: "user@example.com" })
29
- ```
27
+ **If `find_element` returns 0** with an "app is likely Electron" hint → switch
28
+ to workflow #3 (vision fallback).
29
+
30
+ **If `type_in_element` throws TARGET_STALE**`focus_app("Safari")` then retry
31
+ (it auto-refetches equivalent AX nodes).
30
32
 
31
33
  ---
32
34
 
33
35
  ## 2. Operate a menu-bar / tray app (e.g. cc-switch)
34
36
 
35
- Tray apps' status items live in `SystemUIServer`, not the app's own window AX
36
- tree. `focus_app` alone may return `WINDOW_NOT_FOUND`.
37
+ Tray-only apps (LSUIElement) have no window; their status item is hosted by
38
+ `SystemUIServer`. `focus_app` alone returns `WINDOW_NOT_FOUND` unless ucu-mcp
39
+ finds a tray status item and falls back to a tray target.
37
40
 
38
41
  ```
39
- focus_app("cc-switch") # establishes tray target if status item found
40
- click_menu_bar_extra({ app: "cc-switch", name: "switch" }) # opens the menu
41
- # menu is now open — find items inside it:
42
- find_element({ text: "使用统计", app: "cc-switch" })
43
- → elementId
44
- click_element({ elementId })
45
- ```
42
+ # 1. establish tray target (returns windowId:"tray" on success)
43
+ focus_app({ app: "cc-switch" })
46
44
 
47
- If the menu's AX tree is opaque (some Tauri/Electron menus):
48
- ```
49
- click_menu_bar_extra({ app: "cc-switch" })
45
+ # 2. open the tray menu check result.verified!
46
+ click_menu_bar_extra({ app: "cc-switch", name: "switch" })
47
+ # → result: { clicked: true, method: "axpress", verified: true|false }
48
+
49
+ # 3a. IF the menu exposes AX items, find and click them:
50
+ find_element({ text: "Settings", app: "cc-switch" })
51
+ click_element({ elementId: "..." })
52
+
53
+ # 3b. IF find_element returns 0 (menu is also opaque), use vision:
50
54
  screenshot({})
51
- ocr({}) → locate "使用统计" by text → coordinates
52
- click({ x, y })
55
+ ocr({})
56
+ # → find the menu item text in ocr.blocks, compute center, click:
57
+ click({ x: block.x + block.width/2, y: block.y + block.height/2 })
53
58
  ```
54
59
 
60
+ **Known gotcha (cc-switch and similar Tauri tray apps):** `click_menu_bar_extra`
61
+ may open the app's **native application menu** (About / Hide / Quit) rather than
62
+ a custom tray popup. If OCR shows only About/Hide/Quit, the app's real settings
63
+ live in a **WebView window** — you need to open that window first (via a menu
64
+ item, or `focus_app` once a window exists), then drive it with workflow #3
65
+ (Electron-opaque). Don't keep clicking the tray expecting a custom menu.
66
+
67
+ **If `click_menu_bar_extra` returns `method:"coordinate"`** (AXPress was
68
+ swallowed) → re-observe with `screenshot` to confirm the menu actually opened
69
+ before searching its contents.
70
+
55
71
  ---
56
72
 
57
- ## 3. Electron / WebView opaque UI
73
+ ## 3. Electron / Tauri / WebView opaque UI
58
74
 
59
- Electron/Tauri apps often expose only a near-empty `AXGroup`. The runtime `hint`
60
- on `find_element` and `list_windows` tells you when this is happening.
75
+ Electron/Tauri apps render UI in a composited layer AX cannot introspect.
76
+ `find_element` returns 0; `get_window_state` shows a near-empty `AXGroup`;
77
+ `list_windows` emits an Electron hint.
61
78
 
62
79
  ```
80
+ # 1. confirm opacity (optional — the hint in the error tells you)
63
81
  find_element({ text: "Submit" })
64
- → 0 results, hint: "app is likely Electron... screenshot → ocr → click(x,y)"
82
+ # → 0 results, hint: "...likely Electron... screenshot → ocr → click(x,y)"
65
83
 
66
- screenshot({})
67
- ocr({ region: { x, y, width, height } }) # or full screen
68
- → blocks[].text === "Submit" → {x, y, width, height}
84
+ # 2. see the screen via OCR (text + bounding boxes)
85
+ screenshot({}) # for yourself, if you can see images
86
+ ocr({}) # → blocks[].text with {x, y, width, height}
87
+
88
+ # 3. locate target text, compute click center, click by coordinate
89
+ # (OCR coordinates are screen-absolute)
69
90
  click({ x: block.x + block.width/2, y: block.y + block.height/2 })
91
+
92
+ # 4. verify the coordinate click landed (coordinate clicks are unverifiable)
93
+ screenshot({}) # or describe_screen({}) if you can't see images
70
94
  ```
71
95
 
72
- For repeated interaction with a known-opaque app, snapshot once with
73
- `describe_screen` to plan, then drive by coordinates.
96
+ For multi-step interaction with a known-opaque app, call `describe_screen` once
97
+ to plan, then drive by coordinates. Keep re-OCR-ing between steps — WebView
98
+ layouts shift.
74
99
 
75
100
  ---
76
101
 
77
- ## 4. Vision-degraded environment (image content not visible)
102
+ ## 4. Vision-degraded environment (you can't see image content)
78
103
 
79
- When the model cannot see `screenshot` image blocks (relay/downgrade to URLs),
80
- switch to text-based screen reading:
104
+ When `screenshot` image blocks are downgraded to URLs you cannot fetch, switch
105
+ to text-based screen reading:
81
106
 
82
107
  ```
83
108
  describe_screen({ ocr: true, includeAx: true })
84
- → { screen, foregroundWindow, ocr:{blocks}, ax:{elements}, errors }
109
+ # → { capturedAt, screen, foregroundWindow, ocr:{blocks, status}, ax:{elements, status}, errors }
85
110
 
86
- # or, if you also want the image for clients that DO support it:
87
- screenshot({ describe: true })
88
- → [image block, text description block]
111
+ # OCR blocks give you text + screen coordinates drive by click(x,y)
112
+ # AX elements give you a tree you can find_element/click_element on
89
113
  ```
90
114
 
91
- `describe_screen` never throws — OCR and AX each try/catch independently, so a
92
- Vision failure still returns AX state and vice versa. Check `errors[]` to know
93
- what was skipped/failed.
115
+ `describe_screen` **never throws** — OCR and AX each try/catch independently.
116
+ Check `errors[]` to see what was skipped/failed, and `ocr.status` / `ax.status`
117
+ (`"ok"` / `"skipped"` / `"failed"`). If OCR failed, fall back to AX-only
118
+ (`includeAx: true, ocr: false`); if AX failed, fall back to OCR-only.
119
+
120
+ `screenshot({ describe: true })` returns both an image (for clients that can
121
+ see it) **and** a text description block — use this when you're unsure whether
122
+ your client renders images.
94
123
 
95
124
  ---
96
125
 
97
126
  ## 5. Recover from TARGET_STALE
98
127
 
99
- The active window target can go stale (window closed, app restarted, pid
100
- changed). AX tools throw `TARGET_STALE`.
128
+ The active window target goes stale when the window closes, the app restarts,
129
+ or the pid changes. AX tools throw `TARGET_STALE` (receipt includes a `hint`).
101
130
 
102
131
  ```
103
- # error response includes hint: "Run focus_app again for the target app..."
104
- focus_app("Safari") # re-establishes target
105
- find_element({ text: "Save" }) # retry — cache refetches equivalent nodes
106
- click_element({ elementId })
132
+ # 1. re-establish the target
133
+ focus_app({ app: "Safari" })
134
+
135
+ # 2. retry — element cache refetches equivalent AX nodes
136
+ find_element({ text: "Save" })
137
+ click_element({ elementId: "..." })
107
138
  ```
108
139
 
109
- `type_in_element` automatically refetches an equivalent AX node if the original
110
- `elementId` is stale, so a single retry often succeeds without `focus_app`.
140
+ `type_in_element` auto-refetches an equivalent AX node if the `elementId` is
141
+ stale, so a single retry often succeeds without re-running `focus_app`.
111
142
 
112
143
  ---
113
144
 
114
- ## 6. Verify an action succeeded
145
+ ## 6. Verify a click succeeded (v0.5.1+)
115
146
 
116
- Always verify after clicks/types UI may not have updated, or the wrong element
117
- was hit.
147
+ Every `click_element` / `click_menu_bar_extra` response includes `result.method`
148
+ and `result.verified`. Use them to decide whether to trust the click:
118
149
 
119
150
  ```
120
- click_element({ elementId, captureAfter: true }) # screenshot in response
121
- # or explicitly:
122
- screenshot({})
123
- # or check AX state:
124
- get_window_state({}) → focusedElement / tree reflects the change
125
- # or wait for a specific change:
126
- wait_for_element({ text: "Saved", until: "appear", timeout: 3000 })
151
+ click_element({ elementId: "btn1" })
152
+ # result: { clicked: true, method: "axpress", verified: true }
153
+ # ✓ AXPress changed observable state — proceed with confidence
154
+
155
+ # OR
156
+ # result: { clicked: true, method: "coordinate", verified: false }
157
+ # warnings: ["AXPress produced no observable state change...coordinate fallback..."]
158
+ # ⚠ MUST re-observe — coordinate clicks can miss or hit the wrong spot:
159
+ screenshot({}) # see where you are now
160
+ # or
161
+ get_window_state({}) # check AX state changed as expected
162
+ # or
163
+ wait_for_element({ text: "Success", until: "appear", timeout: 3000 })
127
164
  ```
128
165
 
166
+ **Rule: `verified:false` always triggers a follow-up observation.** Do not chain
167
+ another action on top of an unverifiable click.
168
+
129
169
  ---
130
170
 
131
- ## 7. Multi-step task with error recovery
171
+ ## 7. Multi-step task with full loop
172
+
173
+ A realistic CLI sequence with observe → decide → act → verify at each step:
132
174
 
133
175
  ```
134
- doctor() # verify permissions first
135
- list_apps()
136
- focus_app("Notes")
137
- find_element({ text: "New Note" }) → id
138
- click_element({ elementId: id, captureAfter: true })
139
-
140
- # if click_element throws ELEMENT_NOT_FOUND:
141
- find_element({ text: "New Note" }) → id2 # refetch, id may have changed
176
+ # setup
177
+ doctor({}) # permissions + helpers green?
178
+ list_apps({})
179
+ focus_app({ app: "Notes" })
180
+
181
+ # step 1: find and click "New Note"
182
+ find_element({ text: "New Note" }) # id "Notes/w0/3"
183
+ click_element({ elementId: "Notes/w0/3", captureAfter: true })
184
+ # read result.verified; if false → screenshot to confirm a new note opened
185
+
186
+ # step 2: if ELEMENT_NOT_FOUND (UI shifted), refetch
187
+ find_element({ text: "New Note" }) # → id2 (may differ)
142
188
  click_element({ elementId: id2 })
143
189
 
144
- type_in_element({ elementId: bodyId, text: "Hello" })
145
- screenshot({}) # confirm content
190
+ # step 3: type into the new note body
191
+ find_element({ role: "AXTextArea" }) # bodyId
192
+ type_in_element({ elementId: bodyId, text: "Hello", captureAfter: true })
193
+
194
+ # step 4: confirm
195
+ screenshot({}) # or describe_screen({}) if vision-degraded
146
196
  ```
197
+
198
+ ---
199
+
200
+ ## 8. When you're stuck — diagnostic order
201
+
202
+ If tools keep failing and you don't know why:
203
+
204
+ 1. `doctor({})` — permissions/helpers still green? (Screen may have re-locked.)
205
+ 2. `list_apps({})` + `list_windows({})` — is the target app/window still there?
206
+ 3. `screenshot({})` or `describe_screen({})` — what's *actually* on screen right
207
+ now? (You may be looking at a different app than you think.)
208
+ 4. Read the `hint` in the last error response — it names the recovery step.
209
+ 5. Check [troubleshooting.md](troubleshooting.md) for the error code.