ucu-mcp 0.5.0 → 0.5.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -5,6 +5,54 @@ All notable changes to this project will be documented in this file.
5
5
  The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
6
6
  and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
7
 
8
+ ## [0.5.2] - 2026-06-17
9
+
10
+ Agent Skill 重写为 CLI agent 优先(Claude Code / Codex / OpenCode 都是 stdio CLI 环境)。
11
+
12
+ ### Changed
13
+
14
+ - **SKILL.md 重写**:Core Workflow 改为"决策循环"状态机视角(observe → decide → act → verify),强调 CLI agent 每次工具调用无状态、必须每次重新观察。新增"Reading click results"表(v0.5.1 的 method/verified 信号解读:axpress+true 放心走 / axpress+false 复核 / coordinate+false 必须重新观察)。工具选择改为 AX-first / vision-fallback / tray 三分法。Operating Rules 第一条改为"Re-observe before every action"。
15
+ - **workflows.md 重写**:所有 playbook 改为 CLI 可执行序列(每步读响应再决定下一步)。cc-switch playbook(#2)补充真实 gotcha:click_menu_bar_extra 打开的是原生应用菜单(About/Hide/Quit),"使用统计"在 WebView 设置窗口内;托盘菜单在独立调用间会自动关闭,必须用 captureAfter 单次捕获。新增 workflow #6(verify click 结果)和 #8(卡住时的诊断顺序)。
16
+ - **troubleshooting.md**:新增"Click result signals (v0.5.1+)"节,解释 method/verified 不是错误而是置信度信号。
17
+ - **README** Agent Skill 节:明确"为 CLI agent 编写"。
18
+
19
+ ### 真机验证(按重写后的 skill workflow 重跑)
20
+
21
+ - ✅ doctor ready / focus_app(cc-switch) 建 tray target / click_menu_bar_extra 返回 method:axpress verified:true。
22
+ - ✅ 验证了 skill 的核心价值:workflow #2 的 gotcha 指引正确诊断出 cc-switch 托盘菜单只有 About/Hide/Quit(原生应用菜单),"使用统计"需开 WebView 设置窗口。
23
+ - ⚠️ 发现真实 CLI 痛点:cc-switch 托盘菜单在独立 ocr 调用间自动关闭(失焦即关),必须用 captureAfter 在单次 click_menu_bar_extra 调用里捕获——这正是 skill 新 workflow #6 强调的"单次调用捕获"模式。
24
+
25
+ ## [0.5.1] - 2026-06-16
26
+
27
+ 方向4b AXPress verify-then-fallback + 方向5 FocusStealSuppression @deprecated 文档化。
28
+
29
+ ### Added
30
+
31
+ - **方向4b AXPress verify-then-fallback(clickElement + clickMenuBarExtra)**:解决 Tauri/Electron 控件静默吞 AXPress(不抛异常也不执行)。双层策略:(1) 启发式跳过——对已知静默吞的应用(`AX_SILENT_APP_HINTS = ["tauri"]`,按 appName includes 匹配)直接坐标点击,跳过 AXPress;(2) verify 状态签名——对走 AXPress 的应用,采样前后 `value|focused|selected` 签名,spin 80ms 等异步生效,无变化则降级坐标。`ClickResult { method: "axpress"|"coordinate"; verified: boolean }` 透到 ActionReceipt.result,verified:false 时附 warning 引导模型用 screenshot/get_window_state 复核。无可观测状态的元素(纯按钮)保守判 verified:false 但不降级坐标(避免误伤)。共享 JXA 原语 `stateSignature`/`coordinateClick`/`spinMs`。
32
+ - 返回类型波及链:`clickElement`/`clickMenuBarExtra` `Promise<void>` → `Promise<ClickResult>`(base.ts 接口 + windows/linux stub + element.ts consumer + tools-layer mock)。
33
+
34
+ ### Changed
35
+
36
+ - **方向5 FocusStealSuppression @deprecated 文档化**:`focus.ts` 的 `saveFocus`/`restoreFocus` 加 `@deprecated` JSDoc,说明 CGEvent 在 HID 层工作不需前台 focus、焦点管理被有意禁用、`tools-layer.test.ts` 的 `not.toHaveBeenCalled()` 锁定禁用态。不删不接线(文档化设计决策,非遗漏)。`base.ts` 接口注释同步。
37
+
38
+ ### Fixed
39
+
40
+ - 无行为修复(4b/5 是 v0.5.0 的质量强化)。
41
+
42
+ ### 真机验证(2026-06-16)
43
+
44
+ - ✅ doctor 全绿(3 native helpers + 权限 + 26 工具)。
45
+ - ✅ OCR(方向1)返回非空 elements,无 NSNull 崩溃。
46
+ - ✅ press_key 字母键 Cmd+M(方向2)成功合成,不报 Unknown key。
47
+ - ✅ describe_screen(方向3)结构正确,AX 失败时聚合到 errors[] 不抛出。
48
+ - ✅ focus_app(cc-switch)(方向4a)建立 tray target(windowId:"tray"),不抛 WINDOW_NOT_FOUND。
49
+ - ✅ click_menu_bar_extra(方向4b)返回 `method:"axpress", verified:true`——托盘菜单真实打开(OCR 确认 "About/Hide/Quit CC Switch" 菜单项可见)。
50
+ - ⚠️ 发现:cc-switch 托盘菜单是其**原生应用菜单**(About/Hide/Quit),"使用统计"在 WebView 设置面板内,不在托盘菜单。完整 "设置→使用统计" 流程需先打开设置窗口(非托盘菜单可达),留 follow-up。
51
+
52
+ ### Tests
53
+
54
+ - 303→310(+7:clickElement ClickResult 解析、JXA stateSignature/spinMs/coordinateClick/sigBefore/sigAfter/preferCoord source 断言、Tauri 启发式 preferCoord=true、tools-layer method/verified 透出 + warnings)。
55
+
8
56
  ## [0.5.0] - 2026-06-15
9
57
 
10
58
  cc-switch 操控诊断与视觉通道修复:打通 OCR + 字母快捷键 + 托盘 + 视觉降级 fallback + 内置 agent skill。
package/README.md CHANGED
@@ -421,11 +421,13 @@ The same readiness report is also available as the MCP `doctor` tool.
421
421
 
422
422
  ## Agent Skill
423
423
 
424
- UCU-MCP ships an installable **agent skill** that gives Codex, Claude Code, and
425
- other agent runtimes richer guidance than the embedded MCP `instructions:` field
426
- alone structured tool-selection rules, task playbooks, and an error-recovery
427
- reference. The skill lives at [`skills/ucu-mcp/`](skills/ucu-mcp/SKILL.md) and
428
- is included in the npm tarball.
424
+ UCU-MCP ships an installable **agent skill** written for **CLI agents** (Claude
425
+ Code, Codex, OpenCode) that connect via stdio MCP and drive the desktop one
426
+ tool call at a time. It gives richer guidance than the embedded MCP
427
+ `instructions:` field: a decision loop (observe → decide → act → verify),
428
+ tool-selection rules (AX-first / vision-fallback / tray), click-result signal
429
+ interpretation (`method`/`verified`), task playbooks, and an error-recovery
430
+ reference.
429
431
 
430
432
  Install it for your agent runtime with the [`skills` CLI](https://www.npmjs.com/package/skills):
431
433
 
@@ -36,8 +36,13 @@ export function registerElementTools(registerTool) {
36
36
  }, async (params) => {
37
37
  const effectiveApp = params.app || getActiveTarget()?.appName;
38
38
  const safetyCtx = await getSafetyContext();
39
- await withSafety({ action: "click_element", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickElement(params.elementId, effectiveApp) });
40
- return actionResponse("click_element", { clicked: true, elementId: params.elementId }, { elementId: params.elementId, app: effectiveApp }, params.captureAfter, params.captureFormat, params.captureMaxWidth);
39
+ const clickResult = await withSafety({ action: "click_element", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickElement(params.elementId, effectiveApp) });
40
+ const warnings = clickResult.verified
41
+ ? []
42
+ : [clickResult.method === "coordinate"
43
+ ? "AXPress produced no observable state change (or the app is known to silently swallow it); coordinate fallback was used. Re-observe with screenshot/get_window_state to confirm."
44
+ : "AXPress completed but the element exposed no observable state to verify against. Re-observe with screenshot/get_window_state to confirm."];
45
+ return actionResponse("click_element", { clicked: true, elementId: params.elementId, method: clickResult.method, verified: clickResult.verified }, { elementId: params.elementId, app: effectiveApp }, params.captureAfter, params.captureFormat, params.captureMaxWidth, warnings);
41
46
  });
42
47
  registerTool("set_value", "Set the value of an accessibility element", {
43
48
  elementId: z.string().describe("AX element identifier"), value: z.string().describe("Value to set"), app: z.string().optional().describe("Target app"), ...captureAfterFields,
@@ -64,7 +69,12 @@ export function registerElementTools(registerTool) {
64
69
  ...captureAfterFields,
65
70
  }, async (params) => {
66
71
  const safetyCtx = await getSafetyContext();
67
- await withSafety({ action: "click_menu_bar_extra", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickMenuBarExtra(params.app, { description: params.description, name: params.name, index: params.index }) });
68
- return actionResponse("click_menu_bar_extra", { clicked: true, app: params.app }, { app: params.app }, params.captureAfter, params.captureFormat, params.captureMaxWidth);
72
+ const clickResult = await withSafety({ action: "click_menu_bar_extra", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickMenuBarExtra(params.app, { description: params.description, name: params.name, index: params.index }) });
73
+ const warnings = clickResult.verified
74
+ ? []
75
+ : [clickResult.method === "coordinate"
76
+ ? "AXPress produced no observable state change; coordinate fallback was used. Use find_element or screenshot to confirm the menu opened."
77
+ : "AXPress completed but the status item exposed no observable state to verify against. Use find_element or screenshot to confirm the menu opened."];
78
+ return actionResponse("click_menu_bar_extra", { clicked: true, app: params.app, method: clickResult.method, verified: clickResult.verified }, { app: params.app }, params.captureAfter, params.captureFormat, params.captureMaxWidth, warnings);
69
79
  });
70
80
  }
@@ -126,6 +126,17 @@ export interface WindowState {
126
126
  focusedElement?: ElementInfo;
127
127
  tree?: ElementInfo;
128
128
  }
129
+ /**
130
+ * Outcome of an AX-driven click — surfaces whether AXPress was used and whether
131
+ * its effect was observable. `verified:false` means either the element exposed
132
+ * no observable state (inconclusive) or the click fell back to coordinates
133
+ * (silent AXPress swallow on Tauri/Electron). The tool layer surfaces this so
134
+ * the model can decide whether to re-observe via screenshot/get_window_state.
135
+ */
136
+ export interface ClickResult {
137
+ method: "axpress" | "coordinate";
138
+ verified: boolean;
139
+ }
129
140
  /**
130
141
  * Structured text description of the screen — a fallback for environments where
131
142
  * image content blocks are downgraded to URLs (so the model cannot see screenshots).
@@ -169,7 +180,7 @@ export interface Platform {
169
180
  type(text: string, delay?: number): Promise<void>;
170
181
  key(keys: string[]): Promise<void>;
171
182
  findElement(options: FindElementOptions): Promise<FindElementResponse>;
172
- clickElement(elementId: string, app?: string): Promise<void>;
183
+ clickElement(elementId: string, app?: string): Promise<ClickResult>;
173
184
  typeInElement(elementId: string, text: string, app?: string, clearFirst?: boolean): Promise<void>;
174
185
  setElementValue?(elementId: string, value: string, app?: string): Promise<void>;
175
186
  findMenuBarExtra?(app: string): Promise<unknown[]>;
@@ -177,7 +188,7 @@ export interface Platform {
177
188
  description?: string;
178
189
  name?: string;
179
190
  index?: number;
180
- }): Promise<void>;
191
+ }): Promise<ClickResult>;
181
192
  isScreenLocked?(): boolean;
182
193
  saveFocus?(): Promise<void>;
183
194
  restoreFocus?(): Promise<void>;
@@ -1,4 +1,4 @@
1
- import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse } from "./base.js";
1
+ import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse, ClickResult } from "./base.js";
2
2
  /**
3
3
  * Linux platform adapter (AT-SPI2 + xdotool fallback)
4
4
  * TODO: Implement with D-Bus AT-SPI2 bindings
@@ -17,7 +17,7 @@ export declare class LinuxPlatform implements Platform {
17
17
  key(keys: string[]): Promise<void>;
18
18
  ocr(_display?: number, _region?: ScreenRegion): Promise<OcrResult>;
19
19
  findElement(_options: FindElementOptions): Promise<FindElementResponse>;
20
- clickElement(_elementId: string, _app?: string): Promise<void>;
20
+ clickElement(_elementId: string, _app?: string): Promise<ClickResult>;
21
21
  typeInElement(_elementId: string, _text: string, _app?: string, _clearFirst?: boolean): Promise<void>;
22
22
  readClipboard(): Promise<string>;
23
23
  writeClipboard(text: string): Promise<void>;
@@ -1,5 +1,6 @@
1
1
  import type { MacOSPlatform } from "./base.js";
2
- export declare function clickElement(this: MacOSPlatform, elementId: string, app?: string): Promise<void>;
2
+ import type { ClickResult } from "../base.js";
3
+ export declare function clickElement(this: MacOSPlatform, elementId: string, app?: string): Promise<ClickResult>;
3
4
  export declare function typeInElement(this: MacOSPlatform, elementId: string, text: string, app?: string, clearFirst?: boolean): Promise<void>;
4
5
  export declare function setElementValue(this: MacOSPlatform, elementId: string, value: string, app?: string): Promise<void>;
5
6
  export interface MenuBarExtraItem {
@@ -21,4 +22,4 @@ export interface MenuBarExtraSelector {
21
22
  }
22
23
  export declare function findMenuBarExtra(this: MacOSPlatform, app: string): Promise<MenuBarExtraItem[]>;
23
24
  export declare function matchMenuBarExtra(items: MenuBarExtraItem[], selector: MenuBarExtraSelector): MenuBarExtraItem | undefined;
24
- export declare function clickMenuBarExtra(this: MacOSPlatform, app: string, selector?: MenuBarExtraSelector): Promise<void>;
25
+ export declare function clickMenuBarExtra(this: MacOSPlatform, app: string, selector?: MenuBarExtraSelector): Promise<ClickResult>;
@@ -2,6 +2,20 @@ import { execFileSync } from "node:child_process";
2
2
  import { ElementNotFoundError } from "../../util/errors.js";
3
3
  import { rethrowElementActionError } from "./helpers.js";
4
4
  import { jxaElementActionHelpers } from "../jxa-helpers.js";
5
+ /**
6
+ * App-name hints for which AXPress is known to be silently swallowed
7
+ * (Tauri custom window decorations, some Electron controls). When the target
8
+ * app matches, clickElement/clickMenuBarExtra skip AXPress and go straight to
9
+ * a coordinate click. Conservative starting set — extend as more silent-swallow
10
+ * apps are confirmed.
11
+ */
12
+ const AX_SILENT_APP_HINTS = ["tauri"];
13
+ function preferCoordinateClick(appName) {
14
+ if (!appName)
15
+ return false;
16
+ const n = appName.toLowerCase();
17
+ return AX_SILENT_APP_HINTS.some((h) => n.includes(h));
18
+ }
5
19
  function prepareCache(elementId) {
6
20
  this.evictExpiredCacheEntries();
7
21
  const cached = this.elementCache.get(elementId);
@@ -10,22 +24,65 @@ function prepareCache(elementId) {
10
24
  }
11
25
  return this.elementCache.get(elementId) ?? null;
12
26
  }
27
+ /**
28
+ * Shared JXA snippet: read an observable state signature from an AX element.
29
+ * Used by clickElement/clickMenuBarExtra to verify whether AXPress actually
30
+ * changed anything (Tauri/Electron silently swallow AXPress without throwing).
31
+ * Returns a string concatenation of value/focused/selected (each defensively
32
+ * read; missing attributes contribute ''). An empty signature means the element
33
+ * exposes no observable state — verification is inconclusive.
34
+ */
35
+ const JXA_STATE_SIGNATURE = `
36
+ function stateSignature(elem) {
37
+ var parts = [];
38
+ try { var v = elem.value(); parts.push('v=' + (v === undefined || v === null ? '' : String(v))); } catch(e) { parts.push('v='); }
39
+ try { parts.push('f=' + (elem.focused ? (elem.focused() ? 1 : 0) : '')); } catch(e) { parts.push('f='); }
40
+ try { parts.push('s=' + (elem.selected ? (elem.selected() ? 1 : 0) : '')); } catch(e) { parts.push('s='); }
41
+ return parts.join('|');
42
+ }
43
+ // Brief synchronous spin to let an async AXPress propagate state.
44
+ function spinMs(ms) { var t = Date.now(); while (Date.now() - t < ms) {} }
45
+ `;
46
+ /**
47
+ * Shared JXA snippet: perform a coordinate click at the element's bounds center.
48
+ * Returns the {x, y} used, or null if bounds could not be read.
49
+ */
50
+ const JXA_COORDINATE_CLICK = `
51
+ function coordinateClick(elem) {
52
+ var pos = elem.position();
53
+ var sz = elem.size();
54
+ var cx = pos[0] + sz[0] / 2;
55
+ var cy = pos[1] + sz[1] / 2;
56
+ ObjC.import('CoreGraphics');
57
+ var src = $.CGEventSourceCreate($.kCGEventSourceStateHIDSystemState);
58
+ var pt = $.CGPointMake(cx, cy);
59
+ var down = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseDown, pt, $.kCGMouseButtonLeft);
60
+ $.CGEventPost($.kCGHIDEventTap, down);
61
+ var up = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseUp, pt, $.kCGMouseButtonLeft);
62
+ $.CGEventPost($.kCGHIDEventTap, up);
63
+ return {x: cx, y: cy};
64
+ }
65
+ `;
13
66
  export async function clickElement(elementId, app) {
14
67
  const elementIdLiteral = JSON.stringify(elementId);
15
68
  const effectiveApp = app || this.activeTarget?.appName;
16
69
  const appLiteral = JSON.stringify(effectiveApp || "");
17
70
  const cachedDescriptor = prepareCache.call(this, elementId);
18
71
  const cachedJson = JSON.stringify(cachedDescriptor);
72
+ const preferCoord = preferCoordinateClick(effectiveApp);
19
73
  const jxaScript = `
20
74
  var se = Application('System Events');
21
75
  var _result = null;
22
76
  ${jxaElementActionHelpers()}
77
+ ${JXA_STATE_SIGNATURE}
78
+ ${JXA_COORDINATE_CLICK}
23
79
  var elemPath = ${elementIdLiteral};
24
80
  var appName = ${appLiteral};
25
81
  // 容忍大小写/空格/连字符/下划线变体(cc-switch vs CC Switch vs cc_switch)
26
82
  var _norm = function(s) { return String(s||'').toLowerCase().split(' ').join('').split('-').join('').split('_').join(''); };
27
83
  var appNorm = _norm(appName);
28
84
  var cached = ${cachedJson};
85
+ var preferCoord = ${preferCoord ? "true" : "false"};
29
86
 
30
87
  var elem = resolveElementInApp(elemPath, appName) || resolveElementByFullPath(elemPath);
31
88
  if (elem && !descriptorMatches(elem)) {
@@ -37,27 +94,51 @@ export async function clickElement(elementId, app) {
37
94
 
38
95
  if (!elem) {
39
96
  _result = {success: false, error: "Element not found: " + elemPath};
97
+ } else if (preferCoord) {
98
+ // 启发式命中(Tauri 等已知静默吞 AXPress 的应用):直接坐标点击
99
+ try {
100
+ coordinateClick(elem);
101
+ _result = {success: true, method: "coordinate", verified: false};
102
+ } catch(e) {
103
+ _result = {success: false, error: "Could not click element (coordinate): " + String(e.message || e)};
104
+ }
40
105
  } else {
106
+ // AXPress + verify:采样前后状态签名,无变化则降级坐标
107
+ var sigBefore = stateSignature(elem);
108
+ var axpressThrew = false;
41
109
  try {
42
110
  elem.actions.AXPress.perform();
43
- _result = {success: true};
44
111
  } catch(e) {
112
+ axpressThrew = true;
113
+ }
114
+ if (axpressThrew) {
115
+ // AXPress 抛异常 → 坐标 fallback
45
116
  try {
46
- var pos = elem.position();
47
- var sz = elem.size();
48
- var cx = pos[0] + sz[0] / 2;
49
- var cy = pos[1] + sz[1] / 2;
50
- ObjC.import('CoreGraphics');
51
- var src = $.CGEventSourceCreate($.kCGEventSourceStateHIDSystemState);
52
- var pt = $.CGPointMake(cx, cy);
53
- var down = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseDown, pt, $.kCGMouseButtonLeft);
54
- $.CGEventPost($.kCGHIDEventTap, down);
55
- var up = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseUp, pt, $.kCGMouseButtonLeft);
56
- $.CGEventPost($.kCGHIDEventTap, up);
57
- _result = {success: true};
117
+ coordinateClick(elem);
118
+ _result = {success: true, method: "coordinate", verified: false};
58
119
  } catch(e2) {
59
120
  _result = {success: false, error: "Could not click element: " + String(e2.message || e2)};
60
121
  }
122
+ } else {
123
+ // AXPress 未抛异常:spin 等待异步生效,再采样签名判断是否真的生效
124
+ spinMs(80);
125
+ var sigAfter = stateSignature(elem);
126
+ if (sigBefore !== '' && sigAfter !== sigBefore) {
127
+ // 状态变化 → AXPress 生效
128
+ _result = {success: true, method: "axpress", verified: true};
129
+ } else if (sigBefore === '' && sigAfter === '') {
130
+ // 无可观测状态(纯按钮无 value/focused/selected):无法 verify,
131
+ // 保守认为 AXPress 可能成功(不降级坐标,避免对正常按钮误伤)
132
+ _result = {success: true, method: "axpress", verified: false};
133
+ } else {
134
+ // 有可观测状态但无变化 → 静默吞,降级坐标
135
+ try {
136
+ coordinateClick(elem);
137
+ _result = {success: true, method: "coordinate", verified: false};
138
+ } catch(e3) {
139
+ _result = {success: false, error: "Could not click element (coordinate fallback): " + String(e3.message || e3)};
140
+ }
141
+ }
61
142
  }
62
143
  }
63
144
  JSON.stringify(_result);
@@ -73,9 +154,14 @@ export async function clickElement(elementId, app) {
73
154
  ? new Error(result.error)
74
155
  : new ElementNotFoundError(elementId);
75
156
  }
157
+ return {
158
+ method: result.method ?? "axpress",
159
+ verified: Boolean(result.verified),
160
+ };
76
161
  }
77
162
  catch (error) {
78
163
  rethrowElementActionError(error, "click_element", elementId);
164
+ return { method: "axpress", verified: false }; // unreachable — rethrow throws
79
165
  }
80
166
  }
81
167
  export async function typeInElement(elementId, text, app, clearFirst) {
@@ -372,7 +458,8 @@ export async function clickMenuBarExtra(app, selector = {}) {
372
458
  const host = target.host;
373
459
  const tgtNameLiteral = JSON.stringify(target.name || "");
374
460
  const tgtDescLiteral = JSON.stringify(target.description || "");
375
- // AXPress,失败则坐标点击中心(与 clickElement 同模式,应对 Tauri 等静默吞)
461
+ const preferCoord = preferCoordinateClick(app);
462
+ // AXPress + verify-then-fallback(与 clickElement 同模式,应对 Tauri 等静默吞)。
376
463
  // host==="systemuiserver" 时在 SystemUIServer 进程上重定位(托盘 status item 由它托管),
377
464
  // SystemUIServer.menuBarItems 顺序不稳定,用保存的 name/description 二次匹配定位具体 item。
378
465
  // host==="self" 时按 app 进程的 menuBars()[mb].menuBarItems()[idx] 重定位(稳定)。
@@ -411,11 +498,14 @@ export async function clickMenuBarExtra(app, selector = {}) {
411
498
  var appName = ${appLiteral};
412
499
  var tgtName = ${tgtNameLiteral};
413
500
  var tgtDesc = ${tgtDescLiteral};
501
+ ${JXA_STATE_SIGNATURE}
502
+ ${JXA_COORDINATE_CLICK}
414
503
  // 容忍大小写/空格/连字符/下划线变体(cc-switch vs CC Switch vs cc_switch)
415
504
  var _norm = function(s) { return String(s||'').toLowerCase().split(' ').join('').split('-').join('').split('_').join(''); };
416
505
  var appNorm = _norm(appName);
417
506
  var tgtNameNorm = _norm(tgtName);
418
507
  var tgtDescNorm = _norm(tgtDesc);
508
+ var preferCoord = ${preferCoord ? "true" : "false"};
419
509
  var _matchApp = function(itemNorm) {
420
510
  if (!itemNorm) return false;
421
511
  return itemNorm === appNorm || itemNorm.indexOf(appNorm) !== -1 || appNorm.indexOf(itemNorm) !== -1;
@@ -430,21 +520,36 @@ export async function clickMenuBarExtra(app, selector = {}) {
430
520
  }
431
521
  ${resolveItemBlock}
432
522
  if (!_result && item) {
433
- try {
434
- item.actions.AXPress.perform();
435
- _result = {success: true, method: "axpress"};
436
- } catch(e) {
437
- var pos = item.position();
438
- var sz = item.size();
439
- var cx = pos[0] + sz[0] / 2;
440
- var cy = pos[1] + sz[1] / 2;
441
- ObjC.import('CoreGraphics');
442
- var pt = $.CGPointMake(cx, cy);
443
- var down = $.CGEventCreateMouseEvent(null, $.kCGEventLeftMouseDown, pt, $.kCGMouseButtonLeft);
444
- $.CGEventPost($.kCGHIDEventTap, down);
445
- var up = $.CGEventCreateMouseEvent(null, $.kCGEventLeftMouseUp, pt, $.kCGMouseButtonLeft);
446
- $.CGEventPost($.kCGHIDEventTap, up);
447
- _result = {success: true, method: "coordinate", x: cx, y: cy};
523
+ if (preferCoord) {
524
+ // 启发式命中:直接坐标点击
525
+ coordinateClick(item);
526
+ _result = {success: true, method: "coordinate", verified: false};
527
+ } else {
528
+ // AXPress + verify
529
+ var sigBefore = stateSignature(item);
530
+ var axpressThrew = false;
531
+ try {
532
+ item.actions.AXPress.perform();
533
+ } catch(e) {
534
+ axpressThrew = true;
535
+ }
536
+ if (axpressThrew) {
537
+ coordinateClick(item);
538
+ _result = {success: true, method: "coordinate", verified: false};
539
+ } else {
540
+ spinMs(80);
541
+ var sigAfter = stateSignature(item);
542
+ if (sigBefore !== '' && sigAfter !== sigBefore) {
543
+ _result = {success: true, method: "axpress", verified: true};
544
+ } else if (sigBefore === '' && sigAfter === '') {
545
+ // 托盘 status item 通常无可观测状态(无 value/selected);AXPress 开菜单的效果
546
+ // 难以从 item 本身验证。保守认为成功(菜单是否打开由后续 find_element 判断)。
547
+ _result = {success: true, method: "axpress", verified: false};
548
+ } else {
549
+ coordinateClick(item);
550
+ _result = {success: true, method: "coordinate", verified: false};
551
+ }
552
+ }
448
553
  }
449
554
  }
450
555
  } catch(e) {
@@ -458,10 +563,14 @@ export async function clickMenuBarExtra(app, selector = {}) {
458
563
  }
459
564
  catch (error) {
460
565
  rethrowElementActionError(error, "click_menu_bar_extra", app);
461
- return; // unreachable
566
+ return { method: "axpress", verified: false }; // unreachable — rethrow throws
462
567
  }
463
568
  const result = JSON.parse(out);
464
569
  if (!result.success) {
465
570
  throw new Error(`click_menu_bar_extra failed in ${app}: ${result.error}`);
466
571
  }
572
+ return {
573
+ method: result.method ?? "axpress",
574
+ verified: Boolean(result.verified),
575
+ };
467
576
  }
@@ -1,3 +1,13 @@
1
1
  import type { MacOSPlatform } from "./base.js";
2
+ /**
3
+ * @deprecated Focus management is intentionally disabled. CGEvent input
4
+ * synthesis (click/type/key) works at the HID layer and does not require the
5
+ * target app to be frontmost, so stealing/restoring focus is unnecessary and
6
+ * could disrupt the user. `saveFocus`/`restoreFocus` are retained for API
7
+ * compatibility but are NOT called from the action pipeline — see the
8
+ * `not.toHaveBeenCalled()` assertions in `tools-layer.test.ts` that lock this
9
+ * disabled state. Do not wire these into new code; prefer CGEvent-based input.
10
+ */
2
11
  export declare function saveFocus(this: MacOSPlatform): Promise<void>;
12
+ /** @deprecated See {@link saveFocus} — intentionally disabled, not called from the action pipeline. */
3
13
  export declare function restoreFocus(this: MacOSPlatform): Promise<void>;
@@ -1,4 +1,13 @@
1
1
  import { execFileSync } from "node:child_process";
2
+ /**
3
+ * @deprecated Focus management is intentionally disabled. CGEvent input
4
+ * synthesis (click/type/key) works at the HID layer and does not require the
5
+ * target app to be frontmost, so stealing/restoring focus is unnecessary and
6
+ * could disrupt the user. `saveFocus`/`restoreFocus` are retained for API
7
+ * compatibility but are NOT called from the action pipeline — see the
8
+ * `not.toHaveBeenCalled()` assertions in `tools-layer.test.ts` that lock this
9
+ * disabled state. Do not wire these into new code; prefer CGEvent-based input.
10
+ */
2
11
  export async function saveFocus() {
3
12
  try {
4
13
  const apps = await this.listApps();
@@ -16,6 +25,7 @@ export async function saveFocus() {
16
25
  this.savedFocus = undefined;
17
26
  }
18
27
  }
28
+ /** @deprecated See {@link saveFocus} — intentionally disabled, not called from the action pipeline. */
19
29
  export async function restoreFocus() {
20
30
  if (!this.savedFocus)
21
31
  return;
@@ -1,4 +1,4 @@
1
- import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse } from "./base.js";
1
+ import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse, ClickResult } from "./base.js";
2
2
  export declare class WindowsPlatform implements Platform {
3
3
  screenshot(_display?: number, _region?: ScreenRegion): Promise<Buffer>;
4
4
  getScreenSize(_display?: number): ScreenSize;
@@ -13,7 +13,7 @@ export declare class WindowsPlatform implements Platform {
13
13
  key(_keys: string[]): Promise<void>;
14
14
  ocr(_display?: number, _region?: ScreenRegion): Promise<OcrResult>;
15
15
  findElement(_options: FindElementOptions): Promise<FindElementResponse>;
16
- clickElement(_elementId: string, _app?: string): Promise<void>;
16
+ clickElement(_elementId: string, _app?: string): Promise<ClickResult>;
17
17
  typeInElement(_elementId: string, _text: string, _app?: string, _clearFirst?: boolean): Promise<void>;
18
18
  readClipboard(): Promise<string>;
19
19
  writeClipboard(text: string): Promise<void>;
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "ucu-mcp",
3
- "version": "0.5.0",
3
+ "version": "0.5.2",
4
4
  "description": "MCP server for Universal Computer Use — desktop automation for AI agents via Model Context Protocol",
5
5
  "type": "module",
6
6
  "bin": {
@@ -5,96 +5,134 @@ description: >-
5
5
  click, type, OCR, AX element tools, menu-bar tray support). Use when an
6
6
  agent needs to automate macOS desktop apps over MCP — establishing target
7
7
  context, reading screen state, interacting with UI elements, operating
8
- menu-bar/tray apps, or recovering from AX/permission errors. Covers Claude
9
- Code CLI/Desktop, Codex, OpenCode, and other MCP clients.
8
+ menu-bar/tray apps, or recovering from AX/permission errors. Designed for
9
+ CLI agents (Claude Code, Codex, OpenCode) that connect via stdio MCP and
10
+ drive the desktop one tool call at a time.
10
11
  ---
11
12
 
12
13
  # UCU-MCP
13
14
 
14
- UCU-MCP is a cross-client computer-use MCP server for macOS (Windows/Linux are
15
- explicit stubs). It exposes 26 tools that let an agent see the screen and drive
16
- native apps through a combination of Accessibility (AX) APIs, CGEvent input
17
- synthesis, Vision OCR, and ScreenCaptureKit screenshots.
18
-
19
- - npm package: `ucu-mcp`
20
- - Run: `npx -y ucu-mcp` (stdio MCP server) or install globally via `npm i -g ucu-mcp`
21
-
22
- ## Core Workflow
23
-
24
- 1. **Check readiness** → `doctor` verifies Accessibility + Screen Recording
25
- permissions and native helpers. If anything is missing, follow its guidance
26
- (see [troubleshooting](references/troubleshooting.md)).
27
- 2. **Establish target context** → `list_apps` then `focus_app(name)` sets the
28
- active window target. Subsequent AX tools operate against that target.
29
- 3. **Prefer AX over coordinates** → `find_element(text/role/value)` →
30
- `click_element` / `type_in_element` / `set_value`. AX is precise and survives
31
- layout shifts; coordinates are a last resort.
32
- 4. **When AX is opaque (Electron/Tauri/WebView)** → `screenshot` + `ocr` to
33
- locate text by bounding box, then `click(x, y)` at the returned coordinates.
34
- 5. **When image content is not visible to you** (relayed/downgraded to a URL) →
35
- `screenshot(describe: true)` or the standalone `describe_screen` tool to get a
36
- structured text view (OCR blocks + AX tree + foreground window).
37
- 6. **Menu-bar/tray apps** (e.g. cc-switch) → `click_menu_bar_extra(app,
38
- description/name/index)` opens the tray menu, then `find_element` inside it.
39
- 7. **Verify actions** → pass `captureAfter: true` on action tools, or call
40
- `screenshot` / `get_window_state` afterwards.
41
- 8. **Recover from errors** → every error response carries a `hint` with the
42
- next step. See the [error code table](references/troubleshooting.md).
43
-
44
- Full tool inventory with parameters: [tool-reference](references/tool-reference.md).
45
- Common task playbooks: [workflows](references/workflows.md).
15
+ UCU-MCP is a macOS computer-use MCP server for CLI agents. It exposes 26 tools
16
+ that let you see the screen and drive native apps through Accessibility (AX)
17
+ APIs, CGEvent input synthesis, Vision OCR, and ScreenCaptureKit screenshots.
18
+ Windows/Linux are explicit stubs.
46
19
 
47
- ## Operating Rules
20
+ - npm: `ucu-mcp` · run: `npx -y ucu-mcp` (stdio MCP server)
21
+ - You are a **CLI agent**: each tool call is one MCP request over stdio. There
22
+ is no persistent UI session between calls — **always re-observe state before
23
+ acting**, because the user or the app may have changed the screen since your
24
+ last call.
48
25
 
49
- - **AX-first.** Use `find_element` `click_element` / `type_in_element` /
50
- `set_value` whenever the AX tree exposes the target. Fall back to coordinates
51
- only when AX returns nothing (Electron/WebView) or the control silently
52
- swallows AX actions.
53
- - **Observe before acting.** Call `screenshot` / `get_window_state` /
54
- `describe_screen` before destructive or hard-to-reverse actions so you act on
55
- current state, not assumptions.
56
- - **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
57
- retry — the element cache refetches equivalent AX nodes.
58
- - **Tray apps need `click_menu_bar_extra`.** `focus_app` alone cannot reach
59
- pure menu-bar (LSUIElement) apps; their status item is hosted by
60
- `SystemUIServer` and is not in any app window's AX tree.
61
- - **Dangerous actions are blocked.** Quit/logout/lock shortcuts (`cmd+q`,
62
- `cmd+shift+q`, `cmd+l`, …), sensitive-window URLs, and suspicious injected
63
- text are rejected by the safety guard. Choose a safer action or ask the user.
64
- - **Sensitive fields are masked in `describe_screen`.** Password fields
65
- (`AXSecureTextField`, or names matching `/password|secret|token/i`) appear as
66
- `[REDACTED]` — never try to read or exfiltrate them.
67
- - **macOS is locked → actions blocked.** The server refuses to synthesize input
68
- while the screen is locked; wait for unlock or ask the user.
69
-
70
- ## MCP Config
71
-
72
- Add UCU-MCP to your MCP client. Stdio transport, no arguments needed.
73
-
74
- **Codex / generic TOML:**
75
-
76
- ```toml
77
- [mcp_servers.ucu-mcp]
78
- command = "npx"
79
- args = ["-y", "ucu-mcp"]
80
- ```
26
+ ## The Decision Loop (run this for every action)
81
27
 
82
- **Claude Code CLI / Desktop** add via `claude mcp add`:
28
+ Think in cycles of **observe decide act verify**. Do not chain actions
29
+ blindly; the desktop is a moving target.
83
30
 
84
- ```bash
85
- claude mcp add ucu-mcp -- npx -y ucu-mcp
86
31
  ```
32
+ ┌─────────────────────────────────────────────────────────────┐
33
+ │ OBSERVE: what's on screen / what's focused right now? │
34
+ │ screenshot{} · describe_screen{} · get_window_state{} │
35
+ ├─────────────────────────────────────────────────────────────┤
36
+ │ DECIDE: AX-first, coordinates only as fallback │
37
+ │ AX tree exposes target? → find_element → element tools │
38
+ │ AX opaque (Electron/Tauri)? → ocr → click(x,y) │
39
+ │ Tray-only app? → click_menu_bar_extra │
40
+ ├─────────────────────────────────────────────────────────────┤
41
+ │ ACT: click_element / type_in_element / set_value / click │
42
+ │ Pass captureAfter:true to get a screenshot in the reply │
43
+ ├─────────────────────────────────────────────────────────────┤
44
+ │ VERIFY: did it work? (see "Reading click results" below) │
45
+ │ result.verified === true → proceed │
46
+ │ result.verified === false → screenshot/get_window_state │
47
+ │ result.method === "coordinate" → re-observe, may be off │
48
+ └─────────────────────────────────────────────────────────────┘
49
+ ```
50
+
51
+ ## Reading click results (v0.5.1+)
52
+
53
+ `click_element` and `click_menu_bar_extra` return a `result` object with
54
+ `method` and `verified` fields. **Read them every time** — they tell you
55
+ whether your click actually landed:
56
+
57
+ | `method` | `verified` | Meaning | What you do |
58
+ |---|---|---|---|
59
+ | `"axpress"` | `true` | AXPress changed observable state (value/focused/selected) | Proceed — high confidence it worked |
60
+ | `"axpress"` | `false` | AXPress ran but element had no observable state to verify (e.g. plain button) | Verify via `screenshot` or `get_window_state` |
61
+ | `"coordinate"` | `false` | AXPress was silently swallowed (Tauri/Electron) OR threw; fell back to coordinate click | **Always re-observe** — coordinate clicks can miss, or the app may need a second click |
62
+
63
+ A `warnings[]` array in the receipt explains the fallback. Never assume a
64
+ coordinate-fallback click succeeded without checking.
65
+
66
+ ## Tool selection — AX vs vision vs tray
67
+
68
+ **AX-first** (precise, survives layout shifts):
69
+ `find_element` → `click_element` / `type_in_element` / `set_value`. Use when
70
+ the app exposes an AX tree (native macOS apps, most non-Electron apps).
71
+
72
+ **Vision fallback** (when AX is opaque — Electron/Tauri/WebView return an
73
+ empty `AXGroup` or `find_element` returns 0 with an "app is likely Electron"
74
+ hint):
75
+ `screenshot` → `ocr` → compute click point from the OCR block's bounding box
76
+ → `click(x, y)` at `block.x + block.width/2, block.y + block.height/2`.
87
77
 
88
- Run `ucu-mcp doctor` once after first connect to verify macOS permissions
89
- (System Settings Privacy & Security → Accessibility **and** Screen Recording
90
- must be granted to the launching terminal/client).
78
+ **Text-only fallback** (when you cannot see image content blocks — relay
79
+ downgrades them to URLs):
80
+ `describe_screen` or `screenshot({describe: true})` structured text with OCR
81
+ blocks + AX tree. Password fields are masked to `[REDACTED]`.
82
+
83
+ **Tray apps** (menu-bar-only / LSUIElement apps like cc-switch — no window, no
84
+ AX tree entry):
85
+ `focus_app` (falls back to a tray target) → `click_menu_bar_extra` opens the
86
+ menu → `find_element` inside the menu, or `screenshot`+`ocr` if the menu is
87
+ also opaque.
88
+
89
+ ## First-run setup
90
+
91
+ 1. **Connect the server** to your CLI agent:
92
+
93
+ Codex / generic TOML (`.codex/config.toml` or equivalent):
94
+ ```toml
95
+ [mcp_servers.ucu-mcp]
96
+ command = "npx"
97
+ args = ["-y", "ucu-mcp"]
98
+ ```
99
+
100
+ Claude Code CLI:
101
+ ```bash
102
+ claude mcp add ucu-mcp -- npx -y ucu-mcp
103
+ ```
104
+
105
+ 2. **Grant macOS permissions** — Accessibility **and** Screen Recording must be
106
+ enabled for your terminal/client in System Settings → Privacy & Security.
107
+ **Restart the client after granting** (changes don't apply to running
108
+ processes).
109
+
110
+ 3. **Verify** — call `doctor`. It reports per-permission status, native helper
111
+ health, and which process to authorize. Green = ready.
112
+
113
+ ## Operating Rules
114
+
115
+ - **Re-observe before every action.** The screen changes between your calls.
116
+ A `focus_app` from 5 calls ago may be stale; a window may have closed.
117
+ - **AX-first, coordinates only as fallback.** AX clicks are precise and
118
+ verifiable; coordinate clicks can drift and are unverifiable.
119
+ - **`verified:false` means re-observe.** Never trust an unverifiable click
120
+ without a follow-up `screenshot` or `get_window_state`.
121
+ - **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
122
+ retry. `type_in_element` auto-refetches equivalent AX nodes.
123
+ - **Dangerous actions are blocked.** `cmd+q`, `cmd+shift+q`, `cmd+l`, `alt+f4`,
124
+ sensitive-window URLs, and suspicious injected text are rejected. Choose a
125
+ safer action or ask the user.
126
+ - **macOS locked → all input blocked.** Wait for unlock; there is no bypass.
127
+ - **Don't exfiltrate passwords.** `describe_screen` masks them to
128
+ `[REDACTED]`; respect that.
91
129
 
92
130
  ## References
93
131
 
94
132
  - [tool-reference.md](references/tool-reference.md) — all 26 tools, parameters,
95
133
  return shapes, and when to use each.
96
- - [workflows.md](references/workflows.md) — playbooks for common tasks: form
134
+ - [workflows.md](references/workflows.md) — CLI-executable playbooks (form
97
135
  filling, tray apps, opaque Electron UIs, vision-degraded environments, stale
98
- targets.
136
+ targets, click-result verification).
99
137
  - [troubleshooting.md](references/troubleshooting.md) — error code table with
100
138
  recovery steps, permission issues, AX-opacity workarounds, OCR failures.
@@ -30,6 +30,30 @@ to recovery steps (mirrors the runtime `recoveryHint`).
30
30
 
31
31
  ---
32
32
 
33
+ ## Click result signals (v0.5.1+)
34
+
35
+ `click_element` and `click_menu_bar_extra` return `result.method` and
36
+ `result.verified`. These are not errors — they tell you how confident to be:
37
+
38
+ - **`method:"axpress", verified:true`** — AXPress changed observable state.
39
+ Proceed normally.
40
+ - **`method:"axpress", verified:false`** — AXPress ran but the element exposed
41
+ no observable state (plain button, no value/focused/selected). It *probably*
42
+ worked, but re-observe with `screenshot`/`get_window_state` to be sure. A
43
+ `warnings[]` entry explains this.
44
+ - **`method:"coordinate", verified:false`** — AXPress was silently swallowed
45
+ (Tauri/Electron custom controls) or threw; ucu-mcp fell back to a coordinate
46
+ click at the element's bounds center. **Coordinate clicks can miss** — always
47
+ re-observe. A `warnings[]` entry says "coordinate fallback was used".
48
+
49
+ This is not a bug — it's the server telling you it couldn't fully confirm the
50
+ click. The verify-then-fallback logic exists *because* Tauri/Electron controls
51
+ silently swallow AXPress without throwing.
52
+
53
+ ---
54
+
55
+ ---
56
+
33
57
  ## Permission Issues
34
58
 
35
59
  macOS requires two permissions for full functionality:
@@ -1,146 +1,209 @@
1
1
  # Workflows
2
2
 
3
- Common task playbooks. Each shows the preferred tool sequence and the fallback
4
- path when the primary path is blocked.
3
+ CLI-executable playbooks. Each is a sequence of tool calls you make one at a
4
+ time, reading each response before the next. Coordinates are screen-absolute
5
+ unless noted.
5
6
 
6
7
  ---
7
8
 
8
- ## 1. Fill a form field
9
+ ## 1. Fill a form field (native app, AX-visible)
9
10
 
10
- **Primary (AX):**
11
11
  ```
12
- focus_app("Safari")
13
- find_element({ text: "Email", role: "AXTextField" })
14
- elementId "Safari/w0/42"
12
+ # 1. establish target
13
+ list_apps({})
14
+ focus_app({ app: "Safari" })
15
+
16
+ # 2. locate the field via AX
17
+ find_element({ text: "Email", role: "AXTextField", app: "Safari" })
18
+ # → response.result.results[0].id e.g. "Safari/w0/42"
19
+
20
+ # 3. type into it
15
21
  type_in_element({ elementId: "Safari/w0/42", text: "user@example.com" })
16
- ```
17
22
 
18
- **Fallback (AX value set, for non-text controls):**
19
- ```
20
- set_value({ elementId: "...", value: "option" })
23
+ # 4. verify (captureAfter returns a screenshot in the same reply)
24
+ type_in_element({ elementId: "Safari/w0/42", text: "...", captureAfter: true })
21
25
  ```
22
26
 
23
- **Fallback (coordinates, when AX is opaque):**
24
- ```
25
- screenshot({})
26
- ocr({}) → blocks[].text === "Email"{x, y, width, height}
27
- click({ x: block.x + block.width/2, y: block.y + block.height/2 })
28
- type_text({ text: "user@example.com" })
29
- ```
27
+ **If `find_element` returns 0** with an "app is likely Electron" hint → switch
28
+ to workflow #3 (vision fallback).
29
+
30
+ **If `type_in_element` throws TARGET_STALE**`focus_app("Safari")` then retry
31
+ (it auto-refetches equivalent AX nodes).
30
32
 
31
33
  ---
32
34
 
33
35
  ## 2. Operate a menu-bar / tray app (e.g. cc-switch)
34
36
 
35
- Tray apps' status items live in `SystemUIServer`, not the app's own window AX
36
- tree. `focus_app` alone may return `WINDOW_NOT_FOUND`.
37
+ Tray-only apps (LSUIElement) have no window; their status item is hosted by
38
+ `SystemUIServer`. `focus_app` alone returns `WINDOW_NOT_FOUND` unless ucu-mcp
39
+ finds a tray status item and falls back to a tray target.
37
40
 
38
41
  ```
39
- focus_app("cc-switch") # establishes tray target if status item found
40
- click_menu_bar_extra({ app: "cc-switch", name: "switch" }) # opens the menu
41
- # menu is now open — find items inside it:
42
- find_element({ text: "使用统计", app: "cc-switch" })
43
- → elementId
44
- click_element({ elementId })
45
- ```
42
+ # 1. establish tray target (returns windowId:"tray" on success)
43
+ focus_app({ app: "cc-switch" })
46
44
 
47
- If the menu's AX tree is opaque (some Tauri/Electron menus):
48
- ```
49
- click_menu_bar_extra({ app: "cc-switch" })
45
+ # 2. open the tray menu check result.verified!
46
+ click_menu_bar_extra({ app: "cc-switch", name: "switch" })
47
+ # → result: { clicked: true, method: "axpress", verified: true|false }
48
+
49
+ # 3a. IF the menu exposes AX items, find and click them:
50
+ find_element({ text: "Settings", app: "cc-switch" })
51
+ click_element({ elementId: "..." })
52
+
53
+ # 3b. IF find_element returns 0 (menu is also opaque), use vision:
50
54
  screenshot({})
51
- ocr({}) → locate "使用统计" by text → coordinates
52
- click({ x, y })
55
+ ocr({})
56
+ # → find the menu item text in ocr.blocks, compute center, click:
57
+ click({ x: block.x + block.width/2, y: block.y + block.height/2 })
53
58
  ```
54
59
 
60
+ **Known gotcha (cc-switch and similar Tauri tray apps):** `click_menu_bar_extra`
61
+ may open the app's **native application menu** (About / Hide / Quit) rather than
62
+ a custom tray popup. If OCR shows only About/Hide/Quit, the app's real settings
63
+ live in a **WebView window** — you need to open that window first (via a menu
64
+ item, or `focus_app` once a window exists), then drive it with workflow #3
65
+ (Electron-opaque). Don't keep clicking the tray expecting a custom menu.
66
+
67
+ **If `click_menu_bar_extra` returns `method:"coordinate"`** (AXPress was
68
+ swallowed) → re-observe with `screenshot` to confirm the menu actually opened
69
+ before searching its contents.
70
+
55
71
  ---
56
72
 
57
- ## 3. Electron / WebView opaque UI
73
+ ## 3. Electron / Tauri / WebView opaque UI
58
74
 
59
- Electron/Tauri apps often expose only a near-empty `AXGroup`. The runtime `hint`
60
- on `find_element` and `list_windows` tells you when this is happening.
75
+ Electron/Tauri apps render UI in a composited layer AX cannot introspect.
76
+ `find_element` returns 0; `get_window_state` shows a near-empty `AXGroup`;
77
+ `list_windows` emits an Electron hint.
61
78
 
62
79
  ```
80
+ # 1. confirm opacity (optional — the hint in the error tells you)
63
81
  find_element({ text: "Submit" })
64
- → 0 results, hint: "app is likely Electron... screenshot → ocr → click(x,y)"
82
+ # → 0 results, hint: "...likely Electron... screenshot → ocr → click(x,y)"
65
83
 
66
- screenshot({})
67
- ocr({ region: { x, y, width, height } }) # or full screen
68
- → blocks[].text === "Submit" → {x, y, width, height}
84
+ # 2. see the screen via OCR (text + bounding boxes)
85
+ screenshot({}) # for yourself, if you can see images
86
+ ocr({}) # → blocks[].text with {x, y, width, height}
87
+
88
+ # 3. locate target text, compute click center, click by coordinate
89
+ # (OCR coordinates are screen-absolute)
69
90
  click({ x: block.x + block.width/2, y: block.y + block.height/2 })
91
+
92
+ # 4. verify the coordinate click landed (coordinate clicks are unverifiable)
93
+ screenshot({}) # or describe_screen({}) if you can't see images
70
94
  ```
71
95
 
72
- For repeated interaction with a known-opaque app, snapshot once with
73
- `describe_screen` to plan, then drive by coordinates.
96
+ For multi-step interaction with a known-opaque app, call `describe_screen` once
97
+ to plan, then drive by coordinates. Keep re-OCR-ing between steps — WebView
98
+ layouts shift.
74
99
 
75
100
  ---
76
101
 
77
- ## 4. Vision-degraded environment (image content not visible)
102
+ ## 4. Vision-degraded environment (you can't see image content)
78
103
 
79
- When the model cannot see `screenshot` image blocks (relay/downgrade to URLs),
80
- switch to text-based screen reading:
104
+ When `screenshot` image blocks are downgraded to URLs you cannot fetch, switch
105
+ to text-based screen reading:
81
106
 
82
107
  ```
83
108
  describe_screen({ ocr: true, includeAx: true })
84
- → { screen, foregroundWindow, ocr:{blocks}, ax:{elements}, errors }
109
+ # → { capturedAt, screen, foregroundWindow, ocr:{blocks, status}, ax:{elements, status}, errors }
85
110
 
86
- # or, if you also want the image for clients that DO support it:
87
- screenshot({ describe: true })
88
- → [image block, text description block]
111
+ # OCR blocks give you text + screen coordinates drive by click(x,y)
112
+ # AX elements give you a tree you can find_element/click_element on
89
113
  ```
90
114
 
91
- `describe_screen` never throws — OCR and AX each try/catch independently, so a
92
- Vision failure still returns AX state and vice versa. Check `errors[]` to know
93
- what was skipped/failed.
115
+ `describe_screen` **never throws** — OCR and AX each try/catch independently.
116
+ Check `errors[]` to see what was skipped/failed, and `ocr.status` / `ax.status`
117
+ (`"ok"` / `"skipped"` / `"failed"`). If OCR failed, fall back to AX-only
118
+ (`includeAx: true, ocr: false`); if AX failed, fall back to OCR-only.
119
+
120
+ `screenshot({ describe: true })` returns both an image (for clients that can
121
+ see it) **and** a text description block — use this when you're unsure whether
122
+ your client renders images.
94
123
 
95
124
  ---
96
125
 
97
126
  ## 5. Recover from TARGET_STALE
98
127
 
99
- The active window target can go stale (window closed, app restarted, pid
100
- changed). AX tools throw `TARGET_STALE`.
128
+ The active window target goes stale when the window closes, the app restarts,
129
+ or the pid changes. AX tools throw `TARGET_STALE` (receipt includes a `hint`).
101
130
 
102
131
  ```
103
- # error response includes hint: "Run focus_app again for the target app..."
104
- focus_app("Safari") # re-establishes target
105
- find_element({ text: "Save" }) # retry — cache refetches equivalent nodes
106
- click_element({ elementId })
132
+ # 1. re-establish the target
133
+ focus_app({ app: "Safari" })
134
+
135
+ # 2. retry — element cache refetches equivalent AX nodes
136
+ find_element({ text: "Save" })
137
+ click_element({ elementId: "..." })
107
138
  ```
108
139
 
109
- `type_in_element` automatically refetches an equivalent AX node if the original
110
- `elementId` is stale, so a single retry often succeeds without `focus_app`.
140
+ `type_in_element` auto-refetches an equivalent AX node if the `elementId` is
141
+ stale, so a single retry often succeeds without re-running `focus_app`.
111
142
 
112
143
  ---
113
144
 
114
- ## 6. Verify an action succeeded
145
+ ## 6. Verify a click succeeded (v0.5.1+)
115
146
 
116
- Always verify after clicks/types UI may not have updated, or the wrong element
117
- was hit.
147
+ Every `click_element` / `click_menu_bar_extra` response includes `result.method`
148
+ and `result.verified`. Use them to decide whether to trust the click:
118
149
 
119
150
  ```
120
- click_element({ elementId, captureAfter: true }) # screenshot in response
121
- # or explicitly:
122
- screenshot({})
123
- # or check AX state:
124
- get_window_state({}) → focusedElement / tree reflects the change
125
- # or wait for a specific change:
126
- wait_for_element({ text: "Saved", until: "appear", timeout: 3000 })
151
+ click_element({ elementId: "btn1" })
152
+ # result: { clicked: true, method: "axpress", verified: true }
153
+ # ✓ AXPress changed observable state — proceed with confidence
154
+
155
+ # OR
156
+ # result: { clicked: true, method: "coordinate", verified: false }
157
+ # warnings: ["AXPress produced no observable state change...coordinate fallback..."]
158
+ # ⚠ MUST re-observe — coordinate clicks can miss or hit the wrong spot:
159
+ screenshot({}) # see where you are now
160
+ # or
161
+ get_window_state({}) # check AX state changed as expected
162
+ # or
163
+ wait_for_element({ text: "Success", until: "appear", timeout: 3000 })
127
164
  ```
128
165
 
166
+ **Rule: `verified:false` always triggers a follow-up observation.** Do not chain
167
+ another action on top of an unverifiable click.
168
+
129
169
  ---
130
170
 
131
- ## 7. Multi-step task with error recovery
171
+ ## 7. Multi-step task with full loop
172
+
173
+ A realistic CLI sequence with observe → decide → act → verify at each step:
132
174
 
133
175
  ```
134
- doctor() # verify permissions first
135
- list_apps()
136
- focus_app("Notes")
137
- find_element({ text: "New Note" }) → id
138
- click_element({ elementId: id, captureAfter: true })
139
-
140
- # if click_element throws ELEMENT_NOT_FOUND:
141
- find_element({ text: "New Note" }) → id2 # refetch, id may have changed
176
+ # setup
177
+ doctor({}) # permissions + helpers green?
178
+ list_apps({})
179
+ focus_app({ app: "Notes" })
180
+
181
+ # step 1: find and click "New Note"
182
+ find_element({ text: "New Note" }) # id "Notes/w0/3"
183
+ click_element({ elementId: "Notes/w0/3", captureAfter: true })
184
+ # read result.verified; if false → screenshot to confirm a new note opened
185
+
186
+ # step 2: if ELEMENT_NOT_FOUND (UI shifted), refetch
187
+ find_element({ text: "New Note" }) # → id2 (may differ)
142
188
  click_element({ elementId: id2 })
143
189
 
144
- type_in_element({ elementId: bodyId, text: "Hello" })
145
- screenshot({}) # confirm content
190
+ # step 3: type into the new note body
191
+ find_element({ role: "AXTextArea" }) # bodyId
192
+ type_in_element({ elementId: bodyId, text: "Hello", captureAfter: true })
193
+
194
+ # step 4: confirm
195
+ screenshot({}) # or describe_screen({}) if vision-degraded
146
196
  ```
197
+
198
+ ---
199
+
200
+ ## 8. When you're stuck — diagnostic order
201
+
202
+ If tools keep failing and you don't know why:
203
+
204
+ 1. `doctor({})` — permissions/helpers still green? (Screen may have re-locked.)
205
+ 2. `list_apps({})` + `list_windows({})` — is the target app/window still there?
206
+ 3. `screenshot({})` or `describe_screen({})` — what's *actually* on screen right
207
+ now? (You may be looking at a different app than you think.)
208
+ 4. Read the `hint` in the last error response — it names the recovery step.
209
+ 5. Check [troubleshooting.md](troubleshooting.md) for the error code.