ucu-mcp 0.5.0 → 0.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +48 -0
- package/README.md +7 -5
- package/dist/src/mcp/tools/element-tools.js +14 -4
- package/dist/src/platform/base.d.ts +13 -2
- package/dist/src/platform/linux.d.ts +2 -2
- package/dist/src/platform/macos/element.d.ts +3 -2
- package/dist/src/platform/macos/element.js +139 -30
- package/dist/src/platform/macos/focus.d.ts +10 -0
- package/dist/src/platform/macos/focus.js +10 -0
- package/dist/src/platform/windows.d.ts +2 -2
- package/package.json +1 -1
- package/skills/ucu-mcp/SKILL.md +113 -75
- package/skills/ucu-mcp/references/troubleshooting.md +24 -0
- package/skills/ucu-mcp/references/workflows.md +143 -80
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,54 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.5.2] - 2026-06-17
|
|
9
|
+
|
|
10
|
+
Agent Skill 重写为 CLI agent 优先(Claude Code / Codex / OpenCode 都是 stdio CLI 环境)。
|
|
11
|
+
|
|
12
|
+
### Changed
|
|
13
|
+
|
|
14
|
+
- **SKILL.md 重写**:Core Workflow 改为"决策循环"状态机视角(observe → decide → act → verify),强调 CLI agent 每次工具调用无状态、必须每次重新观察。新增"Reading click results"表(v0.5.1 的 method/verified 信号解读:axpress+true 放心走 / axpress+false 复核 / coordinate+false 必须重新观察)。工具选择改为 AX-first / vision-fallback / tray 三分法。Operating Rules 第一条改为"Re-observe before every action"。
|
|
15
|
+
- **workflows.md 重写**:所有 playbook 改为 CLI 可执行序列(每步读响应再决定下一步)。cc-switch playbook(#2)补充真实 gotcha:click_menu_bar_extra 打开的是原生应用菜单(About/Hide/Quit),"使用统计"在 WebView 设置窗口内;托盘菜单在独立调用间会自动关闭,必须用 captureAfter 单次捕获。新增 workflow #6(verify click 结果)和 #8(卡住时的诊断顺序)。
|
|
16
|
+
- **troubleshooting.md**:新增"Click result signals (v0.5.1+)"节,解释 method/verified 不是错误而是置信度信号。
|
|
17
|
+
- **README** Agent Skill 节:明确"为 CLI agent 编写"。
|
|
18
|
+
|
|
19
|
+
### 真机验证(按重写后的 skill workflow 重跑)
|
|
20
|
+
|
|
21
|
+
- ✅ doctor ready / focus_app(cc-switch) 建 tray target / click_menu_bar_extra 返回 method:axpress verified:true。
|
|
22
|
+
- ✅ 验证了 skill 的核心价值:workflow #2 的 gotcha 指引正确诊断出 cc-switch 托盘菜单只有 About/Hide/Quit(原生应用菜单),"使用统计"需开 WebView 设置窗口。
|
|
23
|
+
- ⚠️ 发现真实 CLI 痛点:cc-switch 托盘菜单在独立 ocr 调用间自动关闭(失焦即关),必须用 captureAfter 在单次 click_menu_bar_extra 调用里捕获——这正是 skill 新 workflow #6 强调的"单次调用捕获"模式。
|
|
24
|
+
|
|
25
|
+
## [0.5.1] - 2026-06-16
|
|
26
|
+
|
|
27
|
+
方向4b AXPress verify-then-fallback + 方向5 FocusStealSuppression @deprecated 文档化。
|
|
28
|
+
|
|
29
|
+
### Added
|
|
30
|
+
|
|
31
|
+
- **方向4b AXPress verify-then-fallback(clickElement + clickMenuBarExtra)**:解决 Tauri/Electron 控件静默吞 AXPress(不抛异常也不执行)。双层策略:(1) 启发式跳过——对已知静默吞的应用(`AX_SILENT_APP_HINTS = ["tauri"]`,按 appName includes 匹配)直接坐标点击,跳过 AXPress;(2) verify 状态签名——对走 AXPress 的应用,采样前后 `value|focused|selected` 签名,spin 80ms 等异步生效,无变化则降级坐标。`ClickResult { method: "axpress"|"coordinate"; verified: boolean }` 透到 ActionReceipt.result,verified:false 时附 warning 引导模型用 screenshot/get_window_state 复核。无可观测状态的元素(纯按钮)保守判 verified:false 但不降级坐标(避免误伤)。共享 JXA 原语 `stateSignature`/`coordinateClick`/`spinMs`。
|
|
32
|
+
- 返回类型波及链:`clickElement`/`clickMenuBarExtra` `Promise<void>` → `Promise<ClickResult>`(base.ts 接口 + windows/linux stub + element.ts consumer + tools-layer mock)。
|
|
33
|
+
|
|
34
|
+
### Changed
|
|
35
|
+
|
|
36
|
+
- **方向5 FocusStealSuppression @deprecated 文档化**:`focus.ts` 的 `saveFocus`/`restoreFocus` 加 `@deprecated` JSDoc,说明 CGEvent 在 HID 层工作不需前台 focus、焦点管理被有意禁用、`tools-layer.test.ts` 的 `not.toHaveBeenCalled()` 锁定禁用态。不删不接线(文档化设计决策,非遗漏)。`base.ts` 接口注释同步。
|
|
37
|
+
|
|
38
|
+
### Fixed
|
|
39
|
+
|
|
40
|
+
- 无行为修复(4b/5 是 v0.5.0 的质量强化)。
|
|
41
|
+
|
|
42
|
+
### 真机验证(2026-06-16)
|
|
43
|
+
|
|
44
|
+
- ✅ doctor 全绿(3 native helpers + 权限 + 26 工具)。
|
|
45
|
+
- ✅ OCR(方向1)返回非空 elements,无 NSNull 崩溃。
|
|
46
|
+
- ✅ press_key 字母键 Cmd+M(方向2)成功合成,不报 Unknown key。
|
|
47
|
+
- ✅ describe_screen(方向3)结构正确,AX 失败时聚合到 errors[] 不抛出。
|
|
48
|
+
- ✅ focus_app(cc-switch)(方向4a)建立 tray target(windowId:"tray"),不抛 WINDOW_NOT_FOUND。
|
|
49
|
+
- ✅ click_menu_bar_extra(方向4b)返回 `method:"axpress", verified:true`——托盘菜单真实打开(OCR 确认 "About/Hide/Quit CC Switch" 菜单项可见)。
|
|
50
|
+
- ⚠️ 发现:cc-switch 托盘菜单是其**原生应用菜单**(About/Hide/Quit),"使用统计"在 WebView 设置面板内,不在托盘菜单。完整 "设置→使用统计" 流程需先打开设置窗口(非托盘菜单可达),留 follow-up。
|
|
51
|
+
|
|
52
|
+
### Tests
|
|
53
|
+
|
|
54
|
+
- 303→310(+7:clickElement ClickResult 解析、JXA stateSignature/spinMs/coordinateClick/sigBefore/sigAfter/preferCoord source 断言、Tauri 启发式 preferCoord=true、tools-layer method/verified 透出 + warnings)。
|
|
55
|
+
|
|
8
56
|
## [0.5.0] - 2026-06-15
|
|
9
57
|
|
|
10
58
|
cc-switch 操控诊断与视觉通道修复:打通 OCR + 字母快捷键 + 托盘 + 视觉降级 fallback + 内置 agent skill。
|
package/README.md
CHANGED
|
@@ -421,11 +421,13 @@ The same readiness report is also available as the MCP `doctor` tool.
|
|
|
421
421
|
|
|
422
422
|
## Agent Skill
|
|
423
423
|
|
|
424
|
-
UCU-MCP ships an installable **agent skill**
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
|
|
428
|
-
|
|
424
|
+
UCU-MCP ships an installable **agent skill** written for **CLI agents** (Claude
|
|
425
|
+
Code, Codex, OpenCode) that connect via stdio MCP and drive the desktop one
|
|
426
|
+
tool call at a time. It gives richer guidance than the embedded MCP
|
|
427
|
+
`instructions:` field: a decision loop (observe → decide → act → verify),
|
|
428
|
+
tool-selection rules (AX-first / vision-fallback / tray), click-result signal
|
|
429
|
+
interpretation (`method`/`verified`), task playbooks, and an error-recovery
|
|
430
|
+
reference.
|
|
429
431
|
|
|
430
432
|
Install it for your agent runtime with the [`skills` CLI](https://www.npmjs.com/package/skills):
|
|
431
433
|
|
|
@@ -36,8 +36,13 @@ export function registerElementTools(registerTool) {
|
|
|
36
36
|
}, async (params) => {
|
|
37
37
|
const effectiveApp = params.app || getActiveTarget()?.appName;
|
|
38
38
|
const safetyCtx = await getSafetyContext();
|
|
39
|
-
await withSafety({ action: "click_element", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickElement(params.elementId, effectiveApp) });
|
|
40
|
-
|
|
39
|
+
const clickResult = await withSafety({ action: "click_element", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickElement(params.elementId, effectiveApp) });
|
|
40
|
+
const warnings = clickResult.verified
|
|
41
|
+
? []
|
|
42
|
+
: [clickResult.method === "coordinate"
|
|
43
|
+
? "AXPress produced no observable state change (or the app is known to silently swallow it); coordinate fallback was used. Re-observe with screenshot/get_window_state to confirm."
|
|
44
|
+
: "AXPress completed but the element exposed no observable state to verify against. Re-observe with screenshot/get_window_state to confirm."];
|
|
45
|
+
return actionResponse("click_element", { clicked: true, elementId: params.elementId, method: clickResult.method, verified: clickResult.verified }, { elementId: params.elementId, app: effectiveApp }, params.captureAfter, params.captureFormat, params.captureMaxWidth, warnings);
|
|
41
46
|
});
|
|
42
47
|
registerTool("set_value", "Set the value of an accessibility element", {
|
|
43
48
|
elementId: z.string().describe("AX element identifier"), value: z.string().describe("Value to set"), app: z.string().optional().describe("Target app"), ...captureAfterFields,
|
|
@@ -64,7 +69,12 @@ export function registerElementTools(registerTool) {
|
|
|
64
69
|
...captureAfterFields,
|
|
65
70
|
}, async (params) => {
|
|
66
71
|
const safetyCtx = await getSafetyContext();
|
|
67
|
-
await withSafety({ action: "click_menu_bar_extra", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickMenuBarExtra(params.app, { description: params.description, name: params.name, index: params.index }) });
|
|
68
|
-
|
|
72
|
+
const clickResult = await withSafety({ action: "click_menu_bar_extra", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickMenuBarExtra(params.app, { description: params.description, name: params.name, index: params.index }) });
|
|
73
|
+
const warnings = clickResult.verified
|
|
74
|
+
? []
|
|
75
|
+
: [clickResult.method === "coordinate"
|
|
76
|
+
? "AXPress produced no observable state change; coordinate fallback was used. Use find_element or screenshot to confirm the menu opened."
|
|
77
|
+
: "AXPress completed but the status item exposed no observable state to verify against. Use find_element or screenshot to confirm the menu opened."];
|
|
78
|
+
return actionResponse("click_menu_bar_extra", { clicked: true, app: params.app, method: clickResult.method, verified: clickResult.verified }, { app: params.app }, params.captureAfter, params.captureFormat, params.captureMaxWidth, warnings);
|
|
69
79
|
});
|
|
70
80
|
}
|
|
@@ -126,6 +126,17 @@ export interface WindowState {
|
|
|
126
126
|
focusedElement?: ElementInfo;
|
|
127
127
|
tree?: ElementInfo;
|
|
128
128
|
}
|
|
129
|
+
/**
|
|
130
|
+
* Outcome of an AX-driven click — surfaces whether AXPress was used and whether
|
|
131
|
+
* its effect was observable. `verified:false` means either the element exposed
|
|
132
|
+
* no observable state (inconclusive) or the click fell back to coordinates
|
|
133
|
+
* (silent AXPress swallow on Tauri/Electron). The tool layer surfaces this so
|
|
134
|
+
* the model can decide whether to re-observe via screenshot/get_window_state.
|
|
135
|
+
*/
|
|
136
|
+
export interface ClickResult {
|
|
137
|
+
method: "axpress" | "coordinate";
|
|
138
|
+
verified: boolean;
|
|
139
|
+
}
|
|
129
140
|
/**
|
|
130
141
|
* Structured text description of the screen — a fallback for environments where
|
|
131
142
|
* image content blocks are downgraded to URLs (so the model cannot see screenshots).
|
|
@@ -169,7 +180,7 @@ export interface Platform {
|
|
|
169
180
|
type(text: string, delay?: number): Promise<void>;
|
|
170
181
|
key(keys: string[]): Promise<void>;
|
|
171
182
|
findElement(options: FindElementOptions): Promise<FindElementResponse>;
|
|
172
|
-
clickElement(elementId: string, app?: string): Promise<
|
|
183
|
+
clickElement(elementId: string, app?: string): Promise<ClickResult>;
|
|
173
184
|
typeInElement(elementId: string, text: string, app?: string, clearFirst?: boolean): Promise<void>;
|
|
174
185
|
setElementValue?(elementId: string, value: string, app?: string): Promise<void>;
|
|
175
186
|
findMenuBarExtra?(app: string): Promise<unknown[]>;
|
|
@@ -177,7 +188,7 @@ export interface Platform {
|
|
|
177
188
|
description?: string;
|
|
178
189
|
name?: string;
|
|
179
190
|
index?: number;
|
|
180
|
-
}): Promise<
|
|
191
|
+
}): Promise<ClickResult>;
|
|
181
192
|
isScreenLocked?(): boolean;
|
|
182
193
|
saveFocus?(): Promise<void>;
|
|
183
194
|
restoreFocus?(): Promise<void>;
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse } from "./base.js";
|
|
1
|
+
import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse, ClickResult } from "./base.js";
|
|
2
2
|
/**
|
|
3
3
|
* Linux platform adapter (AT-SPI2 + xdotool fallback)
|
|
4
4
|
* TODO: Implement with D-Bus AT-SPI2 bindings
|
|
@@ -17,7 +17,7 @@ export declare class LinuxPlatform implements Platform {
|
|
|
17
17
|
key(keys: string[]): Promise<void>;
|
|
18
18
|
ocr(_display?: number, _region?: ScreenRegion): Promise<OcrResult>;
|
|
19
19
|
findElement(_options: FindElementOptions): Promise<FindElementResponse>;
|
|
20
|
-
clickElement(_elementId: string, _app?: string): Promise<
|
|
20
|
+
clickElement(_elementId: string, _app?: string): Promise<ClickResult>;
|
|
21
21
|
typeInElement(_elementId: string, _text: string, _app?: string, _clearFirst?: boolean): Promise<void>;
|
|
22
22
|
readClipboard(): Promise<string>;
|
|
23
23
|
writeClipboard(text: string): Promise<void>;
|
|
@@ -1,5 +1,6 @@
|
|
|
1
1
|
import type { MacOSPlatform } from "./base.js";
|
|
2
|
-
|
|
2
|
+
import type { ClickResult } from "../base.js";
|
|
3
|
+
export declare function clickElement(this: MacOSPlatform, elementId: string, app?: string): Promise<ClickResult>;
|
|
3
4
|
export declare function typeInElement(this: MacOSPlatform, elementId: string, text: string, app?: string, clearFirst?: boolean): Promise<void>;
|
|
4
5
|
export declare function setElementValue(this: MacOSPlatform, elementId: string, value: string, app?: string): Promise<void>;
|
|
5
6
|
export interface MenuBarExtraItem {
|
|
@@ -21,4 +22,4 @@ export interface MenuBarExtraSelector {
|
|
|
21
22
|
}
|
|
22
23
|
export declare function findMenuBarExtra(this: MacOSPlatform, app: string): Promise<MenuBarExtraItem[]>;
|
|
23
24
|
export declare function matchMenuBarExtra(items: MenuBarExtraItem[], selector: MenuBarExtraSelector): MenuBarExtraItem | undefined;
|
|
24
|
-
export declare function clickMenuBarExtra(this: MacOSPlatform, app: string, selector?: MenuBarExtraSelector): Promise<
|
|
25
|
+
export declare function clickMenuBarExtra(this: MacOSPlatform, app: string, selector?: MenuBarExtraSelector): Promise<ClickResult>;
|
|
@@ -2,6 +2,20 @@ import { execFileSync } from "node:child_process";
|
|
|
2
2
|
import { ElementNotFoundError } from "../../util/errors.js";
|
|
3
3
|
import { rethrowElementActionError } from "./helpers.js";
|
|
4
4
|
import { jxaElementActionHelpers } from "../jxa-helpers.js";
|
|
5
|
+
/**
|
|
6
|
+
* App-name hints for which AXPress is known to be silently swallowed
|
|
7
|
+
* (Tauri custom window decorations, some Electron controls). When the target
|
|
8
|
+
* app matches, clickElement/clickMenuBarExtra skip AXPress and go straight to
|
|
9
|
+
* a coordinate click. Conservative starting set — extend as more silent-swallow
|
|
10
|
+
* apps are confirmed.
|
|
11
|
+
*/
|
|
12
|
+
const AX_SILENT_APP_HINTS = ["tauri"];
|
|
13
|
+
function preferCoordinateClick(appName) {
|
|
14
|
+
if (!appName)
|
|
15
|
+
return false;
|
|
16
|
+
const n = appName.toLowerCase();
|
|
17
|
+
return AX_SILENT_APP_HINTS.some((h) => n.includes(h));
|
|
18
|
+
}
|
|
5
19
|
function prepareCache(elementId) {
|
|
6
20
|
this.evictExpiredCacheEntries();
|
|
7
21
|
const cached = this.elementCache.get(elementId);
|
|
@@ -10,22 +24,65 @@ function prepareCache(elementId) {
|
|
|
10
24
|
}
|
|
11
25
|
return this.elementCache.get(elementId) ?? null;
|
|
12
26
|
}
|
|
27
|
+
/**
|
|
28
|
+
* Shared JXA snippet: read an observable state signature from an AX element.
|
|
29
|
+
* Used by clickElement/clickMenuBarExtra to verify whether AXPress actually
|
|
30
|
+
* changed anything (Tauri/Electron silently swallow AXPress without throwing).
|
|
31
|
+
* Returns a string concatenation of value/focused/selected (each defensively
|
|
32
|
+
* read; missing attributes contribute ''). An empty signature means the element
|
|
33
|
+
* exposes no observable state — verification is inconclusive.
|
|
34
|
+
*/
|
|
35
|
+
const JXA_STATE_SIGNATURE = `
|
|
36
|
+
function stateSignature(elem) {
|
|
37
|
+
var parts = [];
|
|
38
|
+
try { var v = elem.value(); parts.push('v=' + (v === undefined || v === null ? '' : String(v))); } catch(e) { parts.push('v='); }
|
|
39
|
+
try { parts.push('f=' + (elem.focused ? (elem.focused() ? 1 : 0) : '')); } catch(e) { parts.push('f='); }
|
|
40
|
+
try { parts.push('s=' + (elem.selected ? (elem.selected() ? 1 : 0) : '')); } catch(e) { parts.push('s='); }
|
|
41
|
+
return parts.join('|');
|
|
42
|
+
}
|
|
43
|
+
// Brief synchronous spin to let an async AXPress propagate state.
|
|
44
|
+
function spinMs(ms) { var t = Date.now(); while (Date.now() - t < ms) {} }
|
|
45
|
+
`;
|
|
46
|
+
/**
|
|
47
|
+
* Shared JXA snippet: perform a coordinate click at the element's bounds center.
|
|
48
|
+
* Returns the {x, y} used, or null if bounds could not be read.
|
|
49
|
+
*/
|
|
50
|
+
const JXA_COORDINATE_CLICK = `
|
|
51
|
+
function coordinateClick(elem) {
|
|
52
|
+
var pos = elem.position();
|
|
53
|
+
var sz = elem.size();
|
|
54
|
+
var cx = pos[0] + sz[0] / 2;
|
|
55
|
+
var cy = pos[1] + sz[1] / 2;
|
|
56
|
+
ObjC.import('CoreGraphics');
|
|
57
|
+
var src = $.CGEventSourceCreate($.kCGEventSourceStateHIDSystemState);
|
|
58
|
+
var pt = $.CGPointMake(cx, cy);
|
|
59
|
+
var down = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseDown, pt, $.kCGMouseButtonLeft);
|
|
60
|
+
$.CGEventPost($.kCGHIDEventTap, down);
|
|
61
|
+
var up = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseUp, pt, $.kCGMouseButtonLeft);
|
|
62
|
+
$.CGEventPost($.kCGHIDEventTap, up);
|
|
63
|
+
return {x: cx, y: cy};
|
|
64
|
+
}
|
|
65
|
+
`;
|
|
13
66
|
export async function clickElement(elementId, app) {
|
|
14
67
|
const elementIdLiteral = JSON.stringify(elementId);
|
|
15
68
|
const effectiveApp = app || this.activeTarget?.appName;
|
|
16
69
|
const appLiteral = JSON.stringify(effectiveApp || "");
|
|
17
70
|
const cachedDescriptor = prepareCache.call(this, elementId);
|
|
18
71
|
const cachedJson = JSON.stringify(cachedDescriptor);
|
|
72
|
+
const preferCoord = preferCoordinateClick(effectiveApp);
|
|
19
73
|
const jxaScript = `
|
|
20
74
|
var se = Application('System Events');
|
|
21
75
|
var _result = null;
|
|
22
76
|
${jxaElementActionHelpers()}
|
|
77
|
+
${JXA_STATE_SIGNATURE}
|
|
78
|
+
${JXA_COORDINATE_CLICK}
|
|
23
79
|
var elemPath = ${elementIdLiteral};
|
|
24
80
|
var appName = ${appLiteral};
|
|
25
81
|
// 容忍大小写/空格/连字符/下划线变体(cc-switch vs CC Switch vs cc_switch)
|
|
26
82
|
var _norm = function(s) { return String(s||'').toLowerCase().split(' ').join('').split('-').join('').split('_').join(''); };
|
|
27
83
|
var appNorm = _norm(appName);
|
|
28
84
|
var cached = ${cachedJson};
|
|
85
|
+
var preferCoord = ${preferCoord ? "true" : "false"};
|
|
29
86
|
|
|
30
87
|
var elem = resolveElementInApp(elemPath, appName) || resolveElementByFullPath(elemPath);
|
|
31
88
|
if (elem && !descriptorMatches(elem)) {
|
|
@@ -37,27 +94,51 @@ export async function clickElement(elementId, app) {
|
|
|
37
94
|
|
|
38
95
|
if (!elem) {
|
|
39
96
|
_result = {success: false, error: "Element not found: " + elemPath};
|
|
97
|
+
} else if (preferCoord) {
|
|
98
|
+
// 启发式命中(Tauri 等已知静默吞 AXPress 的应用):直接坐标点击
|
|
99
|
+
try {
|
|
100
|
+
coordinateClick(elem);
|
|
101
|
+
_result = {success: true, method: "coordinate", verified: false};
|
|
102
|
+
} catch(e) {
|
|
103
|
+
_result = {success: false, error: "Could not click element (coordinate): " + String(e.message || e)};
|
|
104
|
+
}
|
|
40
105
|
} else {
|
|
106
|
+
// AXPress + verify:采样前后状态签名,无变化则降级坐标
|
|
107
|
+
var sigBefore = stateSignature(elem);
|
|
108
|
+
var axpressThrew = false;
|
|
41
109
|
try {
|
|
42
110
|
elem.actions.AXPress.perform();
|
|
43
|
-
_result = {success: true};
|
|
44
111
|
} catch(e) {
|
|
112
|
+
axpressThrew = true;
|
|
113
|
+
}
|
|
114
|
+
if (axpressThrew) {
|
|
115
|
+
// AXPress 抛异常 → 坐标 fallback
|
|
45
116
|
try {
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
var cx = pos[0] + sz[0] / 2;
|
|
49
|
-
var cy = pos[1] + sz[1] / 2;
|
|
50
|
-
ObjC.import('CoreGraphics');
|
|
51
|
-
var src = $.CGEventSourceCreate($.kCGEventSourceStateHIDSystemState);
|
|
52
|
-
var pt = $.CGPointMake(cx, cy);
|
|
53
|
-
var down = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseDown, pt, $.kCGMouseButtonLeft);
|
|
54
|
-
$.CGEventPost($.kCGHIDEventTap, down);
|
|
55
|
-
var up = $.CGEventCreateMouseEvent(src, $.kCGEventLeftMouseUp, pt, $.kCGMouseButtonLeft);
|
|
56
|
-
$.CGEventPost($.kCGHIDEventTap, up);
|
|
57
|
-
_result = {success: true};
|
|
117
|
+
coordinateClick(elem);
|
|
118
|
+
_result = {success: true, method: "coordinate", verified: false};
|
|
58
119
|
} catch(e2) {
|
|
59
120
|
_result = {success: false, error: "Could not click element: " + String(e2.message || e2)};
|
|
60
121
|
}
|
|
122
|
+
} else {
|
|
123
|
+
// AXPress 未抛异常:spin 等待异步生效,再采样签名判断是否真的生效
|
|
124
|
+
spinMs(80);
|
|
125
|
+
var sigAfter = stateSignature(elem);
|
|
126
|
+
if (sigBefore !== '' && sigAfter !== sigBefore) {
|
|
127
|
+
// 状态变化 → AXPress 生效
|
|
128
|
+
_result = {success: true, method: "axpress", verified: true};
|
|
129
|
+
} else if (sigBefore === '' && sigAfter === '') {
|
|
130
|
+
// 无可观测状态(纯按钮无 value/focused/selected):无法 verify,
|
|
131
|
+
// 保守认为 AXPress 可能成功(不降级坐标,避免对正常按钮误伤)
|
|
132
|
+
_result = {success: true, method: "axpress", verified: false};
|
|
133
|
+
} else {
|
|
134
|
+
// 有可观测状态但无变化 → 静默吞,降级坐标
|
|
135
|
+
try {
|
|
136
|
+
coordinateClick(elem);
|
|
137
|
+
_result = {success: true, method: "coordinate", verified: false};
|
|
138
|
+
} catch(e3) {
|
|
139
|
+
_result = {success: false, error: "Could not click element (coordinate fallback): " + String(e3.message || e3)};
|
|
140
|
+
}
|
|
141
|
+
}
|
|
61
142
|
}
|
|
62
143
|
}
|
|
63
144
|
JSON.stringify(_result);
|
|
@@ -73,9 +154,14 @@ export async function clickElement(elementId, app) {
|
|
|
73
154
|
? new Error(result.error)
|
|
74
155
|
: new ElementNotFoundError(elementId);
|
|
75
156
|
}
|
|
157
|
+
return {
|
|
158
|
+
method: result.method ?? "axpress",
|
|
159
|
+
verified: Boolean(result.verified),
|
|
160
|
+
};
|
|
76
161
|
}
|
|
77
162
|
catch (error) {
|
|
78
163
|
rethrowElementActionError(error, "click_element", elementId);
|
|
164
|
+
return { method: "axpress", verified: false }; // unreachable — rethrow throws
|
|
79
165
|
}
|
|
80
166
|
}
|
|
81
167
|
export async function typeInElement(elementId, text, app, clearFirst) {
|
|
@@ -372,7 +458,8 @@ export async function clickMenuBarExtra(app, selector = {}) {
|
|
|
372
458
|
const host = target.host;
|
|
373
459
|
const tgtNameLiteral = JSON.stringify(target.name || "");
|
|
374
460
|
const tgtDescLiteral = JSON.stringify(target.description || "");
|
|
375
|
-
|
|
461
|
+
const preferCoord = preferCoordinateClick(app);
|
|
462
|
+
// AXPress + verify-then-fallback(与 clickElement 同模式,应对 Tauri 等静默吞)。
|
|
376
463
|
// host==="systemuiserver" 时在 SystemUIServer 进程上重定位(托盘 status item 由它托管),
|
|
377
464
|
// SystemUIServer.menuBarItems 顺序不稳定,用保存的 name/description 二次匹配定位具体 item。
|
|
378
465
|
// host==="self" 时按 app 进程的 menuBars()[mb].menuBarItems()[idx] 重定位(稳定)。
|
|
@@ -411,11 +498,14 @@ export async function clickMenuBarExtra(app, selector = {}) {
|
|
|
411
498
|
var appName = ${appLiteral};
|
|
412
499
|
var tgtName = ${tgtNameLiteral};
|
|
413
500
|
var tgtDesc = ${tgtDescLiteral};
|
|
501
|
+
${JXA_STATE_SIGNATURE}
|
|
502
|
+
${JXA_COORDINATE_CLICK}
|
|
414
503
|
// 容忍大小写/空格/连字符/下划线变体(cc-switch vs CC Switch vs cc_switch)
|
|
415
504
|
var _norm = function(s) { return String(s||'').toLowerCase().split(' ').join('').split('-').join('').split('_').join(''); };
|
|
416
505
|
var appNorm = _norm(appName);
|
|
417
506
|
var tgtNameNorm = _norm(tgtName);
|
|
418
507
|
var tgtDescNorm = _norm(tgtDesc);
|
|
508
|
+
var preferCoord = ${preferCoord ? "true" : "false"};
|
|
419
509
|
var _matchApp = function(itemNorm) {
|
|
420
510
|
if (!itemNorm) return false;
|
|
421
511
|
return itemNorm === appNorm || itemNorm.indexOf(appNorm) !== -1 || appNorm.indexOf(itemNorm) !== -1;
|
|
@@ -430,21 +520,36 @@ export async function clickMenuBarExtra(app, selector = {}) {
|
|
|
430
520
|
}
|
|
431
521
|
${resolveItemBlock}
|
|
432
522
|
if (!_result && item) {
|
|
433
|
-
|
|
434
|
-
|
|
435
|
-
|
|
436
|
-
|
|
437
|
-
|
|
438
|
-
|
|
439
|
-
var
|
|
440
|
-
var
|
|
441
|
-
|
|
442
|
-
|
|
443
|
-
|
|
444
|
-
|
|
445
|
-
|
|
446
|
-
|
|
447
|
-
|
|
523
|
+
if (preferCoord) {
|
|
524
|
+
// 启发式命中:直接坐标点击
|
|
525
|
+
coordinateClick(item);
|
|
526
|
+
_result = {success: true, method: "coordinate", verified: false};
|
|
527
|
+
} else {
|
|
528
|
+
// AXPress + verify
|
|
529
|
+
var sigBefore = stateSignature(item);
|
|
530
|
+
var axpressThrew = false;
|
|
531
|
+
try {
|
|
532
|
+
item.actions.AXPress.perform();
|
|
533
|
+
} catch(e) {
|
|
534
|
+
axpressThrew = true;
|
|
535
|
+
}
|
|
536
|
+
if (axpressThrew) {
|
|
537
|
+
coordinateClick(item);
|
|
538
|
+
_result = {success: true, method: "coordinate", verified: false};
|
|
539
|
+
} else {
|
|
540
|
+
spinMs(80);
|
|
541
|
+
var sigAfter = stateSignature(item);
|
|
542
|
+
if (sigBefore !== '' && sigAfter !== sigBefore) {
|
|
543
|
+
_result = {success: true, method: "axpress", verified: true};
|
|
544
|
+
} else if (sigBefore === '' && sigAfter === '') {
|
|
545
|
+
// 托盘 status item 通常无可观测状态(无 value/selected);AXPress 开菜单的效果
|
|
546
|
+
// 难以从 item 本身验证。保守认为成功(菜单是否打开由后续 find_element 判断)。
|
|
547
|
+
_result = {success: true, method: "axpress", verified: false};
|
|
548
|
+
} else {
|
|
549
|
+
coordinateClick(item);
|
|
550
|
+
_result = {success: true, method: "coordinate", verified: false};
|
|
551
|
+
}
|
|
552
|
+
}
|
|
448
553
|
}
|
|
449
554
|
}
|
|
450
555
|
} catch(e) {
|
|
@@ -458,10 +563,14 @@ export async function clickMenuBarExtra(app, selector = {}) {
|
|
|
458
563
|
}
|
|
459
564
|
catch (error) {
|
|
460
565
|
rethrowElementActionError(error, "click_menu_bar_extra", app);
|
|
461
|
-
return; // unreachable
|
|
566
|
+
return { method: "axpress", verified: false }; // unreachable — rethrow throws
|
|
462
567
|
}
|
|
463
568
|
const result = JSON.parse(out);
|
|
464
569
|
if (!result.success) {
|
|
465
570
|
throw new Error(`click_menu_bar_extra failed in ${app}: ${result.error}`);
|
|
466
571
|
}
|
|
572
|
+
return {
|
|
573
|
+
method: result.method ?? "axpress",
|
|
574
|
+
verified: Boolean(result.verified),
|
|
575
|
+
};
|
|
467
576
|
}
|
|
@@ -1,3 +1,13 @@
|
|
|
1
1
|
import type { MacOSPlatform } from "./base.js";
|
|
2
|
+
/**
|
|
3
|
+
* @deprecated Focus management is intentionally disabled. CGEvent input
|
|
4
|
+
* synthesis (click/type/key) works at the HID layer and does not require the
|
|
5
|
+
* target app to be frontmost, so stealing/restoring focus is unnecessary and
|
|
6
|
+
* could disrupt the user. `saveFocus`/`restoreFocus` are retained for API
|
|
7
|
+
* compatibility but are NOT called from the action pipeline — see the
|
|
8
|
+
* `not.toHaveBeenCalled()` assertions in `tools-layer.test.ts` that lock this
|
|
9
|
+
* disabled state. Do not wire these into new code; prefer CGEvent-based input.
|
|
10
|
+
*/
|
|
2
11
|
export declare function saveFocus(this: MacOSPlatform): Promise<void>;
|
|
12
|
+
/** @deprecated See {@link saveFocus} — intentionally disabled, not called from the action pipeline. */
|
|
3
13
|
export declare function restoreFocus(this: MacOSPlatform): Promise<void>;
|
|
@@ -1,4 +1,13 @@
|
|
|
1
1
|
import { execFileSync } from "node:child_process";
|
|
2
|
+
/**
|
|
3
|
+
* @deprecated Focus management is intentionally disabled. CGEvent input
|
|
4
|
+
* synthesis (click/type/key) works at the HID layer and does not require the
|
|
5
|
+
* target app to be frontmost, so stealing/restoring focus is unnecessary and
|
|
6
|
+
* could disrupt the user. `saveFocus`/`restoreFocus` are retained for API
|
|
7
|
+
* compatibility but are NOT called from the action pipeline — see the
|
|
8
|
+
* `not.toHaveBeenCalled()` assertions in `tools-layer.test.ts` that lock this
|
|
9
|
+
* disabled state. Do not wire these into new code; prefer CGEvent-based input.
|
|
10
|
+
*/
|
|
2
11
|
export async function saveFocus() {
|
|
3
12
|
try {
|
|
4
13
|
const apps = await this.listApps();
|
|
@@ -16,6 +25,7 @@ export async function saveFocus() {
|
|
|
16
25
|
this.savedFocus = undefined;
|
|
17
26
|
}
|
|
18
27
|
}
|
|
28
|
+
/** @deprecated See {@link saveFocus} — intentionally disabled, not called from the action pipeline. */
|
|
19
29
|
export async function restoreFocus() {
|
|
20
30
|
if (!this.savedFocus)
|
|
21
31
|
return;
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse } from "./base.js";
|
|
1
|
+
import type { Platform, ScreenRegion, ScreenSize, CursorPosition, WindowInfo, WindowState, OcrResult, FindElementOptions, FindElementResponse, ClickResult } from "./base.js";
|
|
2
2
|
export declare class WindowsPlatform implements Platform {
|
|
3
3
|
screenshot(_display?: number, _region?: ScreenRegion): Promise<Buffer>;
|
|
4
4
|
getScreenSize(_display?: number): ScreenSize;
|
|
@@ -13,7 +13,7 @@ export declare class WindowsPlatform implements Platform {
|
|
|
13
13
|
key(_keys: string[]): Promise<void>;
|
|
14
14
|
ocr(_display?: number, _region?: ScreenRegion): Promise<OcrResult>;
|
|
15
15
|
findElement(_options: FindElementOptions): Promise<FindElementResponse>;
|
|
16
|
-
clickElement(_elementId: string, _app?: string): Promise<
|
|
16
|
+
clickElement(_elementId: string, _app?: string): Promise<ClickResult>;
|
|
17
17
|
typeInElement(_elementId: string, _text: string, _app?: string, _clearFirst?: boolean): Promise<void>;
|
|
18
18
|
readClipboard(): Promise<string>;
|
|
19
19
|
writeClipboard(text: string): Promise<void>;
|
package/package.json
CHANGED
package/skills/ucu-mcp/SKILL.md
CHANGED
|
@@ -5,96 +5,134 @@ description: >-
|
|
|
5
5
|
click, type, OCR, AX element tools, menu-bar tray support). Use when an
|
|
6
6
|
agent needs to automate macOS desktop apps over MCP — establishing target
|
|
7
7
|
context, reading screen state, interacting with UI elements, operating
|
|
8
|
-
menu-bar/tray apps, or recovering from AX/permission errors.
|
|
9
|
-
Code
|
|
8
|
+
menu-bar/tray apps, or recovering from AX/permission errors. Designed for
|
|
9
|
+
CLI agents (Claude Code, Codex, OpenCode) that connect via stdio MCP and
|
|
10
|
+
drive the desktop one tool call at a time.
|
|
10
11
|
---
|
|
11
12
|
|
|
12
13
|
# UCU-MCP
|
|
13
14
|
|
|
14
|
-
UCU-MCP is a
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
- npm package: `ucu-mcp`
|
|
20
|
-
- Run: `npx -y ucu-mcp` (stdio MCP server) or install globally via `npm i -g ucu-mcp`
|
|
21
|
-
|
|
22
|
-
## Core Workflow
|
|
23
|
-
|
|
24
|
-
1. **Check readiness** → `doctor` verifies Accessibility + Screen Recording
|
|
25
|
-
permissions and native helpers. If anything is missing, follow its guidance
|
|
26
|
-
(see [troubleshooting](references/troubleshooting.md)).
|
|
27
|
-
2. **Establish target context** → `list_apps` then `focus_app(name)` sets the
|
|
28
|
-
active window target. Subsequent AX tools operate against that target.
|
|
29
|
-
3. **Prefer AX over coordinates** → `find_element(text/role/value)` →
|
|
30
|
-
`click_element` / `type_in_element` / `set_value`. AX is precise and survives
|
|
31
|
-
layout shifts; coordinates are a last resort.
|
|
32
|
-
4. **When AX is opaque (Electron/Tauri/WebView)** → `screenshot` + `ocr` to
|
|
33
|
-
locate text by bounding box, then `click(x, y)` at the returned coordinates.
|
|
34
|
-
5. **When image content is not visible to you** (relayed/downgraded to a URL) →
|
|
35
|
-
`screenshot(describe: true)` or the standalone `describe_screen` tool to get a
|
|
36
|
-
structured text view (OCR blocks + AX tree + foreground window).
|
|
37
|
-
6. **Menu-bar/tray apps** (e.g. cc-switch) → `click_menu_bar_extra(app,
|
|
38
|
-
description/name/index)` opens the tray menu, then `find_element` inside it.
|
|
39
|
-
7. **Verify actions** → pass `captureAfter: true` on action tools, or call
|
|
40
|
-
`screenshot` / `get_window_state` afterwards.
|
|
41
|
-
8. **Recover from errors** → every error response carries a `hint` with the
|
|
42
|
-
next step. See the [error code table](references/troubleshooting.md).
|
|
43
|
-
|
|
44
|
-
Full tool inventory with parameters: [tool-reference](references/tool-reference.md).
|
|
45
|
-
Common task playbooks: [workflows](references/workflows.md).
|
|
15
|
+
UCU-MCP is a macOS computer-use MCP server for CLI agents. It exposes 26 tools
|
|
16
|
+
that let you see the screen and drive native apps through Accessibility (AX)
|
|
17
|
+
APIs, CGEvent input synthesis, Vision OCR, and ScreenCaptureKit screenshots.
|
|
18
|
+
Windows/Linux are explicit stubs.
|
|
46
19
|
|
|
47
|
-
|
|
20
|
+
- npm: `ucu-mcp` · run: `npx -y ucu-mcp` (stdio MCP server)
|
|
21
|
+
- You are a **CLI agent**: each tool call is one MCP request over stdio. There
|
|
22
|
+
is no persistent UI session between calls — **always re-observe state before
|
|
23
|
+
acting**, because the user or the app may have changed the screen since your
|
|
24
|
+
last call.
|
|
48
25
|
|
|
49
|
-
|
|
50
|
-
`set_value` whenever the AX tree exposes the target. Fall back to coordinates
|
|
51
|
-
only when AX returns nothing (Electron/WebView) or the control silently
|
|
52
|
-
swallows AX actions.
|
|
53
|
-
- **Observe before acting.** Call `screenshot` / `get_window_state` /
|
|
54
|
-
`describe_screen` before destructive or hard-to-reverse actions so you act on
|
|
55
|
-
current state, not assumptions.
|
|
56
|
-
- **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
|
|
57
|
-
retry — the element cache refetches equivalent AX nodes.
|
|
58
|
-
- **Tray apps need `click_menu_bar_extra`.** `focus_app` alone cannot reach
|
|
59
|
-
pure menu-bar (LSUIElement) apps; their status item is hosted by
|
|
60
|
-
`SystemUIServer` and is not in any app window's AX tree.
|
|
61
|
-
- **Dangerous actions are blocked.** Quit/logout/lock shortcuts (`cmd+q`,
|
|
62
|
-
`cmd+shift+q`, `cmd+l`, …), sensitive-window URLs, and suspicious injected
|
|
63
|
-
text are rejected by the safety guard. Choose a safer action or ask the user.
|
|
64
|
-
- **Sensitive fields are masked in `describe_screen`.** Password fields
|
|
65
|
-
(`AXSecureTextField`, or names matching `/password|secret|token/i`) appear as
|
|
66
|
-
`[REDACTED]` — never try to read or exfiltrate them.
|
|
67
|
-
- **macOS is locked → actions blocked.** The server refuses to synthesize input
|
|
68
|
-
while the screen is locked; wait for unlock or ask the user.
|
|
69
|
-
|
|
70
|
-
## MCP Config
|
|
71
|
-
|
|
72
|
-
Add UCU-MCP to your MCP client. Stdio transport, no arguments needed.
|
|
73
|
-
|
|
74
|
-
**Codex / generic TOML:**
|
|
75
|
-
|
|
76
|
-
```toml
|
|
77
|
-
[mcp_servers.ucu-mcp]
|
|
78
|
-
command = "npx"
|
|
79
|
-
args = ["-y", "ucu-mcp"]
|
|
80
|
-
```
|
|
26
|
+
## The Decision Loop (run this for every action)
|
|
81
27
|
|
|
82
|
-
|
|
28
|
+
Think in cycles of **observe → decide → act → verify**. Do not chain actions
|
|
29
|
+
blindly; the desktop is a moving target.
|
|
83
30
|
|
|
84
|
-
```bash
|
|
85
|
-
claude mcp add ucu-mcp -- npx -y ucu-mcp
|
|
86
31
|
```
|
|
32
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
33
|
+
│ OBSERVE: what's on screen / what's focused right now? │
|
|
34
|
+
│ screenshot{} · describe_screen{} · get_window_state{} │
|
|
35
|
+
├─────────────────────────────────────────────────────────────┤
|
|
36
|
+
│ DECIDE: AX-first, coordinates only as fallback │
|
|
37
|
+
│ AX tree exposes target? → find_element → element tools │
|
|
38
|
+
│ AX opaque (Electron/Tauri)? → ocr → click(x,y) │
|
|
39
|
+
│ Tray-only app? → click_menu_bar_extra │
|
|
40
|
+
├─────────────────────────────────────────────────────────────┤
|
|
41
|
+
│ ACT: click_element / type_in_element / set_value / click │
|
|
42
|
+
│ Pass captureAfter:true to get a screenshot in the reply │
|
|
43
|
+
├─────────────────────────────────────────────────────────────┤
|
|
44
|
+
│ VERIFY: did it work? (see "Reading click results" below) │
|
|
45
|
+
│ result.verified === true → proceed │
|
|
46
|
+
│ result.verified === false → screenshot/get_window_state │
|
|
47
|
+
│ result.method === "coordinate" → re-observe, may be off │
|
|
48
|
+
└─────────────────────────────────────────────────────────────┘
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
## Reading click results (v0.5.1+)
|
|
52
|
+
|
|
53
|
+
`click_element` and `click_menu_bar_extra` return a `result` object with
|
|
54
|
+
`method` and `verified` fields. **Read them every time** — they tell you
|
|
55
|
+
whether your click actually landed:
|
|
56
|
+
|
|
57
|
+
| `method` | `verified` | Meaning | What you do |
|
|
58
|
+
|---|---|---|---|
|
|
59
|
+
| `"axpress"` | `true` | AXPress changed observable state (value/focused/selected) | Proceed — high confidence it worked |
|
|
60
|
+
| `"axpress"` | `false` | AXPress ran but element had no observable state to verify (e.g. plain button) | Verify via `screenshot` or `get_window_state` |
|
|
61
|
+
| `"coordinate"` | `false` | AXPress was silently swallowed (Tauri/Electron) OR threw; fell back to coordinate click | **Always re-observe** — coordinate clicks can miss, or the app may need a second click |
|
|
62
|
+
|
|
63
|
+
A `warnings[]` array in the receipt explains the fallback. Never assume a
|
|
64
|
+
coordinate-fallback click succeeded without checking.
|
|
65
|
+
|
|
66
|
+
## Tool selection — AX vs vision vs tray
|
|
67
|
+
|
|
68
|
+
**AX-first** (precise, survives layout shifts):
|
|
69
|
+
`find_element` → `click_element` / `type_in_element` / `set_value`. Use when
|
|
70
|
+
the app exposes an AX tree (native macOS apps, most non-Electron apps).
|
|
71
|
+
|
|
72
|
+
**Vision fallback** (when AX is opaque — Electron/Tauri/WebView return an
|
|
73
|
+
empty `AXGroup` or `find_element` returns 0 with an "app is likely Electron"
|
|
74
|
+
hint):
|
|
75
|
+
`screenshot` → `ocr` → compute click point from the OCR block's bounding box
|
|
76
|
+
→ `click(x, y)` at `block.x + block.width/2, block.y + block.height/2`.
|
|
87
77
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
78
|
+
**Text-only fallback** (when you cannot see image content blocks — relay
|
|
79
|
+
downgrades them to URLs):
|
|
80
|
+
`describe_screen` or `screenshot({describe: true})` → structured text with OCR
|
|
81
|
+
blocks + AX tree. Password fields are masked to `[REDACTED]`.
|
|
82
|
+
|
|
83
|
+
**Tray apps** (menu-bar-only / LSUIElement apps like cc-switch — no window, no
|
|
84
|
+
AX tree entry):
|
|
85
|
+
`focus_app` (falls back to a tray target) → `click_menu_bar_extra` opens the
|
|
86
|
+
menu → `find_element` inside the menu, or `screenshot`+`ocr` if the menu is
|
|
87
|
+
also opaque.
|
|
88
|
+
|
|
89
|
+
## First-run setup
|
|
90
|
+
|
|
91
|
+
1. **Connect the server** to your CLI agent:
|
|
92
|
+
|
|
93
|
+
Codex / generic TOML (`.codex/config.toml` or equivalent):
|
|
94
|
+
```toml
|
|
95
|
+
[mcp_servers.ucu-mcp]
|
|
96
|
+
command = "npx"
|
|
97
|
+
args = ["-y", "ucu-mcp"]
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
Claude Code CLI:
|
|
101
|
+
```bash
|
|
102
|
+
claude mcp add ucu-mcp -- npx -y ucu-mcp
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
2. **Grant macOS permissions** — Accessibility **and** Screen Recording must be
|
|
106
|
+
enabled for your terminal/client in System Settings → Privacy & Security.
|
|
107
|
+
**Restart the client after granting** (changes don't apply to running
|
|
108
|
+
processes).
|
|
109
|
+
|
|
110
|
+
3. **Verify** — call `doctor`. It reports per-permission status, native helper
|
|
111
|
+
health, and which process to authorize. Green = ready.
|
|
112
|
+
|
|
113
|
+
## Operating Rules
|
|
114
|
+
|
|
115
|
+
- **Re-observe before every action.** The screen changes between your calls.
|
|
116
|
+
A `focus_app` from 5 calls ago may be stale; a window may have closed.
|
|
117
|
+
- **AX-first, coordinates only as fallback.** AX clicks are precise and
|
|
118
|
+
verifiable; coordinate clicks can drift and are unverifiable.
|
|
119
|
+
- **`verified:false` means re-observe.** Never trust an unverifiable click
|
|
120
|
+
without a follow-up `screenshot` or `get_window_state`.
|
|
121
|
+
- **TARGET_STALE is recoverable.** Re-run `focus_app` for the target app, then
|
|
122
|
+
retry. `type_in_element` auto-refetches equivalent AX nodes.
|
|
123
|
+
- **Dangerous actions are blocked.** `cmd+q`, `cmd+shift+q`, `cmd+l`, `alt+f4`,
|
|
124
|
+
sensitive-window URLs, and suspicious injected text are rejected. Choose a
|
|
125
|
+
safer action or ask the user.
|
|
126
|
+
- **macOS locked → all input blocked.** Wait for unlock; there is no bypass.
|
|
127
|
+
- **Don't exfiltrate passwords.** `describe_screen` masks them to
|
|
128
|
+
`[REDACTED]`; respect that.
|
|
91
129
|
|
|
92
130
|
## References
|
|
93
131
|
|
|
94
132
|
- [tool-reference.md](references/tool-reference.md) — all 26 tools, parameters,
|
|
95
133
|
return shapes, and when to use each.
|
|
96
|
-
- [workflows.md](references/workflows.md) — playbooks
|
|
134
|
+
- [workflows.md](references/workflows.md) — CLI-executable playbooks (form
|
|
97
135
|
filling, tray apps, opaque Electron UIs, vision-degraded environments, stale
|
|
98
|
-
targets.
|
|
136
|
+
targets, click-result verification).
|
|
99
137
|
- [troubleshooting.md](references/troubleshooting.md) — error code table with
|
|
100
138
|
recovery steps, permission issues, AX-opacity workarounds, OCR failures.
|
|
@@ -30,6 +30,30 @@ to recovery steps (mirrors the runtime `recoveryHint`).
|
|
|
30
30
|
|
|
31
31
|
---
|
|
32
32
|
|
|
33
|
+
## Click result signals (v0.5.1+)
|
|
34
|
+
|
|
35
|
+
`click_element` and `click_menu_bar_extra` return `result.method` and
|
|
36
|
+
`result.verified`. These are not errors — they tell you how confident to be:
|
|
37
|
+
|
|
38
|
+
- **`method:"axpress", verified:true`** — AXPress changed observable state.
|
|
39
|
+
Proceed normally.
|
|
40
|
+
- **`method:"axpress", verified:false`** — AXPress ran but the element exposed
|
|
41
|
+
no observable state (plain button, no value/focused/selected). It *probably*
|
|
42
|
+
worked, but re-observe with `screenshot`/`get_window_state` to be sure. A
|
|
43
|
+
`warnings[]` entry explains this.
|
|
44
|
+
- **`method:"coordinate", verified:false`** — AXPress was silently swallowed
|
|
45
|
+
(Tauri/Electron custom controls) or threw; ucu-mcp fell back to a coordinate
|
|
46
|
+
click at the element's bounds center. **Coordinate clicks can miss** — always
|
|
47
|
+
re-observe. A `warnings[]` entry says "coordinate fallback was used".
|
|
48
|
+
|
|
49
|
+
This is not a bug — it's the server telling you it couldn't fully confirm the
|
|
50
|
+
click. The verify-then-fallback logic exists *because* Tauri/Electron controls
|
|
51
|
+
silently swallow AXPress without throwing.
|
|
52
|
+
|
|
53
|
+
---
|
|
54
|
+
|
|
55
|
+
---
|
|
56
|
+
|
|
33
57
|
## Permission Issues
|
|
34
58
|
|
|
35
59
|
macOS requires two permissions for full functionality:
|
|
@@ -1,146 +1,209 @@
|
|
|
1
1
|
# Workflows
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
CLI-executable playbooks. Each is a sequence of tool calls you make one at a
|
|
4
|
+
time, reading each response before the next. Coordinates are screen-absolute
|
|
5
|
+
unless noted.
|
|
5
6
|
|
|
6
7
|
---
|
|
7
8
|
|
|
8
|
-
## 1. Fill a form field
|
|
9
|
+
## 1. Fill a form field (native app, AX-visible)
|
|
9
10
|
|
|
10
|
-
**Primary (AX):**
|
|
11
11
|
```
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
12
|
+
# 1. establish target
|
|
13
|
+
list_apps({})
|
|
14
|
+
focus_app({ app: "Safari" })
|
|
15
|
+
|
|
16
|
+
# 2. locate the field via AX
|
|
17
|
+
find_element({ text: "Email", role: "AXTextField", app: "Safari" })
|
|
18
|
+
# → response.result.results[0].id e.g. "Safari/w0/42"
|
|
19
|
+
|
|
20
|
+
# 3. type into it
|
|
15
21
|
type_in_element({ elementId: "Safari/w0/42", text: "user@example.com" })
|
|
16
|
-
```
|
|
17
22
|
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
set_value({ elementId: "...", value: "option" })
|
|
23
|
+
# 4. verify (captureAfter returns a screenshot in the same reply)
|
|
24
|
+
type_in_element({ elementId: "Safari/w0/42", text: "...", captureAfter: true })
|
|
21
25
|
```
|
|
22
26
|
|
|
23
|
-
**
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
type_text({ text: "user@example.com" })
|
|
29
|
-
```
|
|
27
|
+
**If `find_element` returns 0** with an "app is likely Electron" hint → switch
|
|
28
|
+
to workflow #3 (vision fallback).
|
|
29
|
+
|
|
30
|
+
**If `type_in_element` throws TARGET_STALE** → `focus_app("Safari")` then retry
|
|
31
|
+
(it auto-refetches equivalent AX nodes).
|
|
30
32
|
|
|
31
33
|
---
|
|
32
34
|
|
|
33
35
|
## 2. Operate a menu-bar / tray app (e.g. cc-switch)
|
|
34
36
|
|
|
35
|
-
Tray apps
|
|
36
|
-
|
|
37
|
+
Tray-only apps (LSUIElement) have no window; their status item is hosted by
|
|
38
|
+
`SystemUIServer`. `focus_app` alone returns `WINDOW_NOT_FOUND` unless ucu-mcp
|
|
39
|
+
finds a tray status item and falls back to a tray target.
|
|
37
40
|
|
|
38
41
|
```
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
# menu is now open — find items inside it:
|
|
42
|
-
find_element({ text: "使用统计", app: "cc-switch" })
|
|
43
|
-
→ elementId
|
|
44
|
-
click_element({ elementId })
|
|
45
|
-
```
|
|
42
|
+
# 1. establish tray target (returns windowId:"tray" on success)
|
|
43
|
+
focus_app({ app: "cc-switch" })
|
|
46
44
|
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
45
|
+
# 2. open the tray menu — check result.verified!
|
|
46
|
+
click_menu_bar_extra({ app: "cc-switch", name: "switch" })
|
|
47
|
+
# → result: { clicked: true, method: "axpress", verified: true|false }
|
|
48
|
+
|
|
49
|
+
# 3a. IF the menu exposes AX items, find and click them:
|
|
50
|
+
find_element({ text: "Settings", app: "cc-switch" })
|
|
51
|
+
click_element({ elementId: "..." })
|
|
52
|
+
|
|
53
|
+
# 3b. IF find_element returns 0 (menu is also opaque), use vision:
|
|
50
54
|
screenshot({})
|
|
51
|
-
ocr({})
|
|
52
|
-
|
|
55
|
+
ocr({})
|
|
56
|
+
# → find the menu item text in ocr.blocks, compute center, click:
|
|
57
|
+
click({ x: block.x + block.width/2, y: block.y + block.height/2 })
|
|
53
58
|
```
|
|
54
59
|
|
|
60
|
+
**Known gotcha (cc-switch and similar Tauri tray apps):** `click_menu_bar_extra`
|
|
61
|
+
may open the app's **native application menu** (About / Hide / Quit) rather than
|
|
62
|
+
a custom tray popup. If OCR shows only About/Hide/Quit, the app's real settings
|
|
63
|
+
live in a **WebView window** — you need to open that window first (via a menu
|
|
64
|
+
item, or `focus_app` once a window exists), then drive it with workflow #3
|
|
65
|
+
(Electron-opaque). Don't keep clicking the tray expecting a custom menu.
|
|
66
|
+
|
|
67
|
+
**If `click_menu_bar_extra` returns `method:"coordinate"`** (AXPress was
|
|
68
|
+
swallowed) → re-observe with `screenshot` to confirm the menu actually opened
|
|
69
|
+
before searching its contents.
|
|
70
|
+
|
|
55
71
|
---
|
|
56
72
|
|
|
57
|
-
## 3. Electron / WebView opaque UI
|
|
73
|
+
## 3. Electron / Tauri / WebView opaque UI
|
|
58
74
|
|
|
59
|
-
Electron/Tauri apps
|
|
60
|
-
|
|
75
|
+
Electron/Tauri apps render UI in a composited layer AX cannot introspect.
|
|
76
|
+
`find_element` returns 0; `get_window_state` shows a near-empty `AXGroup`;
|
|
77
|
+
`list_windows` emits an Electron hint.
|
|
61
78
|
|
|
62
79
|
```
|
|
80
|
+
# 1. confirm opacity (optional — the hint in the error tells you)
|
|
63
81
|
find_element({ text: "Submit" })
|
|
64
|
-
|
|
82
|
+
# → 0 results, hint: "...likely Electron... screenshot → ocr → click(x,y)"
|
|
65
83
|
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
84
|
+
# 2. see the screen via OCR (text + bounding boxes)
|
|
85
|
+
screenshot({}) # for yourself, if you can see images
|
|
86
|
+
ocr({}) # → blocks[].text with {x, y, width, height}
|
|
87
|
+
|
|
88
|
+
# 3. locate target text, compute click center, click by coordinate
|
|
89
|
+
# (OCR coordinates are screen-absolute)
|
|
69
90
|
click({ x: block.x + block.width/2, y: block.y + block.height/2 })
|
|
91
|
+
|
|
92
|
+
# 4. verify the coordinate click landed (coordinate clicks are unverifiable)
|
|
93
|
+
screenshot({}) # or describe_screen({}) if you can't see images
|
|
70
94
|
```
|
|
71
95
|
|
|
72
|
-
For
|
|
73
|
-
|
|
96
|
+
For multi-step interaction with a known-opaque app, call `describe_screen` once
|
|
97
|
+
to plan, then drive by coordinates. Keep re-OCR-ing between steps — WebView
|
|
98
|
+
layouts shift.
|
|
74
99
|
|
|
75
100
|
---
|
|
76
101
|
|
|
77
|
-
## 4. Vision-degraded environment (image content
|
|
102
|
+
## 4. Vision-degraded environment (you can't see image content)
|
|
78
103
|
|
|
79
|
-
When
|
|
80
|
-
|
|
104
|
+
When `screenshot` image blocks are downgraded to URLs you cannot fetch, switch
|
|
105
|
+
to text-based screen reading:
|
|
81
106
|
|
|
82
107
|
```
|
|
83
108
|
describe_screen({ ocr: true, includeAx: true })
|
|
84
|
-
|
|
109
|
+
# → { capturedAt, screen, foregroundWindow, ocr:{blocks, status}, ax:{elements, status}, errors }
|
|
85
110
|
|
|
86
|
-
#
|
|
87
|
-
|
|
88
|
-
→ [image block, text description block]
|
|
111
|
+
# OCR blocks give you text + screen coordinates — drive by click(x,y)
|
|
112
|
+
# AX elements give you a tree you can find_element/click_element on
|
|
89
113
|
```
|
|
90
114
|
|
|
91
|
-
`describe_screen` never throws — OCR and AX each try/catch independently
|
|
92
|
-
|
|
93
|
-
|
|
115
|
+
`describe_screen` **never throws** — OCR and AX each try/catch independently.
|
|
116
|
+
Check `errors[]` to see what was skipped/failed, and `ocr.status` / `ax.status`
|
|
117
|
+
(`"ok"` / `"skipped"` / `"failed"`). If OCR failed, fall back to AX-only
|
|
118
|
+
(`includeAx: true, ocr: false`); if AX failed, fall back to OCR-only.
|
|
119
|
+
|
|
120
|
+
`screenshot({ describe: true })` returns both an image (for clients that can
|
|
121
|
+
see it) **and** a text description block — use this when you're unsure whether
|
|
122
|
+
your client renders images.
|
|
94
123
|
|
|
95
124
|
---
|
|
96
125
|
|
|
97
126
|
## 5. Recover from TARGET_STALE
|
|
98
127
|
|
|
99
|
-
The active window target
|
|
100
|
-
|
|
128
|
+
The active window target goes stale when the window closes, the app restarts,
|
|
129
|
+
or the pid changes. AX tools throw `TARGET_STALE` (receipt includes a `hint`).
|
|
101
130
|
|
|
102
131
|
```
|
|
103
|
-
#
|
|
104
|
-
focus_app("Safari")
|
|
105
|
-
|
|
106
|
-
|
|
132
|
+
# 1. re-establish the target
|
|
133
|
+
focus_app({ app: "Safari" })
|
|
134
|
+
|
|
135
|
+
# 2. retry — element cache refetches equivalent AX nodes
|
|
136
|
+
find_element({ text: "Save" })
|
|
137
|
+
click_element({ elementId: "..." })
|
|
107
138
|
```
|
|
108
139
|
|
|
109
|
-
`type_in_element`
|
|
110
|
-
|
|
140
|
+
`type_in_element` auto-refetches an equivalent AX node if the `elementId` is
|
|
141
|
+
stale, so a single retry often succeeds without re-running `focus_app`.
|
|
111
142
|
|
|
112
143
|
---
|
|
113
144
|
|
|
114
|
-
## 6. Verify
|
|
145
|
+
## 6. Verify a click succeeded (v0.5.1+)
|
|
115
146
|
|
|
116
|
-
|
|
117
|
-
|
|
147
|
+
Every `click_element` / `click_menu_bar_extra` response includes `result.method`
|
|
148
|
+
and `result.verified`. Use them to decide whether to trust the click:
|
|
118
149
|
|
|
119
150
|
```
|
|
120
|
-
click_element({ elementId
|
|
121
|
-
#
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
#
|
|
126
|
-
|
|
151
|
+
click_element({ elementId: "btn1" })
|
|
152
|
+
# → result: { clicked: true, method: "axpress", verified: true }
|
|
153
|
+
# ✓ AXPress changed observable state — proceed with confidence
|
|
154
|
+
|
|
155
|
+
# OR
|
|
156
|
+
# → result: { clicked: true, method: "coordinate", verified: false }
|
|
157
|
+
# warnings: ["AXPress produced no observable state change...coordinate fallback..."]
|
|
158
|
+
# ⚠ MUST re-observe — coordinate clicks can miss or hit the wrong spot:
|
|
159
|
+
screenshot({}) # see where you are now
|
|
160
|
+
# or
|
|
161
|
+
get_window_state({}) # check AX state changed as expected
|
|
162
|
+
# or
|
|
163
|
+
wait_for_element({ text: "Success", until: "appear", timeout: 3000 })
|
|
127
164
|
```
|
|
128
165
|
|
|
166
|
+
**Rule: `verified:false` always triggers a follow-up observation.** Do not chain
|
|
167
|
+
another action on top of an unverifiable click.
|
|
168
|
+
|
|
129
169
|
---
|
|
130
170
|
|
|
131
|
-
## 7. Multi-step task with
|
|
171
|
+
## 7. Multi-step task with full loop
|
|
172
|
+
|
|
173
|
+
A realistic CLI sequence with observe → decide → act → verify at each step:
|
|
132
174
|
|
|
133
175
|
```
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
#
|
|
141
|
-
|
|
176
|
+
# setup
|
|
177
|
+
doctor({}) # permissions + helpers green?
|
|
178
|
+
list_apps({})
|
|
179
|
+
focus_app({ app: "Notes" })
|
|
180
|
+
|
|
181
|
+
# step 1: find and click "New Note"
|
|
182
|
+
find_element({ text: "New Note" }) # → id "Notes/w0/3"
|
|
183
|
+
click_element({ elementId: "Notes/w0/3", captureAfter: true })
|
|
184
|
+
# read result.verified; if false → screenshot to confirm a new note opened
|
|
185
|
+
|
|
186
|
+
# step 2: if ELEMENT_NOT_FOUND (UI shifted), refetch
|
|
187
|
+
find_element({ text: "New Note" }) # → id2 (may differ)
|
|
142
188
|
click_element({ elementId: id2 })
|
|
143
189
|
|
|
144
|
-
|
|
145
|
-
|
|
190
|
+
# step 3: type into the new note body
|
|
191
|
+
find_element({ role: "AXTextArea" }) # → bodyId
|
|
192
|
+
type_in_element({ elementId: bodyId, text: "Hello", captureAfter: true })
|
|
193
|
+
|
|
194
|
+
# step 4: confirm
|
|
195
|
+
screenshot({}) # or describe_screen({}) if vision-degraded
|
|
146
196
|
```
|
|
197
|
+
|
|
198
|
+
---
|
|
199
|
+
|
|
200
|
+
## 8. When you're stuck — diagnostic order
|
|
201
|
+
|
|
202
|
+
If tools keep failing and you don't know why:
|
|
203
|
+
|
|
204
|
+
1. `doctor({})` — permissions/helpers still green? (Screen may have re-locked.)
|
|
205
|
+
2. `list_apps({})` + `list_windows({})` — is the target app/window still there?
|
|
206
|
+
3. `screenshot({})` or `describe_screen({})` — what's *actually* on screen right
|
|
207
|
+
now? (You may be looking at a different app than you think.)
|
|
208
|
+
4. Read the `hint` in the last error response — it names the recovery step.
|
|
209
|
+
5. Check [troubleshooting.md](troubleshooting.md) for the error code.
|