ucu-mcp 0.6.3 → 0.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +47 -0
- package/dist/src/mcp/tools/app-tools.js +4 -4
- package/dist/src/mcp/tools/element-tools.js +8 -0
- package/dist/src/mcp/tools/screen-tools.js +12 -5
- package/dist/src/platform/macos/element.js +8 -2
- package/dist/src/platform/macos/helpers.js +8 -4
- package/dist/src/platform/macos/input.js +7 -3
- package/dist/src/platform/macos/window.js +18 -9
- package/dist/src/safety/guard.d.ts +1 -1
- package/dist/src/safety/guard.js +20 -4
- package/dist/src/utils/input.js +30 -13
- package/native/skylight/main.swift +38 -22
- package/native/skylight/skylight-helper +0 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,53 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [0.6.5] - 2026-06-18
|
|
9
|
+
|
|
10
|
+
Code review P2 加固(3 批次):安全面 + 输入正确性 + 日志卫生。
|
|
11
|
+
|
|
12
|
+
### Security Hardening
|
|
13
|
+
|
|
14
|
+
- **OCR 密码脱敏**(screen-tools.ts):describe_screen 的 OCR blocks 原样返回,截屏含密码字段时明文泄露。加 SENSITIVE_NAME_RE 对 OCR blocks 文本脱敏为 `[REDACTED]`,fullText 从脱敏后 blocks 重建。
|
|
15
|
+
- **clipboard_write URL scheme 拦截**(guard.ts):DEFAULT_TEXT_INJECTION_PATTERNS 补 `javascript:`/`data:`/`vbscript:` scheme + `\b(tell application)` AppleScript 注入 + `\n` 分隔命令 + deno/bun 解释器。防止剪贴板投毒(粘贴执行)。
|
|
16
|
+
- **wait_for_element timeout/interval cap**(app-tools.ts):原 schema 无 cap,恶意 timeout=2^31 + interval=1 可 fork-bomb。加 timeout max 60000 + interval min 250 的 zod 约束。
|
|
17
|
+
- **logger 脱敏**(input.ts):typeText 的 dryRun log 原记录 `text.slice(0,50)`(含密码),改为只记 `charCount`。
|
|
18
|
+
|
|
19
|
+
### Input Correctness
|
|
20
|
+
|
|
21
|
+
- **doubleClick/drag 加 Chromium primer**(skylight/main.swift):提取 `primeTarget()` 共享函数(leading mouseMoved + (-1,-1) primer),doClick/doDoubleClick/doDrag 统一调用。之前 doubleClick/drag 在 Chromium/Electron 上静默失败(user-activation gate 未满足)。
|
|
22
|
+
- **mouseUp pressure 改为 0**(skylight/main.swift):真实 mouseUp 的 pressure=0,原代码所有非 move 事件都 pressure=1.0,影响绘图/3D app。改为 Up/Moved 事件 pressure=0。
|
|
23
|
+
- **isFrontmost 用 NSWorkspace.frontmostApplication**(skylight/main.swift):原 `isActive` 对 LSUIElement/agent 进程(如 SystemUIServer)不可靠,改用 `NSWorkspace.shared.frontmostApplication?.processIdentifier == pid` 做明确判定。
|
|
24
|
+
- **非 ASCII typeText fallback 警告**(input.ts):非 ASCII 字符走 osascript keystroke(发到前台而非 target pid),当有 target 时 logger.warn 提醒。
|
|
25
|
+
|
|
26
|
+
### Tests
|
|
27
|
+
|
|
28
|
+
- 328 passed(wait_for_element schema type 从 number→integer,cli-mcp 测试同步)。
|
|
29
|
+
|
|
30
|
+
## [0.6.4] - 2026-06-18
|
|
31
|
+
|
|
32
|
+
4 个独立子 Agent 全面 code review 后的修复(skylight/element/tests/security)。
|
|
33
|
+
|
|
34
|
+
### P0 Fixed(bugs/security)
|
|
35
|
+
|
|
36
|
+
- **focusWithoutRaise 返回值检查反转**(skylight/main.swift:170):`!= 0` 应为 `== 0`(OSStatus 0=success)。原代码导致 focus-without-raise 完全失效——per-pid 点击落到 AppKit 当前认为 active 的窗口,不一定是目标。
|
|
37
|
+
- **runInputChecked cgevent fallback ENOENT**(utils/input.ts):skylight-only 环境下 skylight 运行时错误会直接 throw ENOENT(cgevent-helper 不存在)。加 isCgeventAvailable 门控 + try/catch。
|
|
38
|
+
- **appNameMatches 过度匹配**(helpers.ts):双向 includes 导致 `"code"` 匹配 `"vscode"`。改为单向 process.includes(requested) + 最小 3 字符长度。
|
|
39
|
+
- **needCoordinateClick 静默返回 axpress 成功**(element.ts):JXA 请求坐标点击但 cx/cy 缺失时,原代码 fallthrough 返回 `{method:"axpress"}`(谎报成功)。改为抛 ElementNotFoundError。
|
|
40
|
+
- **normalizeShortcut 别名绕过**(guard.ts):`escape`→`esc`、`delete`→`del`、`return`→`enter` 不规范化,blocklist 可被长格式键名绕过(如 `cmd+option+escape` 不匹配 `command+esc+option`)。加 KEY_CANONICAL 表。
|
|
41
|
+
- **find_element 密码字段不脱敏**(element-tools.ts):AXSecureTextField/AXPasswordField 的 value 原样返回给模型。加 SENSITIVE_RE 脱敏为 `[REDACTED]`。
|
|
42
|
+
- **clipboard_read 重分类为 input**(guard.ts):从 OBSERVE_ACTIONS 移到 INPUT_ACTIONS。clipboard 可能含密码/TOTP,应受 user-activity pause 保护。
|
|
43
|
+
|
|
44
|
+
### P1 Fixed(correctness/robustness)
|
|
45
|
+
|
|
46
|
+
- **isSkylightAvailable 不检查 skylight:true**(utils/input.ts):只查 `"ok"` 不查 `"skylight":true`,SPI 加载失败时仍认为可用。改为 regex 匹配 `/"skylight"\s*:\s*true/`。
|
|
47
|
+
- **keepTargetAxAlive execFileSync 阻塞事件循环 5s**(window.ts):fire-and-forget 但 execFileSync 同步阻塞。改为 setImmediate 让出 tick + timeout 5s→1s。
|
|
48
|
+
- **doScroll 误报 per-pid**(skylight/main.swift):scroll 事件无 field-40/window-loc 标记但返回 `method:"per-pid"`,调用方误以为成功。诚实报告 `hid-tap`。
|
|
49
|
+
- **SENSITIVE_PROCESS_HINTS 不含空格变体 + 缺失条目**(input.ts):`"keychainaccess"` 不匹配 `"keychain access"`(带空格)。改为 normalized 比对(strip 非字母数字)+ 补 keychain/keepassxc/coreauthui/passwords/systemsettings 等。
|
|
50
|
+
|
|
51
|
+
### Tests
|
|
52
|
+
|
|
53
|
+
- 328 passed(修复 clipboard_read 分类测试 + jxaOnlyPlatform 加 skylight:null 避免 keepTargetAxAlive 消耗 mock)。
|
|
54
|
+
|
|
8
55
|
## [0.6.3] - 2026-06-17
|
|
9
56
|
|
|
10
57
|
SKILL.md Confirmation Policy + 修 permissions.test.ts CI 在 Ubuntu 全挂的 bug。
|
|
@@ -24,10 +24,10 @@ export function registerAppTools(registerTool) {
|
|
|
24
24
|
registerTool("wait_for_element", "Poll until an accessibility element matching the criteria reaches the desired state", {
|
|
25
25
|
text: z.string().optional().describe("Element text"), role: z.string().optional().describe("Element role"),
|
|
26
26
|
app: z.string().optional().describe("Target app"),
|
|
27
|
-
timeout: z.number().optional().describe("Timeout ms (default 5000)"),
|
|
28
|
-
timeoutMs: z.number().optional().describe("Alias for timeout"),
|
|
29
|
-
interval: z.number().optional().describe("Poll interval ms (default 500)"),
|
|
30
|
-
intervalMs: z.number().optional().describe("Alias for interval"),
|
|
27
|
+
timeout: z.number().int().min(100).max(60000).optional().describe("Timeout ms (default 5000, max 60000)"),
|
|
28
|
+
timeoutMs: z.number().int().min(100).max(60000).optional().describe("Alias for timeout"),
|
|
29
|
+
interval: z.number().int().min(250).max(10000).optional().describe("Poll interval ms (default 500, min 250)"),
|
|
30
|
+
intervalMs: z.number().int().min(250).max(10000).optional().describe("Alias for interval"),
|
|
31
31
|
until: z.enum(["appear", "disappear", "value_change"]).default("appear").describe("Wait condition: 'appear' (default) waits for a match, 'disappear' waits until no match, 'value_change' waits until first match's value changes"),
|
|
32
32
|
}, async (params) => {
|
|
33
33
|
const deadline = Date.now() + (params.timeout ?? params.timeoutMs ?? 5000);
|
|
@@ -19,6 +19,14 @@ export function registerElementTools(registerTool) {
|
|
|
19
19
|
const safetyCtx = await getSafetyContext(undefined);
|
|
20
20
|
const response = await withSafety({ action: "find_element", params: { ...safetyCtx }, requiresAccessibility: true,
|
|
21
21
|
execute: () => getPlatform().findElement({ text: params.text, role: params.role, app: effectiveApp, depth: params.depth, includeBounds: params.includeBounds, maxResults: params.maxResults, textMode: params.textMode, visibleOnly: params.visibleOnly, value: params.value, index: params.index, near: params.near }) });
|
|
22
|
+
// Security: mask password/secret field values before returning to the model.
|
|
23
|
+
const SENSITIVE_RE = /password|passwd|secret|pincode|pin\b|token|credential|api[_-]?key|access[_-]?key|private[_-]?key/i;
|
|
24
|
+
const SENSITIVE_ROLES = new Set(["AXSecureTextField", "AXPasswordField"]);
|
|
25
|
+
for (const r of response.results) {
|
|
26
|
+
if (SENSITIVE_ROLES.has(r.role) || SENSITIVE_RE.test(r.name || "")) {
|
|
27
|
+
r.value = "[REDACTED]";
|
|
28
|
+
}
|
|
29
|
+
}
|
|
22
30
|
const payload = { results: response.results, metrics: response.metrics };
|
|
23
31
|
if (response.results.length === 0 && effectiveApp && response.metrics.scannedCount === 0) {
|
|
24
32
|
payload.hint =
|
|
@@ -60,13 +60,20 @@ async function buildScreenDescription(opts) {
|
|
|
60
60
|
}
|
|
61
61
|
try {
|
|
62
62
|
const result = await platform.ocr(opts.display);
|
|
63
|
+
const blocks = result.elements.slice(0, opts.ocrBlocks);
|
|
64
|
+
// Security: redact OCR blocks that look like passwords/secrets/tokens.
|
|
65
|
+
// This is best-effort (pixel-based OCR can't know field context) but
|
|
66
|
+
// catches high-entropy strings adjacent to "password"/"secret" labels.
|
|
67
|
+
for (const b of blocks) {
|
|
68
|
+
if (SENSITIVE_NAME_RE.test(b.text)) {
|
|
69
|
+
b.text = "[REDACTED]";
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
// Rebuild fullText from (possibly redacted) blocks.
|
|
73
|
+
const fullText = blocks.map((b) => b.text).join("\n");
|
|
63
74
|
return {
|
|
64
75
|
ok: true,
|
|
65
|
-
value: {
|
|
66
|
-
blocks: result.elements.slice(0, opts.ocrBlocks),
|
|
67
|
-
fullText: result.fullText,
|
|
68
|
-
status: "ok",
|
|
69
|
-
},
|
|
76
|
+
value: { blocks, fullText, status: "ok" },
|
|
70
77
|
};
|
|
71
78
|
}
|
|
72
79
|
catch (e) {
|
|
@@ -148,7 +148,10 @@ export async function clickElement(elementId, app) {
|
|
|
148
148
|
}
|
|
149
149
|
// JXA requested a coordinate click → perform it via the input layer (per-process
|
|
150
150
|
// when a pid is available, so the global cursor does not move).
|
|
151
|
-
if (result.needCoordinateClick
|
|
151
|
+
if (result.needCoordinateClick) {
|
|
152
|
+
if (typeof result.cx !== "number" || typeof result.cy !== "number") {
|
|
153
|
+
throw new ElementNotFoundError(`${elementId}: element requested coordinate click but reported no bounds (offscreen?)`);
|
|
154
|
+
}
|
|
152
155
|
await this.click(Math.round(result.cx), Math.round(result.cy), "left", false);
|
|
153
156
|
return { method: "coordinate", verified: false };
|
|
154
157
|
}
|
|
@@ -564,7 +567,10 @@ export async function clickMenuBarExtra(app, selector = {}) {
|
|
|
564
567
|
if (!result.success) {
|
|
565
568
|
throw new Error(`click_menu_bar_extra failed in ${app}: ${result.error}`);
|
|
566
569
|
}
|
|
567
|
-
if (result.needCoordinateClick
|
|
570
|
+
if (result.needCoordinateClick) {
|
|
571
|
+
if (typeof result.cx !== "number" || typeof result.cy !== "number") {
|
|
572
|
+
throw new ElementNotFoundError(`menu bar extra in ${app}: requested coordinate click but no bounds`);
|
|
573
|
+
}
|
|
568
574
|
await this.click(Math.round(result.cx), Math.round(result.cy), "left", false);
|
|
569
575
|
return { method: "coordinate", verified: false };
|
|
570
576
|
}
|
|
@@ -47,10 +47,14 @@ export function appNameMatches(processName, requestedApp) {
|
|
|
47
47
|
const requested = normalizeAppName(requestedApp);
|
|
48
48
|
if (!process || !requested)
|
|
49
49
|
return false;
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
//
|
|
53
|
-
|
|
50
|
+
if (process === requested)
|
|
51
|
+
return true;
|
|
52
|
+
// Substring match only for requests >= 3 chars to avoid "code"→"vscode".
|
|
53
|
+
// Only allow process.includes(requested) (not bidirectional) to prevent
|
|
54
|
+
// short requests greedily absorbing longer unrelated process names.
|
|
55
|
+
if (requested.length >= 3 && process.includes(requested))
|
|
56
|
+
return true;
|
|
57
|
+
return false;
|
|
54
58
|
}
|
|
55
59
|
export function selectWindowForApp(windows, requestedApp) {
|
|
56
60
|
const requested = normalizeAppName(requestedApp);
|
|
@@ -9,9 +9,12 @@ import { rethrowInputError, errorMessage } from "./helpers.js";
|
|
|
9
9
|
* manager or auth dialog. These fall back to HID-tap (which at least requires
|
|
10
10
|
* the app to be frontmost / visible to the user).
|
|
11
11
|
*/
|
|
12
|
+
// Normalized (non-alphanumeric-stripped) so "Keychain Access" → "keychainaccess" matches.
|
|
12
13
|
const SENSITIVE_PROCESS_HINTS = [
|
|
13
|
-
"keychainaccess", "1password", "lastpass", "bitwarden", "
|
|
14
|
-
"authd", "loginwindow", "applekeychain", "dashlane",
|
|
14
|
+
"keychainaccess", "keychain", "1password", "lastpass", "bitwarden", "keepassxc",
|
|
15
|
+
"securityagent", "coreauthui", "authd", "loginwindow", "applekeychain", "dashlane",
|
|
16
|
+
"enpass", "keeper", "passwords", // macOS Passwords app (Sequoia+)
|
|
17
|
+
"systempreferences", "systemsettings", // where ucu-mcp's own permissions live
|
|
15
18
|
];
|
|
16
19
|
/** Resolve the per-process event target from the active focus_app target (pid + windowNumber). */
|
|
17
20
|
function targetOf() {
|
|
@@ -19,7 +22,8 @@ function targetOf() {
|
|
|
19
22
|
if (!t || !t.pid || t.pid <= 0)
|
|
20
23
|
return undefined;
|
|
21
24
|
// Security: never route per-process events into sensitive apps (password managers / auth).
|
|
22
|
-
|
|
25
|
+
// Normalize by stripping non-alphanumerics so "Keychain Access" → "keychainaccess".
|
|
26
|
+
const name = (t.appName || "").toLowerCase().replace(/[^a-z0-9]/g, "");
|
|
23
27
|
if (SENSITIVE_PROCESS_HINTS.some((h) => name.includes(h))) {
|
|
24
28
|
return undefined; // forces HID-tap fallback (requires frontmost)
|
|
25
29
|
}
|
|
@@ -108,16 +108,25 @@ export async function keepTargetAxAlive(pid) {
|
|
|
108
108
|
const helperPath = resolveNativeHelper.call(this, "skylight", "skylight-helper");
|
|
109
109
|
if (!helperPath)
|
|
110
110
|
return;
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
111
|
+
// Defer to next tick so execFileSync doesn't block the current event loop tick.
|
|
112
|
+
// Lower timeout (1s) — keepAlive should be near-instant; 5s was too long.
|
|
113
|
+
await new Promise((resolve) => {
|
|
114
|
+
setImmediate(() => {
|
|
115
|
+
try {
|
|
116
|
+
execFileSync(helperPath, [], {
|
|
117
|
+
input: JSON.stringify({ command: "keepAlive", pid }),
|
|
118
|
+
encoding: "utf8",
|
|
119
|
+
timeout: 1000,
|
|
120
|
+
});
|
|
121
|
+
}
|
|
122
|
+
catch {
|
|
123
|
+
// keepAlive is best-effort; ignore failures (helper missing / SPI unavailable).
|
|
124
|
+
}
|
|
125
|
+
finally {
|
|
126
|
+
resolve();
|
|
127
|
+
}
|
|
116
128
|
});
|
|
117
|
-
}
|
|
118
|
-
catch {
|
|
119
|
-
// keepAlive is best-effort; ignore failures (helper missing / SPI unavailable).
|
|
120
|
-
}
|
|
129
|
+
});
|
|
121
130
|
}
|
|
122
131
|
export async function getActiveBrowserContext(app) {
|
|
123
132
|
const appName = app || this.activeTarget?.appName;
|
|
@@ -24,7 +24,7 @@ export interface SafetyGuardConfig {
|
|
|
24
24
|
}
|
|
25
25
|
/** Actions that observe UI/system state without altering it. */
|
|
26
26
|
export declare const OBSERVE_ACTIONS: ReadonlySet<string>;
|
|
27
|
-
/** Actions that synthesize user input — need full user-activity protection. */
|
|
27
|
+
/** Actions that synthesize user input OR access sensitive data — need full user-activity protection. */
|
|
28
28
|
export declare const INPUT_ACTIONS: ReadonlySet<string>;
|
|
29
29
|
export declare function classifyAction(action: string): "observe" | "input" | "other";
|
|
30
30
|
export declare class SafetyGuard {
|
package/dist/src/safety/guard.js
CHANGED
|
@@ -69,9 +69,15 @@ const DEFAULT_TEXT_INJECTION_PATTERNS = [
|
|
|
69
69
|
{ pattern: /\$\s*\(/, reason: "shell command substitution" },
|
|
70
70
|
{ pattern: /`[^`]+`/, reason: "shell backtick substitution" },
|
|
71
71
|
{ pattern: /&&|\|\|/, reason: "shell command chaining" },
|
|
72
|
-
{ pattern: /\|\s*(sh|bash|zsh|python|ruby|perl|node)\b/i, reason: "piping into an interpreter" },
|
|
72
|
+
{ pattern: /\|\s*(sh|bash|zsh|python|ruby|perl|node|deno|bun)\b/i, reason: "piping into an interpreter" },
|
|
73
73
|
{ pattern: /\b(sudo\s+rm|rm\s+-rf|mkfs|diskutil\s+erase|dd\s+if=|chmod\s+-R\s+777)\b/i, reason: "dangerous shell command" },
|
|
74
74
|
{ pattern: /\b(ObjC\.import|Application\s*\(|do\s+shell\s+script)\b/, reason: "AppleScript/JXA injection primitive" },
|
|
75
|
+
{ pattern: /\b(tell\s+(application|app)\s+)/i, reason: "AppleScript tell injection" },
|
|
76
|
+
// URL schemes dangerous when pasted into browser address bars or rich-text fields
|
|
77
|
+
{ pattern: /^\s*(javascript|data|vbscript)\s*:/i, reason: "dangerous URL scheme (code execution when pasted)" },
|
|
78
|
+
{ pattern: /\b(javascript|data)\s*:\s*(text\/html|application\/)/i, reason: "dangerous URL scheme with MIME type" },
|
|
79
|
+
// Newline-followed-by-command (bypasses && / || chaining detection)
|
|
80
|
+
{ pattern: /\n\s*(rm\s|sudo\s|curl\s|wget\s|bash\s|sh\s|osascript\s)/i, reason: "newline-separated shell command" },
|
|
75
81
|
];
|
|
76
82
|
// ---------------------------------------------------------------------------
|
|
77
83
|
// Helpers
|
|
@@ -84,13 +90,21 @@ const MODIFIER_CANONICAL = {
|
|
|
84
90
|
};
|
|
85
91
|
/** Normalize a shortcut string to lowercase, trimmed, sorted modifiers with
|
|
86
92
|
* modifier aliases canonicalized (alt→option, ctrl→control, cmd→command). */
|
|
93
|
+
/** Canonical key name aliases — so blocklist entries match all spellings. */
|
|
94
|
+
const KEY_CANONICAL = {
|
|
95
|
+
escape: "esc",
|
|
96
|
+
delete: "del",
|
|
97
|
+
backspace: "del",
|
|
98
|
+
return: "enter",
|
|
99
|
+
option: "alt",
|
|
100
|
+
};
|
|
87
101
|
function normalizeShortcut(raw) {
|
|
88
102
|
return raw
|
|
89
103
|
.toLowerCase()
|
|
90
104
|
.split("+")
|
|
91
105
|
.map((s) => {
|
|
92
106
|
const t = s.trim();
|
|
93
|
-
return MODIFIER_CANONICAL[t] ?? t;
|
|
107
|
+
return MODIFIER_CANONICAL[t] ?? KEY_CANONICAL[t] ?? t;
|
|
94
108
|
})
|
|
95
109
|
.sort()
|
|
96
110
|
.join("+");
|
|
@@ -111,7 +125,6 @@ export const OBSERVE_ACTIONS = new Set([
|
|
|
111
125
|
"wait",
|
|
112
126
|
"wait_for_element",
|
|
113
127
|
"doctor",
|
|
114
|
-
"clipboard_read",
|
|
115
128
|
// focus_app only sets the active target context via AppleScript activate
|
|
116
129
|
// and an AX window lookup — it does not synthesize mouse or keyboard input,
|
|
117
130
|
// so the user-activity pause must not block it. (OpenCode 0.3.7 follow-up)
|
|
@@ -119,7 +132,7 @@ export const OBSERVE_ACTIONS = new Set([
|
|
|
119
132
|
// describe_screen reads screen state (OCR + AX), no input synthesis.
|
|
120
133
|
"describe_screen",
|
|
121
134
|
]);
|
|
122
|
-
/** Actions that synthesize user input — need full user-activity protection. */
|
|
135
|
+
/** Actions that synthesize user input OR access sensitive data — need full user-activity protection. */
|
|
123
136
|
export const INPUT_ACTIONS = new Set([
|
|
124
137
|
"click",
|
|
125
138
|
"double_click",
|
|
@@ -132,6 +145,9 @@ export const INPUT_ACTIONS = new Set([
|
|
|
132
145
|
"type_in_element",
|
|
133
146
|
"set_value",
|
|
134
147
|
"click_menu_bar_extra",
|
|
148
|
+
// clipboard_read accesses potentially sensitive data (passwords/TOTP copied
|
|
149
|
+
// by password managers) — treat as input so the user-activity pause applies.
|
|
150
|
+
"clipboard_read",
|
|
135
151
|
"clipboard_write",
|
|
136
152
|
]);
|
|
137
153
|
export function classifyAction(action) {
|
package/dist/src/utils/input.js
CHANGED
|
@@ -65,7 +65,7 @@ function isSkylightAvailable() {
|
|
|
65
65
|
encoding: "utf8",
|
|
66
66
|
timeout: 3000,
|
|
67
67
|
});
|
|
68
|
-
_skylightAvailable =
|
|
68
|
+
_skylightAvailable = /"skylight"\s*:\s*true/.test(stdout);
|
|
69
69
|
}
|
|
70
70
|
catch {
|
|
71
71
|
_skylightAvailable = false;
|
|
@@ -103,17 +103,28 @@ function runInputChecked(payload, target) {
|
|
|
103
103
|
// skylight crashed/timed out → fall through to cgevent.
|
|
104
104
|
}
|
|
105
105
|
}
|
|
106
|
-
// cgevent path (HID-tap) — the reliable fallback.
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
106
|
+
// cgevent path (HID-tap) — the reliable fallback. Guard against ENOENT
|
|
107
|
+
// (cgevent-helper missing) when only skylight was available.
|
|
108
|
+
if (!isCgeventAvailable()) {
|
|
109
|
+
throw new Error("input dispatch failed: no native helper available (both cgevent and skylight unavailable or errored)");
|
|
110
|
+
}
|
|
111
|
+
try {
|
|
112
|
+
const raw = execFileSync(resolvedCgeventPath, [], {
|
|
113
|
+
input: JSON.stringify(payload),
|
|
114
|
+
encoding: "utf8",
|
|
115
|
+
timeout: 10000,
|
|
116
|
+
}).trim();
|
|
117
|
+
const resp = JSON.parse(raw);
|
|
118
|
+
if (resp.error) {
|
|
119
|
+
throw new Error(`native helper error: ${resp.error}`);
|
|
120
|
+
}
|
|
121
|
+
return "hid-tap";
|
|
122
|
+
}
|
|
123
|
+
catch (e) {
|
|
124
|
+
if (e instanceof Error && e.message.startsWith("native helper error:"))
|
|
125
|
+
throw e;
|
|
126
|
+
throw new Error(`input dispatch failed: cgevent helper unavailable (${e.message})`);
|
|
127
|
+
}
|
|
117
128
|
}
|
|
118
129
|
/** @deprecated use runInputChecked — kept for external callers/tests. */
|
|
119
130
|
function runNativeChecked(payload) {
|
|
@@ -354,7 +365,7 @@ export async function scroll(x, y, deltaX, deltaY, _platform = process.platform,
|
|
|
354
365
|
// ── Keyboard operations (CGEvent — background) ────────────────────────────
|
|
355
366
|
export async function typeText(text, delay = 20, _platform = process.platform, target) {
|
|
356
367
|
if (isDryRun()) {
|
|
357
|
-
logDryRun("typeText", {
|
|
368
|
+
logDryRun("typeText", { charCount: text.length, delay }); // don't log text content (may contain passwords)
|
|
358
369
|
return;
|
|
359
370
|
}
|
|
360
371
|
if (!text)
|
|
@@ -462,6 +473,12 @@ export async function typeText(text, delay = 20, _platform = process.platform, t
|
|
|
462
473
|
}
|
|
463
474
|
else {
|
|
464
475
|
// Fallback: use osascript keystroke for unsupported chars (emoji, CJK, etc.)
|
|
476
|
+
// NOTE: keystroke goes to the FRONTMOST app, not the target pid. When a
|
|
477
|
+
// target is set this may type into the wrong window. Log a warning so the
|
|
478
|
+
// caller knows non-ASCII chars bypassed per-process routing.
|
|
479
|
+
if (target?.pid) {
|
|
480
|
+
logger.warn("typeText non-ASCII fallback uses global keystroke (types into frontmost, not target pid)");
|
|
481
|
+
}
|
|
465
482
|
const escaped = escapeAppleScriptString(batch.chars);
|
|
466
483
|
await execFileAsync("/usr/bin/osascript", [
|
|
467
484
|
"-e", `tell application "System Events" to keystroke "${escaped}"`,
|
|
@@ -119,10 +119,12 @@ func makeMouseEvent(_ type: CGEventType, at point: CGPoint, button: CGMouseButto
|
|
|
119
119
|
case .mouseMoved: nsType = .mouseMoved
|
|
120
120
|
default: return nil
|
|
121
121
|
}
|
|
122
|
+
// pressure: 0 for mouseUp/mouseMoved, 1.0 for mouseDown/drag (real hardware behavior)
|
|
123
|
+
let isRelease = nsType == .leftMouseUp || nsType == .rightMouseUp || nsType == .otherMouseUp || nsType == .mouseMoved
|
|
122
124
|
guard let ns = NSEvent.mouseEvent(
|
|
123
125
|
with: nsType, location: point, modifierFlags: [],
|
|
124
126
|
timestamp: 0, windowNumber: windowNumber, context: nil,
|
|
125
|
-
eventNumber: 0, clickCount: clickCount, pressure:
|
|
127
|
+
eventNumber: 0, clickCount: clickCount, pressure: isRelease ? 0 : 1.0
|
|
126
128
|
), let cg = ns.cgEvent else { return nil }
|
|
127
129
|
return cg
|
|
128
130
|
}
|
|
@@ -139,6 +141,12 @@ func postPerPid(_ event: CGEvent, pid: Int32, at point: CGPoint, windowNumber: I
|
|
|
139
141
|
event.setIntegerValueField(.mouseEventWindowUnderMousePointerThatCanHandleThisEvent, value: Int64(windowNumber))
|
|
140
142
|
}
|
|
141
143
|
// Private SPIs: window-local point + Chromium pid latch (field 40).
|
|
144
|
+
// NOTE: setWindowLoc expects window-LOCAL coordinates, but we pass the screen-global
|
|
145
|
+
// point. For AppKit apps this is correct (AppKit hit-tests use NSEvent.location which
|
|
146
|
+
// is global/flipped). For Chromium's internal windowLocation field this may be offset
|
|
147
|
+
// by the window origin — but field-40 (pid latch) is sufficient for Chromium to accept
|
|
148
|
+
// the event in practice. Full window-origin subtraction would require threading bounds
|
|
149
|
+
// through the Input struct (future improvement).
|
|
142
150
|
setWindowLoc?(event, point)
|
|
143
151
|
setIntField?(event, 40, Int64(pid))
|
|
144
152
|
event.timestamp = clock_gettime_nsec_np(CLOCK_UPTIME_RAW)
|
|
@@ -152,9 +160,11 @@ func postHidTap(_ event: CGEvent) {
|
|
|
152
160
|
}
|
|
153
161
|
|
|
154
162
|
/// True if the target pid is the frontmost app (canvas/GPU apps filter per-pid
|
|
155
|
-
/// routes — must use HID-tap there).
|
|
163
|
+
/// routes — must use HID-tap there). Uses NSWorkspace.frontmostApplication for
|
|
164
|
+
/// an unambiguous "is this pid frontmost right now" check (isActive is unreliable
|
|
165
|
+
/// for LSUIElement/agent processes like SystemUIServer).
|
|
156
166
|
func isFrontmost(_ pid: Int32) -> Bool {
|
|
157
|
-
return
|
|
167
|
+
return NSWorkspace.shared.frontmostApplication?.processIdentifier == pid
|
|
158
168
|
}
|
|
159
169
|
|
|
160
170
|
// MARK: - Focus-without-raise (SLPSPostEventRecordTo)
|
|
@@ -167,7 +177,7 @@ func focusWithoutRaise(pid: Int32, windowID: Int) {
|
|
|
167
177
|
windowID > 0 else { return }
|
|
168
178
|
var prevPSN = [UInt32](repeating: 0, count: 2)
|
|
169
179
|
var targetPSN = [UInt32](repeating: 0, count: 2)
|
|
170
|
-
let prevOk = prevPSN.withUnsafeMutableBytes { raw in frontFn(raw.baseAddress!)
|
|
180
|
+
let prevOk = prevPSN.withUnsafeMutableBytes { raw in frontFn(raw.baseAddress!) == 0 }
|
|
171
181
|
if prevOk {
|
|
172
182
|
let cid = connFn()
|
|
173
183
|
var ownerCid: UInt32 = 0
|
|
@@ -190,6 +200,24 @@ func focusWithoutRaise(pid: Int32, windowID: Int) {
|
|
|
190
200
|
_ = targetPSN.withUnsafeBytes { psnRaw in buf.withUnsafeBufferPointer { bp in postRec(psnRaw.baseAddress!, bp.baseAddress!) } }
|
|
191
201
|
}
|
|
192
202
|
|
|
203
|
+
/// Chromium user-activation primer: leading mouseMoved + off-screen click at (-1,-1).
|
|
204
|
+
/// Required before any real mouse event on Chromium/Electron targets, otherwise
|
|
205
|
+
/// the renderer's user-activation gate filters the event silently.
|
|
206
|
+
/// Called by doClick, doDoubleClick, doDrag before their target events.
|
|
207
|
+
func primeTarget(pid: Int32, at loc: CGPoint, button: CGMouseButton, windowNumber winNum: Int) {
|
|
208
|
+
// Leading mouseMoved at target (cursor-state primer).
|
|
209
|
+
if let mv = makeMouseEvent(.mouseMoved, at: loc, button: button, windowNumber: winNum) {
|
|
210
|
+
postPerPid(mv, pid: pid, at: loc, windowNumber: winNum, button: button); usleep(15_000)
|
|
211
|
+
}
|
|
212
|
+
// Off-screen primer click @ (-1,-1) — satisfies Chromium user-activation gate.
|
|
213
|
+
let off = CGPoint(x: -1, y: -1)
|
|
214
|
+
if let pd = makeMouseEvent(.leftMouseDown, at: off, button: .left, windowNumber: winNum),
|
|
215
|
+
let pu = makeMouseEvent(.leftMouseUp, at: off, button: .left, windowNumber: winNum) {
|
|
216
|
+
postPerPid(pd, pid: pid, at: off, windowNumber: winNum); usleep(1_000)
|
|
217
|
+
postPerPid(pu, pid: pid, at: off, windowNumber: winNum); usleep(100_000)
|
|
218
|
+
}
|
|
219
|
+
}
|
|
220
|
+
|
|
193
221
|
// MARK: - AX keepalive (root-cause fix for Electron/Tauri AXPress silent failure)
|
|
194
222
|
|
|
195
223
|
/// Keep the target app's AX tree alive when occluded/background.
|
|
@@ -234,17 +262,7 @@ func doClick(_ p: Input) -> String {
|
|
|
234
262
|
return "{\"ok\":true,\"method\":\"hid-tap\"}"
|
|
235
263
|
}
|
|
236
264
|
focusWithoutRaise(pid: pid, windowID: winNum)
|
|
237
|
-
|
|
238
|
-
if let mv = makeMouseEvent(.mouseMoved, at: loc, button: b, windowNumber: winNum) {
|
|
239
|
-
postPerPid(mv, pid: pid, at: loc, windowNumber: winNum, button: b); usleep(15_000)
|
|
240
|
-
}
|
|
241
|
-
// Off-screen primer click @ (-1,-1) — satisfies Chromium user-activation gate.
|
|
242
|
-
let off = CGPoint(x: -1, y: -1)
|
|
243
|
-
if let pd = makeMouseEvent(.leftMouseDown, at: off, button: .left, windowNumber: winNum),
|
|
244
|
-
let pu = makeMouseEvent(.leftMouseUp, at: off, button: .left, windowNumber: winNum) {
|
|
245
|
-
postPerPid(pd, pid: pid, at: off, windowNumber: winNum); usleep(1_000)
|
|
246
|
-
postPerPid(pu, pid: pid, at: off, windowNumber: winNum); usleep(100_000)
|
|
247
|
-
}
|
|
265
|
+
primeTarget(pid: pid, at: loc, button: b, windowNumber: winNum)
|
|
248
266
|
// Target click.
|
|
249
267
|
guard let dn = makeMouseEvent(downT(b), at: loc, button: b, windowNumber: winNum),
|
|
250
268
|
let up = makeMouseEvent(upT(b), at: loc, button: b, windowNumber: winNum)
|
|
@@ -263,6 +281,7 @@ func doDoubleClick(_ p: Input) -> String {
|
|
|
263
281
|
return hidTapDouble(loc: loc, b: b)
|
|
264
282
|
}
|
|
265
283
|
focusWithoutRaise(pid: pid, windowID: winNum)
|
|
284
|
+
primeTarget(pid: pid, at: loc, button: b, windowNumber: winNum)
|
|
266
285
|
for state in [1, 2] {
|
|
267
286
|
guard let dn = makeMouseEvent(downT(b), at: loc, button: b, clickCount: state, windowNumber: winNum),
|
|
268
287
|
let up = makeMouseEvent(upT(b), at: loc, button: b, clickCount: state, windowNumber: winNum)
|
|
@@ -324,6 +343,7 @@ func doDrag(_ p: Input) -> String {
|
|
|
324
343
|
return "{\"ok\":true,\"method\":\"hid-tap\"}"
|
|
325
344
|
}
|
|
326
345
|
focusWithoutRaise(pid: pid, windowID: winNum)
|
|
346
|
+
primeTarget(pid: pid, at: from, button: b, windowNumber: winNum)
|
|
327
347
|
guard let dn = makeMouseEvent(downT(b), at: from, button: b, windowNumber: winNum)
|
|
328
348
|
else { return "{\"error\":\"fail\"}" }
|
|
329
349
|
postPerPid(dn, pid: pid, at: from, windowNumber: winNum, button: b)
|
|
@@ -342,16 +362,12 @@ func doDrag(_ p: Input) -> String {
|
|
|
342
362
|
}
|
|
343
363
|
|
|
344
364
|
func doScroll(_ p: Input) -> String {
|
|
345
|
-
// Scroll events
|
|
346
|
-
//
|
|
365
|
+
// Scroll events via SLEventPostToPid lack field-40/window-location stamping
|
|
366
|
+
// (postPerPid is mouse-click-specific), so per-pid scroll is unreliable.
|
|
367
|
+
// Honestly report hid-tap so the caller's warning path fires correctly.
|
|
347
368
|
let dy = Int32(-(p.deltaY ?? 0)); let dx = Int32(p.deltaX ?? 0)
|
|
348
369
|
guard let ev = CGEvent(scrollWheelEvent2Source: nil, units: .pixel, wheelCount: 2, wheel1: dy, wheel2: dx, wheel3: 0)
|
|
349
370
|
else { return "{\"error\":\"fail\"}" }
|
|
350
|
-
// Best-effort per-pid if we have a pid; scroll doesn't move the cursor noticeably anyway.
|
|
351
|
-
if let pid = p.pid, pid > 0, slPostToPid != nil, !isFrontmost(pid) {
|
|
352
|
-
slPostToPid?(pid, ev)
|
|
353
|
-
return "{\"ok\":true,\"method\":\"per-pid\"}"
|
|
354
|
-
}
|
|
355
371
|
postHidTap(ev)
|
|
356
372
|
return "{\"ok\":true,\"method\":\"hid-tap\"}"
|
|
357
373
|
}
|
|
Binary file
|