ucu-mcp 0.4.0 → 0.4.3

package/CHANGELOG.md

@@ -5,7 +5,54 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.4.0] - 2026-06-11
+## [0.4.2] - 2026-06-13
+
+### Security
+
+- **JXA escaping unified (P0/P1)**: All 8 JXA code-injection sites that built strings via manual `.replace()` escaping now use `JSON.stringify()` for every interpolated value. Eliminates shell-substitution, backtick, newline, and AppleScript injection vectors across `restoreFocus`, `focusApp`, `getActiveBrowserContext`, `getWindowState`, `ocrJxa`, `findElement`, `clickElement`, `typeInElement`, `setElementValue`. (SEC-P0-1, SEC-P0-2, SEC-P1-2, SEC-P1-3, ERR-P1-6)
+- **`isScreenLocked` fail-closed (P1)**: Previously returned `false` when the `ioreg` check threw, letting actions proceed on a potentially locked screen. Now returns `true` on error so actions are blocked until lock state can be confirmed. (ERR-P1-4)
+- **Window-skip / URL blocklist guards activated (P1)**: `SafetyGuard`'s `windowTitle` and `url` checks were dead code because no tool handler passed those fields. New `getSafetyContext()` helper resolves both from the active target and is spread into all 11 action tools' `withSafety` params. (SEC-P1-1)
+
+### Fixed
+
+- `listApps` and `listWindowsJxa` now wrap their `osascript` calls in `try/catch` and rethrow via `rethrowAccessibilityError`, so a permission failure surfaces as `PermissionError("accessibility")` with a recovery hint instead of a generic `PlatformError`. (ERR-P1-3)
+- `getScreenSize` logs the failure and returns an `{ estimated: true }` flag on the fallback `1920x1080` value, instead of silently returning a default that callers cannot distinguish from a real measurement. (ERR-P1-5)
+
+### Changed
+
+- **`macos.ts` split into 10 domain modules**: The 1995-line monolith is now `base.ts`, `helpers.ts`, `focus.ts`, `screen.ts`, `window.ts`, `ax-tree.ts`, `input.ts`, `element.ts`, `clipboard.ts`, `index.ts` under `src/platform/macos/`. `base.ts` defines the class and re-binds methods; all `(this as any)` casts removed.
+- **`tools.ts` split into 7 domain modules**: The 908-line tool registry is now `helpers.ts`, `screen-tools.ts`, `input-tools.ts`, `keyboard-tools.ts`, `element-tools.ts`, `app-tools.ts`, `index.ts` under `src/mcp/tools/`. A `ToolRegistry` class + `registerTool` callback pattern replaces the flat registration.
+- **JXA helper templates extracted**: New `src/platform/jxa-helpers.ts` (216 lines) centralizes the `childElements`, `resolveElementByFullPath`, `resolveElementInApp`, `elemString`, `getBounds`, `isVisible`, `descriptorMatches`, `scoreEquivalent`, `refetchEquivalent` JXA functions. `element.ts` and `ax-tree.ts` now import and interpolate these instead of inlining 3× duplicated copies.
+- `FindElementResult` type now has formal `subrole?: string` and `identifier?: string` fields; the `as any` casts that read them are gone.
+
+### Tests
+
+- 279 tests pass (13 unit + 2 integration), 12 skipped (2 GUI smoke suites gated by env vars). +34 new security tests: clipboard injection patterns (shell substitution, backtick, chaining, piping, JXA/AppleScript injection), permission-denied paths for all AX element tools, and platform-method integration coverage.
+- Verified on Node v22.22.3 / macOS 26.6 (arm64). `npm run build` compiles 3 native Swift helpers.
+
+## [0.4.1] - 2026-06-11
+
+### Changed
+
+- `UcuError` base class now has a formal `hint?: string` field with serialization in `toJSON()`. Platform errors that carry remediation hints (e.g. `WindowNotFoundError` from `focusApp` for Electron apps) now pass the hint through the constructor instead of using duck-type property assignment. (Fox 0.3.8 M1)
+- `findElementInputSchema` export annotated with `@internal` — not part of the public API, may change without semver bump. (Raman 0.3.7 M4)
+- `list_windows` empty-result branch now uses the top-level `checkPermission` import instead of a per-call dynamic `import()`. (Fox 0.3.8 M2)
+- `doctor` `tried` arrays are now `readonly string[]` and no longer silently truncated via `.slice(0, 3)`. (Fox 0.3.8 N2/N3)
+- Text-side and value-side regex pre-validation tests consolidated into a single `it.each` parameterized case. (Raman 0.3.7 N2)
+
+### Fixed
+
+- `find_element` now attaches a pixel-level fallback hint when it returns 0 results AND `scannedCount === 0` for a specific app (meaning the AX tree is empty — common with Electron/Chromium apps). The hint directs the model to use `screenshot` + `ocr` + `click(x, y)` instead of retrying `find_element` forever.
+- CHANGELOG 0.3.7 `prepublishOnly` description now matches the actual script (`npx vitest run tests/unit/ && npm run build` instead of `npm test && npm run build`). (Raman 0.3.7 M2)
+- CHANGELOG 0.3.7 scope claim "corrected in both `src/platform/macos.ts` and this CHANGELOG" narrowed to "corrected in this CHANGELOG" — the source file was already correct. (Raman 0.3.7 M3)
+- CHANGELOG 0.3.5 duplicate "three fewer" / "two fewer" entry cleaned up — only the corrected "two fewer" version remains. (Raman 0.3.7 N4)
+- Test comment "modern V8" clarified to "Node >= 12 / V8" for the all-no-bounds near-sort test. (Raman 0.3.7 N1)
+
+### Tests
+
+- 225 unit tests pass (13 test files).
+- `macos`: value-side and text-side regex pre-validation tests consolidated into single `it.each` with preserved regression context.
+- Verified on Node v22.22.3 / macOS 26.6 (arm64).
 
 ### Fixed
 
@@ -43,7 +90,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Fixed
 
 - `find_element` value-schema test is no longer a tautology. The 0.3.6 release fixed a *symptom* of the bug (the old test called `handler()` directly, bypassing the McpServer schema-validation wrapper, and then asserted `r.isError === true` which was `undefined`); the underlying tautology remained: the test re-created a local `z.string().min(1).optional()` instead of exercising the real schema. 0.3.7 exports the actual `findElementInputSchema` from `src/mcp/tools.ts` and the test now imports it via `findElementInputSchema.value`, so the assertion genuinely pins the production schema. Pins the 0.3.2 commit `46d4ddd` semantic.
-- CHANGELOG/JXA `textMatches` comment math is now correct: 3 sources → 1 RegExp = **2 fewer** compilations per matched element. The 0.3.5/0.3.6 wording "three fewer" was off by one and has been corrected in both `src/platform/macos.ts` and this CHANGELOG.
+- CHANGELOG/JXA `textMatches` comment math is now correct: 3 sources → 1 RegExp = **2 fewer** compilations per matched element. The 0.3.5/0.3.6 wording "three fewer" was off by one and has been corrected in this CHANGELOG (the `src/platform/macos.ts` comment was already correct at that point).
 
 ### Tests
 
@@ -53,7 +100,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ### Hygiene
 
 - `findElementInputSchema` is now a named export from `src/mcp/tools.ts` (with a JSDoc comment explaining why the schema is exported) so the unit test can assert the production schema directly instead of constructing a local copy.
-- Added `prepublishOnly` script to `package.json` that runs `npm test && npm run build` before `npm publish`. This is a structural guard against the yank rhythm that hit 0.3.3 and 0.3.5: a failed test or build will now block the publish at the npm level, not at the human level. (Raman review Minor #3)
+- Added `prepublishOnly` script to `package.json` that runs `npx vitest run tests/unit/ && npm run build` before `npm publish`. This is a structural guard against the yank rhythm that hit 0.3.3 and 0.3.5: a failed test or build will now block the publish at the npm level, not at the human level. (Raman review Minor #3)
 
 ## [0.3.5] - 2026-06-06  *(Yanked — see 0.3.6)*
 
@@ -65,7 +112,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 
-- JXA `textMatches` regex branch now compiles the `RegExp` once per element instead of once per source (name / value / description) — three fewer compilations per matched element when `textMode="regex"`. The TS-side pre-validation in `findElement` guarantees the pattern is valid, so the `RegExp` constructor cannot throw here. (Herschel review perf Minor)
 - JXA `textMatches` regex branch now compiles the `RegExp` once per element instead of once per source (name / value / description) — **two** fewer compilations per matched element when `textMode="regex"` (corrected in 0.3.7; 0.3.5/0.3.6 said "three fewer" which was off by one: 3 sources → 1 regex = 2 saved). The TS-side pre-validation in `findElement` guarantees the pattern is valid, so the `RegExp` constructor cannot throw here. (Herschel review perf Minor)
 
 ### Fixed

package/dist/bin/ucu-mcp.js

@@ -1,7 +1,7 @@
 #!/usr/bin/env node
 import { startServer } from "../src/mcp/server.js";
 import { checkPermissions } from "../src/safety/permissions.js";
-import { MacOSPlatform } from "../src/platform/macos.js";
+import { MacOSPlatform } from "../src/platform/macos/index.js";
 async function runDoctor() {
     const permissions = await checkPermissions();
     const screenLocked = process.platform === "darwin"

package/dist/src/index.d.ts

@@ -1,7 +1,7 @@
 export { startServer } from "./mcp/server.js";
-export { ToolRegistry } from "./mcp/tools.js";
+export { ToolRegistry } from "./mcp/tools/index.js";
 export { createStdioTransport } from "./mcp/transport.js";
 export { Platform } from "./platform/base.js";
 export { SafetyGuard } from "./safety/guard.js";
 export { checkPermissions, checkPermission, type PermissionCheckResult, type PermissionType, } from "./safety/permissions.js";
-export { MacOSPlatform } from "./platform/macos.js";
+export { MacOSPlatform } from "./platform/macos/index.js";

package/dist/src/index.js

@@ -1,6 +1,6 @@
 export { startServer } from "./mcp/server.js";
-export { ToolRegistry } from "./mcp/tools.js";
+export { ToolRegistry } from "./mcp/tools/index.js";
 export { createStdioTransport } from "./mcp/transport.js";
 export { SafetyGuard } from "./safety/guard.js";
 export { checkPermissions, checkPermission, } from "./safety/permissions.js";
-export { MacOSPlatform } from "./platform/macos.js";
+export { MacOSPlatform } from "./platform/macos/index.js";

package/dist/src/mcp/server.js

@@ -3,7 +3,7 @@ import { existsSync, readFileSync } from "node:fs";
 import { dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
 import { createStdioTransport } from "./transport.js";
-import { registerTools, startUserActivityMonitor } from "./tools.js";
+import { registerTools, startUserActivityMonitor } from "./tools/index.js";
 const UCU_MCP_INSTRUCTIONS = `
 UCU-MCP is a cross-client computer-use server for Claude Code CLI/Desktop, OpenCode, and other MCP clients.
 

package/dist/src/mcp/tools/app-tools.d.ts

@@ -0,0 +1,2 @@
+import { type RegisterToolFn } from "./helpers.js";
+export declare function registerAppTools(registerTool: RegisterToolFn): void;

package/dist/src/mcp/tools/app-tools.js

@@ -0,0 +1,225 @@
+import { z } from "zod";
+import { checkPermission } from "../../safety/permissions.js";
+import { PermissionError } from "../../util/errors.js";
+import { createLogger } from "../../util/logger.js";
+import { metrics } from "../../util/metrics.js";
+import { getPlatform, getActiveTarget, setActiveTarget, withSafety, } from "./helpers.js";
+const log = createLogger("tools");
+export function registerAppTools(registerTool) {
+    registerTool("list_apps", "List all running applications", {}, async () => {
+        const apps = await withSafety({ action: "list_apps", params: {}, requiresAccessibility: true, execute: async () => getPlatform().listApps() });
+        return { content: [{ type: "text", text: JSON.stringify(apps, null, 2) }] };
+    });
+    registerTool("focus_app", "Select an application/window as the active target context", {
+        app: z.string().describe("Application name to focus"),
+    }, async (params) => {
+        const target = await withSafety({ action: "focus_app", params: {}, requiresAccessibility: true, execute: () => getPlatform().focusApp(params.app) });
+        setActiveTarget(target);
+        return { content: [{ type: "text", text: JSON.stringify(target, null, 2) }] };
+    });
+    registerTool("wait", "Wait for a specified duration", { ms: z.number().int().min(1).max(60000).describe("Duration in milliseconds (1–60000)") }, async (params) => {
+        await new Promise(r => setTimeout(r, params.ms));
+        return { content: [{ type: "text", text: JSON.stringify({ waited: params.ms }) }] };
+    });
+    registerTool("wait_for_element", "Poll until an accessibility element matching the criteria reaches the desired state", {
+        text: z.string().optional().describe("Element text"), role: z.string().optional().describe("Element role"),
+        app: z.string().optional().describe("Target app"),
+        timeout: z.number().optional().describe("Timeout ms (default 5000)"),
+        timeoutMs: z.number().optional().describe("Alias for timeout"),
+        interval: z.number().optional().describe("Poll interval ms (default 500)"),
+        intervalMs: z.number().optional().describe("Alias for interval"),
+        until: z.enum(["appear", "disappear", "value_change"]).default("appear").describe("Wait condition: 'appear' (default) waits for a match, 'disappear' waits until no match, 'value_change' waits until first match's value changes"),
+    }, async (params) => {
+        const deadline = Date.now() + (params.timeout ?? params.timeoutMs ?? 5000);
+        const interval = params.interval ?? params.intervalMs ?? 500;
+        const until = params.until ?? "appear";
+        const effectiveApp = params.app || getActiveTarget()?.appName;
+        const query = { text: params.text, role: params.role, app: effectiveApp, maxResults: 1 };
+        const { granted } = await checkPermission("accessibility");
+        if (!granted)
+            throw new PermissionError("accessibility", process.platform);
+        let initialValue;
+        let hasInitial = false;
+        while (Date.now() < deadline) {
+            const response = await getPlatform().findElement(query);
+            const matched = response.results[0];
+            if (until === "appear") {
+                if (matched)
+                    return { content: [{ type: "text", text: JSON.stringify({ found: true, element: matched }, null, 2) }] };
+            }
+            else if (until === "disappear") {
+                if (!matched)
+                    return { content: [{ type: "text", text: JSON.stringify({ found: true, reason: "disappeared" }, null, 2) }] };
+            }
+            else {
+                if (matched) {
+                    if (!hasInitial) {
+                        initialValue = matched.value;
+                        hasInitial = true;
+                    }
+                    else if (matched.value !== initialValue) {
+                        return { content: [{ type: "text", text: JSON.stringify({ found: true, oldValue: initialValue, newValue: matched.value }, null, 2) }] };
+                    }
+                }
+            }
+            await new Promise(r => setTimeout(r, interval));
+        }
+        const reason = until === "value_change" ? (hasInitial ? "value_unchanged" : "never_appeared") : "timeout";
+        return { content: [{ type: "text", text: JSON.stringify({ found: false, reason }, null, 2) }] };
+    });
+    registerTool("doctor", "Check system permissions, native helpers, and client readiness", {}, async () => {
+        const { checkPermissions, getPermissionInstructions, getTerminalAppName } = await import("../../safety/permissions.js");
+        const { existsSync, statSync } = await import("node:fs");
+        const { join, dirname, resolve } = await import("node:path");
+        const { fileURLToPath } = await import("node:url");
+        const { execFileSync } = await import("node:child_process");
+        const permissions = await checkPermissions();
+        let screenLocked = false;
+        try {
+            screenLocked = getPlatform().isScreenLocked?.() ?? false;
+        }
+        catch {
+            // Non-darwin platform or uninitialized — screen lock not applicable.
+        }
+        const termApp = process.platform === "darwin" ? getTerminalAppName() : undefined;
+        function resolveHelperPath(relParts) {
+            const tried = [];
+            const tryPaths = [];
+            const moduleDir = dirname(fileURLToPath(import.meta.url));
+            const argv1 = process.argv[1] ? resolve(process.argv[1]) : "";
+            const argv1Dir = argv1 ? dirname(argv1) : "";
+            tryPaths.push(join(process.cwd(), ...relParts));
+            if (argv1Dir) {
+                tryPaths.push(join(argv1Dir, ...relParts));
+                tryPaths.push(join(argv1Dir, "..", ...relParts));
+                tryPaths.push(join(argv1Dir, "..", "..", ...relParts));
+            }
+            tryPaths.push(join(moduleDir, "..", ...relParts));
+            tryPaths.push(join(moduleDir, "..", "..", ...relParts));
+            tryPaths.push(join(moduleDir, "..", "..", "..", ...relParts));
+            tryPaths.push(join(moduleDir, "..", "..", "..", "..", ...relParts));
+            if (process.platform === "darwin") {
+                try {
+                    const npmRoot = execFileSync("npm", ["root", "-g"], { encoding: "utf-8", timeout: 2000 }).trim();
+                    if (npmRoot) {
+                        tryPaths.push(join(npmRoot, "ucu-mcp", ...relParts));
+                    }
+                }
+                catch { /* npm not on PATH is fine */ }
+            }
+            for (const p of tryPaths) {
+                tried.push(p);
+                try {
+                    if (existsSync(p) && statSync(p).isFile())
+                        return { path: p, tried };
+                }
+                catch { /* skip */ }
+            }
+            return { path: null, tried };
+        }
+        let nativeHelpers;
+        if (process.platform === "darwin") {
+            const cgevent = resolveHelperPath(["native", "cgevent", "cgevent-helper"]);
+            const ocr = resolveHelperPath(["native", "ocr", "ocr-helper"]);
+            const windowlist = resolveHelperPath(["native", "windowlist", "windowlist-helper"]);
+            nativeHelpers = {
+                cgevent: { ok: cgevent.path !== null, path: cgevent.path, tried: cgevent.tried },
+                ocr: { ok: ocr.path !== null, path: ocr.path, tried: ocr.tried },
+                windowlist: { ok: windowlist.path !== null, path: windowlist.path, tried: windowlist.tried },
+            };
+        }
+        let readiness = "ready";
+        const issues = [];
+        if (!permissions.granted) {
+            readiness = "blocked";
+            for (const m of (permissions.missing ?? [])) {
+                issues.push(`Missing macOS permission: ${m}`);
+            }
+        }
+        if (screenLocked) {
+            readiness = "blocked";
+            issues.push("Screen is locked");
+        }
+        if (process.platform === "darwin" && nativeHelpers) {
+            if (!nativeHelpers.cgevent.ok) {
+                readiness = readiness === "ready" ? "degraded" : readiness;
+                issues.push("Native CGEvent helper not found (input synthesis may crash on macOS Sequoia+). Run `npm run build` to compile it, or reinstall ucu-mcp so the helper ships from the tarball.");
+            }
+            if (!nativeHelpers.ocr.ok) {
+                readiness = readiness === "ready" ? "degraded" : readiness;
+                issues.push("Native OCR helper not found (OCR may fail on macOS Sequoia+). Run `npm run build` to compile it, or reinstall ucu-mcp so the helper ships from the tarball.");
+            }
+            if (!nativeHelpers.windowlist.ok) {
+                readiness = readiness === "ready" ? "degraded" : readiness;
+                issues.push("Native windowlist helper not found (window enumeration will fall back to slow JXA). Run `npm run build` to compile it.");
+            }
+        }
+        const electronHint = "If the target app is Electron (e.g. CC Switch, VS Code, Discord), list_windows may return [] even with Accessibility granted to your terminal. Grant Accessibility to the Electron app itself in System Settings > Privacy & Security > Accessibility, and restart the app. Pixel-level workaround: use screenshot + ocr to locate UI elements by text, then click(x, y) at the detected bounding box coordinates. Alternatively, modify the app\'s config file or database directly.";
+        const clients = {};
+        for (const bin of ["claude", "codex", "opencode", "npx"]) {
+            try {
+                const path = execFileSync("which", [bin], { encoding: "utf-8", timeout: 2000 }).trim();
+                clients[bin] = path || "not found";
+            }
+            catch {
+                clients[bin] = "not found";
+            }
+        }
+        const recommendations = [];
+        if (readiness === "blocked") {
+            for (const m of (permissions.missing ?? [])) {
+                const app = termApp ?? "your terminal app";
+                recommendations.push(`${m}: ${getPermissionInstructions(m)} (Grant to ${app}.)`);
+            }
+            if (screenLocked)
+                recommendations.push("Unlock the screen, then retry.");
+        }
+        if (readiness !== "ready") {
+            if (process.platform === "darwin" && nativeHelpers && (!nativeHelpers.cgevent.ok || !nativeHelpers.ocr.ok)) {
+                recommendations.push("Run `npm run build` in the ucu-mcp project to compile native Swift helpers (cgevent-helper, ocr-helper, windowlist-helper).");
+            }
+            if (process.platform === "darwin" && nativeHelpers && !nativeHelpers.windowlist.ok) {
+                recommendations.push("windowlist helper missing — list_windows will fall back to JXA (~3-6s, unreliable for Electron). Run `npm run build`.");
+            }
+        }
+        if (readiness === "ready") {
+            recommendations.push("All checks passed. MCP client can proceed with automation.");
+        }
+        else if (process.platform === "darwin") {
+            recommendations.push(electronHint);
+        }
+        const report = {
+            readiness,
+            issues: issues.length > 0 ? issues : undefined,
+            recommendations,
+            platform: process.platform,
+            node: process.version,
+            permissions,
+            screenLocked,
+            terminalApp: termApp,
+            nativeHelpers,
+            clients,
+            safety: {
+                urlBlocklist: true,
+                lockScreenGuard: process.platform === "darwin",
+                typedTextInjectionScan: true,
+            },
+            stdioCommand: "ucu-mcp",
+            metrics: {
+                global: metrics.stats(),
+                byTool: metrics.byTool(),
+            },
+        };
+        return { content: [{ type: "text", text: JSON.stringify(report, null, 2) }] };
+    });
+    registerTool("clipboard_read", "Read the current contents of the system clipboard", {}, async () => {
+        const text = await withSafety({ action: "clipboard_read", params: {}, execute: () => getPlatform().readClipboard() });
+        return { content: [{ type: "text", text: JSON.stringify({ text }, null, 2) }] };
+    });
+    registerTool("clipboard_write", "Write text to the system clipboard (text injection patterns are blocked)", {
+        text: z.string().describe("Text to place on the clipboard"),
+    }, async (params) => {
+        await withSafety({ action: "clipboard_write", params: { text: params.text }, execute: () => getPlatform().writeClipboard(params.text) });
+        return { content: [{ type: "text", text: JSON.stringify({ written: true }, null, 2) }] };
+    });
+}

package/dist/src/mcp/tools/element-tools.d.ts

@@ -0,0 +1,23 @@
+import { z } from "zod";
+import { type RegisterToolFn } from "./helpers.js";
+export declare const findElementInputSchema: {
+    text: z.ZodOptional<z.ZodString>;
+    role: z.ZodOptional<z.ZodString>;
+    app: z.ZodOptional<z.ZodString>;
+    depth: z.ZodOptional<z.ZodNumber>;
+    includeBounds: z.ZodDefault<z.ZodBoolean>;
+    maxResults: z.ZodDefault<z.ZodNumber>;
+    textMode: z.ZodDefault<z.ZodEnum<{
+        contains: "contains";
+        exact: "exact";
+        regex: "regex";
+    }>>;
+    visibleOnly: z.ZodDefault<z.ZodBoolean>;
+    value: z.ZodOptional<z.ZodString>;
+    index: z.ZodOptional<z.ZodNumber>;
+    near: z.ZodOptional<z.ZodObject<{
+        x: z.ZodNumber;
+        y: z.ZodNumber;
+    }, z.core.$strip>>;
+};
+export declare function registerElementTools(registerTool: RegisterToolFn): void;

package/dist/src/mcp/tools/element-tools.js

@@ -0,0 +1,59 @@
+import { z } from "zod";
+import { getPlatform, getActiveTarget, getSafetyContext, withSafety, actionResponse, captureAfterFields, } from "./helpers.js";
+export const findElementInputSchema = {
+    text: z.string().optional().describe("Text to search"),
+    role: z.string().optional().describe("AX role"),
+    app: z.string().optional().describe("Target app"),
+    depth: z.number().optional().describe("AX tree depth"),
+    includeBounds: z.boolean().default(true).describe("Include bounds"),
+    maxResults: z.number().min(1).max(200).default(50).describe("Max results"),
+    textMode: z.enum(["contains", "exact", "regex"]).default("contains").describe("Text matching mode: contains (default), exact, or regex"),
+    visibleOnly: z.boolean().default(false).describe("Only return elements with valid on-screen bounds"),
+    value: z.string().min(1).optional().describe("Filter by AX element value (text/regex/exact, see textMode). Empty string is treated as unset (omit the field instead)."),
+    index: z.number().int().nonnegative().optional().describe("Return only the Nth match (0-based) after all other filtering and sorting"),
+    near: z.object({ x: z.number(), y: z.number() }).optional().describe("Sort results by ascending distance to this point and return closest first"),
+};
+export function registerElementTools(registerTool) {
+    registerTool("find_element", "Find accessibility elements by text, role, or value. Supports value/index/near selectors.", findElementInputSchema, async (params) => {
+        const effectiveApp = params.app || getActiveTarget()?.appName;
+        const safetyCtx = await getSafetyContext(undefined);
+        const response = await withSafety({ action: "find_element", params: { ...safetyCtx }, requiresAccessibility: true,
+            execute: () => getPlatform().findElement({ text: params.text, role: params.role, app: effectiveApp, depth: params.depth, includeBounds: params.includeBounds, maxResults: params.maxResults, textMode: params.textMode, visibleOnly: params.visibleOnly, value: params.value, index: params.index, near: params.near }) });
+        const payload = { results: response.results, metrics: response.metrics };
+        if (response.results.length === 0 && effectiveApp && response.metrics.scannedCount === 0) {
+            payload.hint =
+                `${effectiveApp} returned 0 AX elements (scannedCount=0, meaning the AX tree is empty). ` +
+                    "This is typical for Electron/Chromium apps whose AX tree is not exposed to System Events. " +
+                    "Pixel-level workaround: call screenshot to capture the screen, then ocr to locate " +
+                    "the target UI text and get its bounding box coordinates, then click(x, y) at those " +
+                    "screen coordinates. Alternatively, use type_text or press_key for keyboard-based " +
+                    "interaction, or modify the app's config file or database directly.";
+        }
+        return { content: [{ type: "text", text: JSON.stringify(payload, null, 2) }] };
+    });
+    registerTool("click_element", "Click an accessibility element by its ID", {
+        elementId: z.string().describe("AX element identifier"), app: z.string().optional().describe("Target app"), ...captureAfterFields,
+    }, async (params) => {
+        const effectiveApp = params.app || getActiveTarget()?.appName;
+        const safetyCtx = await getSafetyContext();
+        await withSafety({ action: "click_element", params: { ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().clickElement(params.elementId, effectiveApp) });
+        return actionResponse("click_element", { clicked: true, elementId: params.elementId }, { elementId: params.elementId, app: effectiveApp }, params.captureAfter, params.captureFormat, params.captureMaxWidth);
+    });
+    registerTool("set_value", "Set the value of an accessibility element", {
+        elementId: z.string().describe("AX element identifier"), value: z.string().describe("Value to set"), app: z.string().optional().describe("Target app"), ...captureAfterFields,
+    }, async (params) => {
+        const effectiveApp = params.app || getActiveTarget()?.appName;
+        const safetyCtx = await getSafetyContext();
+        await withSafety({ action: "set_value", params: { value: params.value, ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().setElementValue(params.elementId, params.value, effectiveApp) });
+        return actionResponse("set_value", { setValue: true, elementId: params.elementId }, { elementId: params.elementId, app: effectiveApp }, params.captureAfter, params.captureFormat, params.captureMaxWidth);
+    });
+    registerTool("type_in_element", "Type text into an accessibility element, optionally clearing first", {
+        elementId: z.string().describe("AX element identifier"), text: z.string().describe("Text to type"),
+        app: z.string().optional().describe("Target app"), clearFirst: z.boolean().optional().describe("Clear existing text before typing"), ...captureAfterFields,
+    }, async (params) => {
+        const effectiveApp = params.app || getActiveTarget()?.appName;
+        const safetyCtx = await getSafetyContext();
+        await withSafety({ action: "type_in_element", params: { text: params.text, ...safetyCtx }, requiresAccessibility: true, execute: () => getPlatform().typeInElement(params.elementId, params.text, effectiveApp, params.clearFirst) });
+        return actionResponse("type_in_element", { typed: true, elementId: params.elementId, charCount: params.text.length }, { elementId: params.elementId, app: effectiveApp }, params.captureAfter, params.captureFormat, params.captureMaxWidth);
+    });
+}

package/dist/src/mcp/tools/helpers.d.ts

@@ -0,0 +1,84 @@
+import { z } from "zod";
+import type { Platform, AppTarget } from "../../platform/base.js";
+import { SafetyGuard } from "../../safety/guard.js";
+export declare function getPlatform(): Platform;
+/** @internal Test-only injection point. */
+export declare function __setPlatformForTesting(platform: Platform | undefined): void;
+export declare const safety: SafetyGuard;
+export declare function getActiveTarget(): AppTarget | undefined;
+export declare function setActiveTarget(target: AppTarget): void;
+export declare const captureAfterFields: {
+    captureAfter: z.ZodDefault<z.ZodBoolean>;
+    captureMaxWidth: z.ZodDefault<z.ZodNumber>;
+    captureFormat: z.ZodDefault<z.ZodEnum<{
+        png: "png";
+        jpeg: "jpeg";
+    }>>;
+};
+export declare function resolvePoint(x: number, y: number, windowId?: string): Promise<{
+    x: number;
+    y: number;
+}>;
+export declare function getSafetyContext(windowId?: string): Promise<{
+    windowTitle?: string;
+    url?: string;
+}>;
+export type ToolContent = {
+    type: "text";
+    text: string;
+} | {
+    type: "image";
+    data: string;
+    mimeType: string;
+};
+export type ToolResult = {
+    content: ToolContent[];
+    isError?: boolean;
+};
+export declare function jsonText(value: unknown): ToolContent;
+export declare function recoveryHint(code: string): string;
+export declare function errorDetails(error: unknown): Record<string, unknown>;
+export interface ActionReceipt {
+    actionId: string;
+    action: string;
+    status: "ok" | "partial" | "blocked";
+    target: {
+        app?: string;
+        windowId?: string;
+        elementId?: string;
+        x?: number;
+        y?: number;
+        startX?: number;
+        startY?: number;
+        endX?: number;
+        endY?: number;
+    };
+    result: Record<string, unknown>;
+    capture: {
+        requested: boolean;
+        status: "ok" | "skipped" | "error";
+        format?: string;
+        maxWidth?: number;
+        error?: Record<string, unknown>;
+    };
+    warnings: string[];
+    next: string;
+}
+export declare function buildActionReceipt(action: string, status: ActionReceipt["status"], target: ActionReceipt["target"], result: Record<string, unknown>, captureRequested: boolean, captureFormat?: string, captureMaxWidth?: number, captureError?: Record<string, unknown>, warnings?: string[]): ActionReceipt;
+export declare function mcpErrorResponse(error: unknown): ToolResult;
+export declare function actionResponse(action: string, result: Record<string, unknown>, target: ActionReceipt["target"], captureAfter?: boolean, captureFormat?: "png" | "jpeg", captureMaxWidth?: number, warnings?: string[]): Promise<{
+    content: ToolContent[];
+}>;
+export interface SafetyAction {
+    action: string;
+    params: Record<string, unknown>;
+    requiresAccessibility?: boolean;
+    requiresScreenRecording?: boolean;
+    skipUserActivityPause?: boolean;
+    dryRun?: () => Promise<string>;
+    execute: () => Promise<unknown>;
+}
+export type RegisterToolFn = (name: string, description: string, schema: Record<string, z.ZodTypeAny>, handler: (params: any) => Promise<ToolResult>) => void;
+export declare function withSafety<T>(sa: SafetyAction): Promise<T>;
+export declare function startUserActivityMonitor(): void;
+export declare function stopUserActivityMonitor(): void;