ucu-mcp 0.1.3 → 0.3.0

package/CHANGELOG.md

@@ -5,32 +5,30 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [0.1.0] - 2025-06-02
+## [0.3.0] - 2026-06-06
+
 
 ### Added
 
-- Initial release of UCU-MCP (Universal Computer Use MCP Server)
-- **22 MCP tools** for desktop automation via Model Context Protocol:
-  - Screen capture: `screen_capture`, `screen_capture_active_window`
-  - Mouse control: `mouse_move`, `mouse_click`, `mouse_double_click`, `mouse_drag`, `mouse_scroll`
-  - Keyboard control: `keyboard_type`, `keyboard_hotkey`, `keyboard_key`
-  - Clipboard: `clipboard_read`, `clipboard_write`
-  - Window management: `window_list`, `window_activate`, `window_close`
-  - Application control: `app_launch`, `app_quit`
-  - System: `system_info`, `process_list`, `process_terminate`
-  - Safety: `doctor` command for permission and environment diagnostics
-- **Safety features**:
-  - URL blocklist to prevent navigation to sensitive sites
-  - Lock screen guard (macOS) — blocks automation when screen is locked
-  - Typed text injection scan — validates keyboard input before injection
-  - Focus steal suppression — prevents accidental focus changes during automation
-  - User interaction monitor — tracks user activity for safety coordination
-- **macOS platform support** with Accessibility API integration
-- TypeScript-first codebase with full type definitions
-- CLI entry point with `doctor` diagnostic command
+- Scenario-based MCP Instructions — tool-usage guidance organized by task pattern (form fill, menu bar click, screen read, app switch, verify action, wait for change, recover stale target, clipboard)
+- findElement multi-strategy: `value` filter (AX value, respects textMode), `index` selector (0-based Nth match), `near` sorter (ascending distance to point)
+- wait_for_element `until` parameter: `appear` (default), `disappear` (poll until gone), `value_change` (poll until first match value differs)
+- Action Receipt v1 — unified receipt structure for all action-class tools (click, double_click, scroll, drag, move, type_text, press_key, click_element, set_value, type_in_element)
+- Receipt fields: actionId (base36-timestamp unique ID), action, status (ok/partial/blocked), target (location context), result (business result), capture (screenshot metadata), warnings, next (suggested next step)
+- Partial receipt when action succeeds but post-action screenshot fails: status="partial", capture.error contains error details, warnings includes "Post-action screenshot capture failed"
+- Target Session v1 — `focus_app` now returns stable target metadata (`targetId`, `appName`, `pid`, `windowId`, `title`, `capturedAt`) for follow-up tool calls
+- `TARGET_STALE` structured errors for active target windows that disappear before `get_window_state`
 
 ### Changed
 
+- MCP instructions rewritten from generic description to scenario-driven workflow recommendations
+- `wait_for_element` description updated to reflect `until` parameter semantics
+- `find_element` schema extended with `value`, `index`, `near` parameters
+- Action tool responses now wrap business results under `result` instead of returning them at the top level
+- captureAfter failures now surface through receipt.capture.error instead of a flat captureError object
+- `get_window_state` can use the prior `focus_app` target when `windowId` is omitted
+- AX tools (`find_element`, `wait_for_element`, `click_element`, `set_value`, `type_in_element`) can use the prior `focus_app` target when `app` is omitted
+
 - Rewrote `src/mcp/tools.ts` with comprehensive 22-tool registry:
   - Unified `withSafety` wrapper for all automation actions
   - `captureAfter` helper for post-action screenshots

package/README.md

@@ -78,10 +78,10 @@ UCU-MCP provides 22 tools across five categories:
 
 | Tool | Description | Key Parameters |
 |------|-------------|----------------|
-| `screenshot` | Capture screen, window, or region as base64 PNG/JPEG | `display?`, `windowId?`, `region?`, `maxWidth?`, `format?` |
+| `screenshot` | Capture screen, window, or region as PNG/JPEG image content | `display?`, `windowId?`, `region?`, `maxWidth?`, `format?` |
 | `list_windows` | List all on-screen windows with IDs, titles, bounds | `includeMinimized?` |
 | `list_apps` | List visible macOS apps with pid, frontmost state, and window count | — |
-| `focus_app` | Select an app/window target context without raising it | `app` |
+| `focus_app` | Select an app/window target context for later AX tools; returns `targetId`, `appName`, `pid`, `windowId`, `title`, and `capturedAt` | `app` |
 | `get_window_state` | Get accessibility tree of a window, or the prior focus_app target when windowId is omitted | `windowId?`, `depth?`, `includeBounds?` |
 | `get_screen_size` | Get screen dimensions | `display?` |
 | `ocr` | Perform OCR on screen or region; returns text with bounding boxes and confidence | `display?`, `region?` |
@@ -92,9 +92,9 @@ UCU-MCP provides 22 tools across five categories:
 |------|-------------|----------------|
 | `click` | Click at screen coordinates (non-invasive) | `x`, `y`, `windowId?`, `button?` |
 | `double_click` | Double-click at screen coordinates | `x`, `y`, `windowId?`, `button?` |
-| `scroll` | Scroll at a position (vertical/horizontal) | `x`, `y`, `deltaX?`, `deltaY`, `captureAfter?` |
-| `drag` | Drag from one position to another | `startX`, `startY`, `endX`, `endY`, `duration?`, `button?`, `captureAfter?` |
-| `move` | Move the physical cursor to a position (invasive) | `x`, `y` |
+| `scroll` | Scroll at a position (vertical/horizontal) | `x`, `y`, `deltaX?`, `deltaY`, `windowId?`, `captureAfter?` |
+| `drag` | Drag from one position to another | `startX`, `startY`, `endX`, `endY`, `windowId?`, `duration?`, `button?`, `captureAfter?` |
+| `move` | Move the physical cursor to a position (invasive) | `x`, `y`, `windowId?`, `captureAfter?` |
 | `get_cursor_position` | Get current cursor position | — |
 
 ### Keyboard
@@ -102,14 +102,14 @@ UCU-MCP provides 22 tools across five categories:
 | Tool | Description | Key Parameters |
 |------|-------------|----------------|
 | `type_text` | Type text into the currently focused element via OS key events (not clipboard) | `text`, `delay?`, `captureAfter?` |
-| `press_key` | Press key or keyboard shortcut in the focused window | `key`, `modifiers?`, `captureAfter?` |
+| `press_key` | Press key or keyboard shortcut in the focused window | `key?`, `modifiers?`, `keys?`, `captureAfter?` |
 
 ### AX Element Interaction
 
 | Tool | Description | Key Parameters |
 |------|-------------|----------------|
-| `find_element` | Find UI element by text, role, or description using AX APIs | `text?`, `role?`, `app?`, `depth?`, `includeBounds?`, `maxResults?` |
-| `click_element` | Click an AX element by its id (from find_element); refetches equivalent elements after UI updates | `elementId`, `app?`, `captureAfter?` |
+| `find_element` | Find UI element by text, role, or description using AX APIs, using the current focus_app target when app is omitted | `text?`, `role?`, `app?`, `depth?`, `includeBounds?`, `maxResults?` |
+| `click_element` | Click an AX element by its id (from find_element), using the current focus_app target when app is omitted; refetches equivalent elements after UI updates | `elementId`, `app?`, `captureAfter?` |
 | `set_value` | Set an AX element's value directly without focusing it, using the current focus_app target when app is omitted | `elementId`, `value`, `app?`, `captureAfter?` |
 | `type_in_element` | Type text into a specific AX text field element; may focus the element and refetches equivalent elements after UI updates | `elementId`, `text`, `app?`, `clearFirst?`, `captureAfter?` |
 
@@ -118,13 +118,15 @@ UCU-MCP provides 22 tools across five categories:
 | Tool | Description | Key Parameters |
 |------|-------------|----------------|
 | `doctor` | Check platform readiness, permissions, lock-screen state, and client integration hints | — |
-| `wait` | Wait for UI state to settle after launches, animations, or navigation | `ms?` |
-| `wait_for_element` | Poll the AX tree until a matching element appears | `text?`, `role?`, `app?`, `timeoutMs?`, `intervalMs?` |
+| `wait` | Wait for UI state to settle after launches, animations, or navigation | `ms` |
+| `wait_for_element` | Poll the AX tree until a matching element appears | `text?`, `role?`, `app?`, `timeout?`, `timeoutMs?`, `interval?`, `intervalMs?` |
 
-Action tools accept `captureAfter`, `captureMaxWidth`, and `captureFormat` so an agent can receive a post-action screenshot in the same MCP response instead of spending another round trip on `screenshot`.
+Action tools accept `captureAfter`, `captureMaxWidth`, and `captureFormat` so an agent can receive a post-action screenshot as a second MCP image content item in the same response instead of spending another round trip on `screenshot`. When `captureAfter` is requested and the action succeeds, the tool returns an `ActionReceipt` (see the Action Receipt section below) with `capture.status: "ok"`. If post-action capture fails, the receipt has `status: "partial"` and `capture.status: "error"` with the error details. If `captureAfter` is omitted, `capture.status` is `"skipped"`.
 
 For fast AX discovery on large windows, use `find_element` with `includeBounds=false` and a small `maxResults`. Keep bounds enabled when the result may be used for coordinate fallback.
 
+`focus_app` establishes a session target for follow-up observation and AX actions. After focusing an app, `get_window_state` may omit `windowId`, and AX tools may omit `app`. If the focused window closes or is replaced, UCU-MCP returns a structured `TARGET_STALE` error so the agent can refresh with `focus_app` or `list_windows` instead of silently acting on a different target.
+
 ## OCR Tool Usage
 
 The `ocr` tool captures a screenshot and runs optical character recognition, returning each detected text element with its position and confidence score.
@@ -153,11 +155,14 @@ The `ocr` tool captures a screenshot and runs optical character recognition, ret
 
 ```json
 {
-  "text": "Detected text here",
+  "fullText": "Detected text here",
   "elements": [
     {
       "text": "Hello",
-      "bounds": { "x": 120, "y": 210, "width": 80, "height": 24 },
+      "x": 120,
+      "y": 210,
+      "width": 80,
+      "height": 24,
       "confidence": 0.97
     }
   ]
@@ -207,6 +212,90 @@ The AX (Accessibility) element tools let you interact with UI controls by their
 }
 ```
 
+## Action Receipt
+
+Action tools (`click`, `double_click`, `scroll`, `drag`, `move`, `type_text`, `press_key`, `click_element`, `set_value`, `type_in_element`) return a unified `ActionReceipt` JSON object that wraps the action result, target information, and optional post-action screenshot metadata.
+
+### Receipt structure
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `actionId` | `string` | Unique base36-timestamp ID (e.g. `a1x9z2k-1`) |
+| `action` | `string` | Tool name that produced this receipt |
+| `status` | `"ok" \| "partial" \| "blocked"` | Overall action status |
+| `target` | `object` | What was acted upon (coordinates, elementId, app, windowId) |
+| `result` | `object` | Original business result (clicked, x, y, etc.) |
+| `capture` | `object` | Screenshot metadata (requested, status, format, maxWidth, error) |
+| `warnings` | `string[]` | Non-fatal warnings array |
+| `next` | `string` | Suggested next action |
+
+### Examples
+
+**Success with captureAfter:**
+
+```json
+{
+  "actionId": "a1x9z2k-1",
+  "action": "click",
+  "status": "ok",
+  "target": { "x": 100, "y": 200 },
+  "result": { "clicked": true, "x": 100, "y": 200 },
+  "capture": {
+    "requested": true,
+    "status": "ok",
+    "format": "jpeg",
+    "maxWidth": 1280
+  },
+  "warnings": [],
+  "next": "find_element or get_window_state"
+}
+```
+
+**Success without captureAfter:**
+
+```json
+{
+  "actionId": "a1x9z2k-2",
+  "action": "click_element",
+  "status": "ok",
+  "target": { "elementId": "AXButton-42", "app": "Safari" },
+  "result": { "clicked": true, "elementId": "AXButton-42" },
+  "capture": {
+    "requested": false,
+    "status": "skipped"
+  },
+  "warnings": [],
+  "next": "find_element or get_window_state"
+}
+```
+
+**Partial when capture fails:**
+
+```json
+{
+  "actionId": "a1x9z2k-3",
+  "action": "click",
+  "status": "partial",
+  "target": { "x": 100, "y": 200 },
+  "result": { "clicked": true, "x": 100, "y": 200 },
+  "capture": {
+    "requested": true,
+    "status": "error",
+    "format": "jpeg",
+    "maxWidth": 1280,
+    "error": {
+      "name": "CaptureError",
+      "code": "CAPTURE_FAILED",
+      "retryable": true,
+      "message": "Screenshot capture failed after action",
+      "recovery": "Check Screen Recording permission and retry."
+    }
+  },
+  "warnings": ["Post-action screenshot capture failed"],
+  "next": "screenshot"
+}
+```
+
 ## macOS Permission Setup
 
 UCU-MCP on macOS requires two system permissions:
@@ -245,10 +334,20 @@ UCU-MCP runs as a stdio MCP server. This is the common integration path for Clau
 
 ### Claude Code CLI
 
+Verified CLI setup:
+
+```bash
+claude mcp add --scope user ucu -- ucu-mcp
+claude mcp list
+```
+
+Equivalent config shape:
+
 ```json
 {
   "mcpServers": {
     "ucu": {
+      "type": "stdio",
       "command": "ucu-mcp"
     }
   }
@@ -263,25 +362,51 @@ Use the same local MCP server shape as Claude Desktop. Grant Accessibility and S
 {
   "mcpServers": {
     "ucu": {
+      "type": "stdio",
       "command": "ucu-mcp"
     }
   }
 }
 ```
 
+### Codex CLI
+
+Verified CLI setup:
+
+```bash
+codex mcp add ucu -- ucu-mcp
+codex mcp list
+```
+
+Equivalent `~/.codex/config.toml` shape:
+
+```toml
+[mcp_servers.ucu]
+command = "ucu-mcp"
+```
+
 ### OpenCode
 
+OpenCode reads MCP servers from `~/.config/opencode/opencode.json`.
+
 ```json
 {
   "mcp": {
-    "ucu": {
+    "ucu-mcp": {
       "type": "local",
+      "enabled": true,
       "command": ["ucu-mcp"]
     }
   }
 }
 ```
 
+Verify with:
+
+```bash
+opencode mcp list
+```
+
 ### Runtime Doctor
 
 ```bash
@@ -362,15 +487,32 @@ src/
 
 ## Error Handling
 
+Tool execution failures return standard MCP tool results with `isError: true`. The first content item is JSON text so clients can make policy decisions without string matching:
+
+```json
+{
+  "error": {
+    "name": "WindowNotFoundError",
+    "code": "WINDOW_NOT_FOUND",
+    "retryable": false,
+    "message": "Window win-1 not found. It may have been closed. Run list_windows to get fresh IDs.",
+    "recovery": "Run list_windows again, then retry with a fresh windowId or omit windowId for screen coordinates."
+  }
+}
+```
+
 | Error Code | Description | Retryable |
 |------------|-------------|-----------|
 | `PLATFORM_ERROR` | Platform API call failed | Yes |
 | `PERMISSION_DENIED` | Missing system permission | No |
 | `SAFETY_BLOCKED` | Blocked by safety rule | No |
 | `WINDOW_NOT_FOUND` | Window does not exist | No |
+| `ELEMENT_NOT_FOUND` | Accessibility element is stale or missing | No |
+| `UNSUPPORTED_PARAMETER` | Valid JSON requested an unsupported parameter combination | No |
 | `COORDINATE_OUT_OF_BOUNDS` | Coordinate outside screen | No |
 | `INPUT_FAILED` | Input synthesis failed | Yes |
 | `CAPTURE_FAILED` | Screenshot/OCR capture failed | Yes |
+| `UNKNOWN_ERROR` | Unexpected internal failure | No |
 
 ## Development
 

package/dist/src/mcp/server.js

@@ -1,25 +1,46 @@
 import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import { existsSync, readFileSync } from "node:fs";
+import { dirname, join } from "node:path";
+import { fileURLToPath } from "node:url";
 import { createStdioTransport } from "./transport.js";
-import { registerTools } from "./tools.js";
+import { registerTools, startUserActivityMonitor } from "./tools.js";
 const UCU_MCP_INSTRUCTIONS = `
-UCU-MCP is a cross-client computer-use server for Claude Code CLI, Claude Code Desktop, OpenCode, and other MCP clients.
+UCU-MCP is a cross-client computer-use server for Claude Code CLI/Desktop, OpenCode, and other MCP clients.
 
-Use screenshots and window state to observe before acting. On macOS, prefer list_apps/focus_app to establish the target app context, then use AX element tools when an element can be identified: find_element, then click_element, set_value, or type_in_element. Fall back to coordinates only when AX lookup is unavailable or ambiguous.
+Pick the right tool sequence for the task:
 
-Before repeated UI work, call get_screen_size and list_windows so coordinates and target windows are explicit. Use get_window_state for structured UI trees. Use ocr when visible text is not exposed through Accessibility. For tight observe-act loops, set captureAfter=true on action tools to receive a post-action screenshot in the same tool response.
+• Fill a form field → find_element (text/role) + type_in_element or set_value. Prefer AX over coordinates.
+• Click a menu bar item → get_screen_size + click with coordinates (menu bar is not in the AX tree).
+• Read what's on screen → screenshot; for text not in AX use ocr; for a structured tree use get_window_state.
+• Switch between apps → list_apps, then focus_app; subsequent tools use the active target context.
+• Verify an action succeeded → captureAfter=true on action tools, or call screenshot afterwards.
+• Wait for UI to change → wait_for_element (until: "appear" default; also "disappear" or "value_change").
+• Recover from TARGET_STALE → call focus_app again for the target app, then retry the action.
+• Read or write the clipboard → clipboard_read / clipboard_write.
 
-Safety model: actions are blocked while macOS is locked, dangerous shortcuts and sensitive windows are blocked, and suspicious injected text is rejected. For text entry into UI controls, prefer type_in_element because it can refetch equivalent AX elements if the UI tree changes.
-
-For Claude Code CLI/Desktop and OpenCode configs, run the ucu-mcp executable over stdio. If tools fail on macOS, run doctor first to check Accessibility and Screen Recording permissions. Windows and Linux adapters are explicit stubs until their native backends are implemented.
+General rules: on macOS call list_apps/focus_app first to establish target context, then prefer AX tools (find_element → click_element / type_in_element / set_value). Use coordinates only when AX lookup is unavailable. Actions are blocked while macOS is locked; dangerous shortcuts and sensitive windows are blocked; suspicious injected text is rejected. type_in_element can refetch equivalent AX elements when the UI tree changes. Run doctor to check Accessibility and Screen Recording permissions. Windows and Linux adapters are explicit stubs until their native backends are implemented.
 `.trim();
+function getPackageVersion() {
+    let dir = dirname(fileURLToPath(import.meta.url));
+    for (let i = 0; i < 6; i++) {
+        const path = join(dir, "package.json");
+        if (existsSync(path)) {
+            const parsed = JSON.parse(readFileSync(path, "utf-8"));
+            return parsed.version ?? "0.0.0";
+        }
+        dir = dirname(dir);
+    }
+    return "0.0.0";
+}
 export async function startServer() {
     const server = new McpServer({
         name: "ucu-mcp",
-        version: "0.1.0",
+        version: getPackageVersion(),
     }, {
         instructions: UCU_MCP_INSTRUCTIONS,
     });
     registerTools(server);
+    startUserActivityMonitor();
     const transport = createStdioTransport();
     await server.connect(transport);
     console.error("ucu-mcp server started on stdio");

package/dist/src/mcp/tools.d.ts

@@ -1,11 +1,17 @@
 /**
  * Tool registry for UCU-MCP.
  *
- * Registers 22 MCP tools on the server and dispatches each call through
+ * Registers 24 MCP tools on the server and dispatches each call through
  * a shared safety/permission/retry pipeline (`withSafety`).
  */
 import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
+import type { AppTarget } from "../platform/base.js";
+/**
+ * Get the currently active target context (set by focus_app).
+ */
+export declare function getActiveTarget(): AppTarget | undefined;
 export declare function startUserActivityMonitor(): void;
+export declare function stopUserActivityMonitor(): void;
 export declare function registerTools(server: McpServer): void;
 export declare class ToolRegistry {
     private static _instance;