ucu-mcp 0.4.2 → 0.5.0

package/skills/ucu-mcp/references/tool-reference.md

@@ -0,0 +1,255 @@
+# Tool Reference
+
+UCU-MCP exposes 26 tools across five categories. All action tools accept
+optional `captureAfter` / `captureMaxWidth` / `captureFormat` parameters that
+screenshot the result and append it to the response.
+
+Coordinate inputs are **screen-absolute** unless noted (window-relative only
+when a `windowId` is explicitly passed).
+
+---
+
+## Screen & Window
+
+### `screenshot`
+Capture the full screen, a region, or a specific window.
+
+| Param | Type | Default | Notes |
+|---|---|---|---|
+| `display` | number | 0 | Display index |
+| `windowId` | string | — | From `list_windows`; captures that window |
+| `region` | `{x,y,width,height}` | — | Mutually exclusive with `windowId` |
+| `format` | `"png"` \| `"jpeg"` | `"png"` | |
+| `maxWidth` | number | 1280 | Resize preserving aspect ratio |
+| `describe` | boolean | false | Append a text `ScreenDescription` block (OCR + AX) after the image |
+| `describeOptions` | object | — | `{axDepth=3, ocrBlocks=50, includeAx=true}` when `describe=true` |
+
+Returns one image content block (+ one text block if `describe=true`). Use
+`describe=true` when image content may not reach the model (relay/URL downgrade).
+
+### `list_windows`
+List visible windows. Returns `WindowInfo[]` (`{id, title, processName, pid,
+bounds, isMinimized, isOnScreen}`). When empty, includes a `diagnostics` hint
+distinguishing permission-denied vs Electron-opacity.
+
+| Param | Type | Default |
+|---|---|---|
+| `includeMinimized` | boolean | false |
+
+### `get_window_state`
+AX tree of a window. Returns `WindowState` = `{window, focusedElement?, tree?}`.
+The `tree` is a depth-limited `ElementInfo` (`{role, name, value, states,
+bounds?, children?}`).
+
+| Param | Type | Default |
+|---|---|---|
+| `windowId` | string | active target |
+| `depth` | number | 3 (capped at 10) |
+| `includeBounds` | boolean | false |
+
+### `get_screen_size`
+Returns `{width, height, scaleFactor, estimated?}`. Synchronous, low-cost.
+
+| Param | Type | Default |
+|---|---|---|
+| `display` | number | 0 |
+
+### `ocr`
+Run Vision OCR on the full screen or a region. Returns `{elements:
+OcrElement[], fullText}`. Each `OcrElement` = `{text, x, y, width, height,
+confidence}` in screen-absolute coordinates.
+
+| Param | Type | Default |
+|---|---|---|
+| `display` | number | 0 |
+| `region` | `{x,y,width,height}` | full screen |
+
+### `describe_screen`
+Structured text description of the screen — the **vision-degraded fallback**.
+Returns `ScreenDescription` = `{capturedAt, screen, foregroundWindow,
+ocr:{blocks, fullText, status}, ax:{elements?, status, windowId?}, errors[]}`.
+Each source is collected independently; failures land in `errors` (never thrown).
+Password fields are masked to `[REDACTED]`.
+
+| Param | Type | Default | Notes |
+|---|---|---|---|
+| `display` | number | 0 | |
+| `ocr` | boolean | true | Requires Screen Recording when true |
+| `includeAx` | boolean | true | Requires Accessibility when true |
+| `axDepth` | number | 3 | Capped at 10 |
+| `ocrBlocks` | number | 50 | Max OCR elements returned |
+| `windowId` | string | active target | AX traversal target |
+
+Use when: image content blocks are not visible to you; you need
+machine-readable layout; you want OCR + AX in one call with graceful failure.
+
+---
+
+## Mouse & Input
+
+All accept `captureAfter` / `captureMaxWidth` / `captureFormat`.
+
+### `click` / `double_click`
+Click at screen coordinates. `button` ∈ `left|right|middle`.
+
+| Param | Type |
+|---|---|
+| `x`, `y` | number |
+| `button` | `"left"` \| `"right"` \| `"middle"` |
+| `windowId` | string (optional, makes x/y window-relative) |
+
+### `scroll`
+| Param | Type | Notes |
+|---|---|---|
+| `x`, `y` | number | position |
+| `deltaX`, `deltaY` | number | negative deltaY = scroll up |
+
+### `drag`
+| Param | Type |
+|---|---|
+| `startX`, `startY`, `endX`, `endY` | number |
+| `button` | `"left"` \| `"right"` \| `"middle"` |
+| `duration` | number (ms) |
+
+### `move`
+Move cursor without clicking. Params: `x`, `y`.
+
+### `get_cursor_position`
+Returns `{x, y}`.
+
+---
+
+## Keyboard
+
+### `type_text`
+Type a string at the current cursor position (CGEvent background injection).
+
+| Param | Type |
+|---|---|
+| `text` | string |
+| `delay` | number (ms, optional) |
+
+### `press_key`
+Press a key combo. Supports special keys, single letters a–z, single digits 0–9.
+
+| Param | Type | Notes |
+|---|---|---|
+| `key` | string | e.g. `"enter"`, `"m"`, `"5"` |
+| `keys` | string[] | alternative to `key` for multi-tap |
+| `modifiers` | string[] | `cmd`, `shift`, `alt`/`option`, `ctrl`/`control`, `capslock` |
+
+Blocked combos: `cmd+q`, `cmd+shift+q`, `cmd+option+q`, `cmd+l`, `alt+f4`,
+`ctrl+alt+del` (logout/lock).
+
+---
+
+## AX Element Interaction
+
+These operate on the active target's window (set via `focus_app`). Prefer these
+over coordinate clicks.
+
+### `find_element`
+Find AX elements by text/role/value. Returns `{results: FindElementResult[],
+metrics}`. Each result has an `id` for use in the element tools below. When 0
+results, includes a hint guiding to `screenshot`+`ocr`+`click(x,y)` (Electron
+opacity).
+
+| Param | Type | Default | Notes |
+|---|---|---|---|
+| `text` | string | — | Match element name/description |
+| `role` | string | — | e.g. `AXButton`, `AXTextField` |
+| `value` | string | — | Match current value |
+| `textMode` | `"contains"` \| `"exact"` \| `"regex"` | `"contains"` | |
+| `app` | string | active target | |
+| `depth` | number | 5 | |
+| `index` | number | — | Return only the Nth match (0-based) |
+| `near` | `{x,y}` | — | Sort by ascending distance, closest first |
+| `visibleOnly` | boolean | false | |
+| `includeBounds` | boolean | false | |
+
+### `click_element`
+Click by element `id`. AXPress first; on AX failure falls back to coordinate
+click at the element's bounds center (handles Tauri/Electron silent swallows).
+
+| Param | Type |
+|---|---|
+| `elementId` | string |
+| `app` | string (optional) |
+
+### `set_value`
+Set an AX element's value directly (no key synthesis). Best for text fields,
+checkboxes, sliders.
+
+| Param | Type |
+|---|---|
+| `elementId` | string |
+| `value` | string |
+| `app` | string (optional) |
+
+### `type_in_element`
+Focus an element and type into it. Refetches an equivalent AX node if the
+original `elementId` is stale (UI tree changed).
+
+| Param | Type | Default |
+|---|---|---|
+| `elementId` | string | |
+| `text` | string | |
+| `clearFirst` | boolean | false |
+| `app` | string | active target |
+
+### `click_menu_bar_extra`
+Click a menu-bar status item (tray icon) — for menu-bar-only apps (e.g.
+cc-switch) that `focus_app` cannot target. After clicking, the menu opens; use
+`find_element` to locate menu items, or `screenshot` + `ocr` if the menu's AX
+tree is opaque.
+
+| Param | Type | Notes |
+|---|---|---|
+| `app` | string | Target app name |
+| `description` | string | Match by description/name substring |
+| `name` | string | Match by name/description substring |
+| `index` | number | 0-based among matched items |
+
+---
+
+## Runtime & Synchronization
+
+### `list_apps`
+Returns `AppInfo[]` = `{name, pid, isFrontmost, windowCount}`. Background-only
+processes are filtered.
+
+### `focus_app`
+Set the active target context. Establishes a window for AX tools; falls back to
+a tray target (`windowId: "tray"`) for menu-bar-only apps if
+`click_menu_bar_extra` status items are found.
+
+| Param | Type |
+|---|---|
+| `app` | string |
+
+### `wait`
+Pause execution.
+
+| Param | Type |
+|---|---|
+| `ms` | number (1–60000) |
+
+### `wait_for_element`
+Poll until an AX element matches. Returns the match or times out.
+
+| Param | Type | Default |
+|---|---|---|
+| `text` / `role` / `value` | string | — |
+| `app` | string | active target |
+| `until` | `"appear"` \| `"disappear"` \| `"value_change"` | `"appear"` |
+| `timeout` / `timeoutMs` | number (ms) | 5000 |
+| `interval` / `intervalMs` | number (ms) | 500 |
+
+### `doctor`
+Verify permissions, native helpers, and client readiness. Returns a JSON report
+with `platform`, `safety`, `nativeHelpers`, `clients`, and `recommendations`.
+Run this first when something is misbehaving.
+
+### `clipboard_read` / `clipboard_write`
+Read/write the system clipboard. `clipboard_write` text-injection patterns
+(e.g. shell-escape sequences) are blocked by the safety guard.

package/skills/ucu-mcp/references/troubleshooting.md

@@ -0,0 +1,142 @@
+# Troubleshooting
+
+## First Checks
+
+1. Run `doctor` — verifies Accessibility + Screen Recording permissions and
+   native helpers. Most failures are permission issues.
+2. Confirm the target app is running and not minimized to the point of having no
+   on-screen window: `list_apps` + `list_windows`.
+3. Check `errors[]` in `describe_screen` responses — it names which source
+   (ocr/ax/foreground/screen) failed and why.
+
+---
+
+## Error Code Table
+
+Every error response carries a `code` and a `hint`. The table below maps codes
+to recovery steps (mirrors the runtime `recoveryHint`).
+
+| Code | Meaning | Recovery |
+|---|---|---|
+| `WINDOW_NOT_FOUND` | The target window does not exist or is not on screen. | `list_windows` again, retry with a fresh `windowId`, or omit `windowId` for screen coordinates. |
+| `TARGET_STALE` | The active target window changed pid or closed. | `focus_app` for the target app again, then retry. `type_in_element` auto-refetches equivalent nodes. |
+| `ELEMENT_NOT_FOUND` | No AX element matched the selector. | `find_element` again with broader selectors (different `text`, `textMode:"contains"`, drop `role`). If still empty, the app may be Electron-opaque — see below. |
+| `PERMISSION_DENIED` | Accessibility or Screen Recording not granted. | Run `doctor`, then grant the missing permission in System Settings → Privacy & Security, and **restart the launching client** (changes do not apply to already-running processes). |
+| `SAFETY_BLOCKED` | Action rejected by the safety guard (dangerous shortcut, sensitive window, suspicious text). | Choose a less risky action, or ask the user to perform it manually. Blocked shortcuts include `cmd+q`, `cmd+shift+q`, `cmd+l`, `alt+f4`. |
+| `INPUT_FAILED` | Input synthesis (click/type/keypress) failed at the CGEvent layer. | Observe current state with `screenshot` or `get_window_state`, then retry only if safe. |
+| `CAPTURE_FAILED` | Screenshot/OCR failed (usually Screen Recording permission). | `doctor` → grant Screen Recording → restart client. |
+| `COORDINATE_OUT_OF_BOUNDS` | Click/drag coordinates are outside the active display/window. | `get_screen_size` or `list_windows`, retry with coordinates inside bounds. |
+| `UNSUPPORTED_PARAMETER` | A parameter combination is invalid (e.g. `screenshot` with both `windowId` and `region`). | Remove or replace the unsupported parameter; inspect `tools/list` for the schema. |
+
+---
+
+## Permission Issues
+
+macOS requires two permissions for full functionality:
+
+- **Accessibility** — needed for all AX tools (`find_element`,
+  `click_element`, `get_window_state`, `list_windows`, `click_menu_bar_extra`).
+- **Screen Recording** — needed for `screenshot`, `ocr`, and `describe_screen`
+  (with `ocr: true`).
+
+Grant via **System Settings → Privacy & Security → Accessibility / Screen
+Recording**, enabling the entry for the launching terminal/client app (Terminal,
+iTerm, Claude Code, Codex, etc.).
+
+**Critical:** permission changes do not apply to already-running processes.
+After granting, **quit and restart the client** that launches `ucu-mcp`.
+
+Run `ucu-mcp doctor` (or the `doctor` tool) to verify — it reports per-permission
+status and which process to authorize.
+
+---
+
+## Electron / Tauri / WebView AX Opacity
+
+**Symptom:** `find_element` returns 0 results, `get_window_state` returns a near-
+empty tree (just an `AXGroup`), `list_windows` shows the window but AX tools
+can't see into it.
+
+**Cause:** Electron/Tauri/WebView apps render their UI in a composited layer
+that macOS AX cannot introspect. The AX tree exposes only the window frame and
+traffic-light buttons.
+
+**Workaround — pixel path:**
+```
+screenshot({})
+ocr({})
+  → blocks[].text locates the target UI text with bounding box {x,y,width,height}
+click({ x: block.x + block.width/2, y: block.y + block.height/2 })
+```
+
+`find_element` and `list_windows` emit a `hint` describing this fallback when
+they detect the pattern. For one-shot planning, `describe_screen` gives OCR + AX
+together.
+
+---
+
+## Menu-Bar / Tray App Not Reachable
+
+**Symptom:** `focus_app("tray-app")` throws `WINDOW_NOT_FOUND`; the app has no
+window in `list_windows`.
+
+**Cause:** Pure menu-bar (LSUIElement) apps have no window; their status item is
+hosted by the `SystemUIServer` system process.
+
+**Workaround:**
+```
+click_menu_bar_extra({ app: "tray-app", name: "TrayApp" })   # opens tray menu
+find_element({ text: "Settings", app: "tray-app" })          # menu items are AX-visible
+click_element({ elementId })
+```
+
+`focus_app` automatically falls back to a tray target when `click_menu_bar_extra`
+finds a matching status item, so subsequent AX tools work against the menu.
+
+If the tray menu itself is Electron-opaque:
+```
+click_menu_bar_extra({ app: "tray-app" })
+screenshot({})
+ocr({}) → locate menu item by text → click(x, y)
+```
+
+---
+
+## OCR Failures
+
+**Symptom:** `ocr` or `describe_screen` reports OCR failure (`ocr.status:
+"failed"`), or native OCR helper not found in `doctor`.
+
+**Checks:**
+1. Screen Recording permission granted and client restarted.
+2. Native helper present — `doctor` reports `ocr` helper status. If missing, the
+   npm package may be corrupted; reinstall.
+3. Screen is not locked (OCR captures a black frame when locked).
+
+`describe_screen` degrades gracefully — OCR failure still returns AX state, so
+you can fall back to `get_window_state` / `find_element`.
+
+---
+
+## describe_screen Returns Empty / All-skipped
+
+**Symptom:** `describe_screen` returns `errors: []` but `ocr.status:
+"skipped"` and `ax.status: "skipped"`.
+
+**Cause:** you passed `ocr: false, includeAx: false`, or the params defaulted to
+that (note: in live MCP use the SDK applies `ocr: true, includeAx: true`
+defaults; if you see both skipped, the client stripped defaults).
+
+**Fix:** explicitly pass `ocr: true, includeAx: true`.
+
+---
+
+## Actions Blocked While macOS Is Locked
+
+**Symptom:** input actions (`click`, `type_text`, `press_key`, …) fail; observe
+actions (`screenshot`, `ocr`) return black/empty frames.
+
+**Cause:** the safety guard refuses to synthesize input while the screen is
+locked (the user is not present to supervise).
+
+**Fix:** wait for the user to unlock, or ask them to unlock. There is no bypass.

package/skills/ucu-mcp/references/workflows.md

@@ -0,0 +1,146 @@
+# Workflows
+
+Common task playbooks. Each shows the preferred tool sequence and the fallback
+path when the primary path is blocked.
+
+---
+
+## 1. Fill a form field
+
+**Primary (AX):**
+```
+focus_app("Safari")
+find_element({ text: "Email", role: "AXTextField" })
+  → elementId "Safari/w0/42"
+type_in_element({ elementId: "Safari/w0/42", text: "user@example.com" })
+```
+
+**Fallback (AX value set, for non-text controls):**
+```
+set_value({ elementId: "...", value: "option" })
+```
+
+**Fallback (coordinates, when AX is opaque):**
+```
+screenshot({})
+ocr({})  → blocks[].text === "Email" → {x, y, width, height}
+click({ x: block.x + block.width/2, y: block.y + block.height/2 })
+type_text({ text: "user@example.com" })
+```
+
+---
+
+## 2. Operate a menu-bar / tray app (e.g. cc-switch)
+
+Tray apps' status items live in `SystemUIServer`, not the app's own window AX
+tree. `focus_app` alone may return `WINDOW_NOT_FOUND`.
+
+```
+focus_app("cc-switch")           # establishes tray target if status item found
+click_menu_bar_extra({ app: "cc-switch", name: "switch" })  # opens the menu
+# menu is now open — find items inside it:
+find_element({ text: "使用统计", app: "cc-switch" })
+  → elementId
+click_element({ elementId })
+```
+
+If the menu's AX tree is opaque (some Tauri/Electron menus):
+```
+click_menu_bar_extra({ app: "cc-switch" })
+screenshot({})
+ocr({})  → locate "使用统计" by text → coordinates
+click({ x, y })
+```
+
+---
+
+## 3. Electron / WebView opaque UI
+
+Electron/Tauri apps often expose only a near-empty `AXGroup`. The runtime `hint`
+on `find_element` and `list_windows` tells you when this is happening.
+
+```
+find_element({ text: "Submit" })
+  → 0 results, hint: "app is likely Electron... screenshot → ocr → click(x,y)"
+
+screenshot({})
+ocr({ region: { x, y, width, height } })  # or full screen
+  → blocks[].text === "Submit" → {x, y, width, height}
+click({ x: block.x + block.width/2, y: block.y + block.height/2 })
+```
+
+For repeated interaction with a known-opaque app, snapshot once with
+`describe_screen` to plan, then drive by coordinates.
+
+---
+
+## 4. Vision-degraded environment (image content not visible)
+
+When the model cannot see `screenshot` image blocks (relay/downgrade to URLs),
+switch to text-based screen reading:
+
+```
+describe_screen({ ocr: true, includeAx: true })
+  → { screen, foregroundWindow, ocr:{blocks}, ax:{elements}, errors }
+
+# or, if you also want the image for clients that DO support it:
+screenshot({ describe: true })
+  → [image block, text description block]
+```
+
+`describe_screen` never throws — OCR and AX each try/catch independently, so a
+Vision failure still returns AX state and vice versa. Check `errors[]` to know
+what was skipped/failed.
+
+---
+
+## 5. Recover from TARGET_STALE
+
+The active window target can go stale (window closed, app restarted, pid
+changed). AX tools throw `TARGET_STALE`.
+
+```
+# error response includes hint: "Run focus_app again for the target app..."
+focus_app("Safari")              # re-establishes target
+find_element({ text: "Save" })   # retry — cache refetches equivalent nodes
+click_element({ elementId })
+```
+
+`type_in_element` automatically refetches an equivalent AX node if the original
+`elementId` is stale, so a single retry often succeeds without `focus_app`.
+
+---
+
+## 6. Verify an action succeeded
+
+Always verify after clicks/types — UI may not have updated, or the wrong element
+was hit.
+
+```
+click_element({ elementId, captureAfter: true })   # screenshot in response
+# or explicitly:
+screenshot({})
+# or check AX state:
+get_window_state({})  → focusedElement / tree reflects the change
+# or wait for a specific change:
+wait_for_element({ text: "Saved", until: "appear", timeout: 3000 })
+```
+
+---
+
+## 7. Multi-step task with error recovery
+
+```
+doctor()                                  # verify permissions first
+list_apps()
+focus_app("Notes")
+find_element({ text: "New Note" }) → id
+click_element({ elementId: id, captureAfter: true })
+
+# if click_element throws ELEMENT_NOT_FOUND:
+find_element({ text: "New Note" }) → id2  # refetch, id may have changed
+click_element({ elementId: id2 })
+
+type_in_element({ elementId: bodyId, text: "Hello" })
+screenshot({})                            # confirm content
+```