@amaster.ai/pi-computer-use 0.1.1-beta.0 → 0.1.1

package/README.md

@@ -0,0 +1,136 @@
+# @amaster.ai/pi-computer-use
+
+![pi-computer-use preview](https://raw.githubusercontent.com/TGYD-helige/pi/master/packages/pi-computer-use/preview.png)
+
+pi-coding-agent extension that wraps [cua-driver-rs](https://github.com/trycua/cua/), exposing desktop automation tools with a `computer_use_` prefix.
+
+## Features
+
+- **Zero external dependencies** — pre-compiled cua-driver-rs binaries bundled for all platforms
+- **MCP stdio communication** — spawns `cua-driver mcp` via `StdioClientTransport`, JSON-RPC over stdio
+- **Dynamic tool discovery** — auto-discovers upstream MCP tools and registers with `computer_use_` prefix; falls back to a built-in tool list when cua-driver fails to start
+- **Smart tool filtering** — excludes non-essential tools (agent cursor, recording, config, raw screenshot), exposes 17 action tools + 1 vision tool
+- **Optional visual analysis** — `computer_use_analyze_screenshot` via configurable vision model
+- **Cross-platform permission handling** — detects platform-specific permission issues (macOS TCC, Windows UAC, Linux display server access) and returns actionable guidance
+- **Graceful degradation** — tools are always registered even when cua-driver cannot connect; lazy reconnect is attempted on each tool call
+
+## Install
+
+```bash
+bun add @amaster.ai/pi-computer-use
+```
+
+Requires Node.js >= 20 and `@earendil-works/pi-coding-agent >= 0.74.0`.
+
+## Usage
+
+Install the package and pi-coding-agent will automatically discover and load the extension. All tools are registered on `session_start`.
+
+Configure via `.pi/settings.json` (project-level) or `~/.pi/agent/settings.json` (user-level) under the `"pi-computer-use"` key:
+
+```json
+{
+  "pi-computer-use": {
+    "mode": "bundled"
+  }
+}
+```
+
+## Configuration
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `mode` | `'bundled' \| 'path'` | `'bundled'` | Binary resolution strategy |
+| `binaryPath` | `string` | — | Custom cua-driver binary path (requires `mode: 'path'`) |
+| `extraArgs` | `string[]` | — | Extra CLI arguments passed to cua-driver |
+| `visionModel` | `VisionModelConfig` | — | Enable visual screenshot analysis |
+
+### Vision Model (Optional)
+
+Enable `computer_use_analyze_screenshot` by referencing a model already configured in Pi's model registry (`models.json`):
+
+```json
+{
+  "pi-computer-use": {
+    "visionModel": {
+      "provider": "openai",
+      "model": "gpt-4o"
+    }
+  }
+}
+```
+
+The extension resolves API key, base URL, and headers from the model registry automatically — no need to duplicate credentials here.
+
+## Exposed Tools (17 + 1 vision)
+
+### Input
+
+| Tool | Description |
+|------|-------------|
+| `computer_use_click` | Left-click via element_index or x/y coordinates |
+| `computer_use_double_click` | Double-click at x/y or on an AX element |
+| `computer_use_right_click` | Right-click (context menu) |
+| `computer_use_type_text` | Insert text via AX or CGEvent fallback |
+| `computer_use_press_key` | Press and release a single key |
+| `computer_use_hotkey` | Press a key combination (e.g. Cmd+C) |
+| `computer_use_scroll` | Scroll by line or page in a direction |
+| `computer_use_drag` | Press-drag-release gesture between two points |
+| `computer_use_set_value` | Set value on UI elements (popups, sliders, steppers) |
+
+### Query
+
+| Tool | Description |
+|------|-------------|
+| `computer_use_get_screen_size` | Get display dimensions and scale factor |
+| `computer_use_get_cursor_position` | Get current mouse cursor position |
+| `computer_use_get_accessibility_tree` | Lightweight desktop snapshot (apps, windows, bounds) |
+| `computer_use_get_window_state` | Full AX tree of a window with actionable element indices |
+| `computer_use_list_windows` | List all top-level windows with bounds and z-order |
+| `computer_use_list_apps` | List running and installed apps with state flags |
+
+### App Lifecycle
+
+| Tool | Description |
+|------|-------------|
+| `computer_use_launch_app` | Launch an app in the background without focus steal |
+| `computer_use_kill_app` | Force-terminate a process by pid |
+
+### Vision (requires `visionModel` config)
+
+| Tool | Description |
+|------|-------------|
+| `computer_use_analyze_screenshot` | Take a screenshot and analyze it with a vision model |
+
+## Excluded Tools (16)
+
+Agent cursor styling, recording/replay, config management, zoom, raw screenshot (use `analyze_screenshot` instead), and browser-specific operations are filtered out.
+
+## Permissions
+
+On `session_start`, the extension checks permissions via cua-driver's `check_permissions` tool. Platform-specific guidance is provided:
+
+| Platform | Accessibility | Screen Capture |
+|----------|--------------|----------------|
+| macOS | System Settings → Privacy & Security → Accessibility | System Settings → Privacy & Security → Screen & System Audio Recording |
+| Windows | Run as Administrator / UI Automation access | Check DRM or security policy |
+| Linux | AT-SPI accessibility service | PipeWire portal or X11 access |
+
+When cua-driver fails to connect (missing permissions, binary not found, etc.):
+1. User is notified with a platform-appropriate warning
+2. Tools are still registered using a built-in fallback schema
+3. On each tool call, lazy reconnect is attempted; if it still fails, a friendly error with permission instructions is returned
+
+## Supported Platforms
+
+| Platform | Binary |
+|----------|--------|
+| macOS ARM64 | `bin/darwin-arm64/cua-driver` |
+| macOS x64 | `bin/darwin-x64/cua-driver` |
+| Linux x64 | `bin/linux-x64/cua-driver` |
+| Windows x64 | `bin/win32-x64/cua-driver.exe` |
+| Windows ARM64 | `bin/win32-arm64/cua-driver.exe` |
+
+## License
+
+Apache-2.0

package/bin/darwin-arm64/.version

@@ -0,0 +1,2 @@
+cua-driver-v0.2.0
+swift

package/bin/darwin-arm64/CuaDriver.app/Contents/Info.plist

@@ -0,0 +1,32 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+<plist version="1.0">
+<dict>
+    <key>CFBundleIdentifier</key>
+    <string>com.trycua.driver</string>
+    <key>CFBundleName</key>
+    <string>Cua Driver</string>
+    <key>CFBundleDisplayName</key>
+    <string>Cua Driver</string>
+    <key>CFBundleExecutable</key>
+    <string>cua-driver</string>
+    <key>CFBundleIconFile</key>
+    <string>AppIcon</string>
+    <key>CFBundleIconName</key>
+    <string>AppIcon</string>
+    <key>CFBundlePackageType</key>
+    <string>APPL</string>
+    <key>CFBundleShortVersionString</key>
+    <string>0.2.0</string>
+    <key>CFBundleVersion</key>
+    <string>1</string>
+    <key>LSMinimumSystemVersion</key>
+    <string>14.0</string>
+    <key>LSUIElement</key>
+    <true/>
+    <key>NSHighResolutionCapable</key>
+    <true/>
+    <key>NSSupportsAutomaticTermination</key>
+    <true/>
+</dict>
+</plist>

package/bin/darwin-arm64/CuaDriver.app/Contents/Resources/Skills/cua-driver/README.md

@@ -0,0 +1,140 @@
+# cua-driver — Claude Code skill
+
+A [Claude Code](https://code.claude.com) skill that teaches Claude to
+drive native macOS apps via the
+[`cua-driver`](https://github.com/trycua/cua/tree/main/libs/cua-driver)
+CLI — snapshot an app's accessibility tree, click/type/scroll by
+`element_index`, and verify via re-snapshot. Backgrounded-first: no
+focus steal, no cursor warp, no Space follow.
+
+## What the skill covers
+
+- The snapshot-before-AND-after invariant that keeps the agent honest
+  about whether an action actually landed.
+- The backgrounded-click recipe (yabai focus-without-raise + stamped
+  SLEventPostToPid) that lets synthetic clicks land on Chrome web
+  content without raising the window or pulling the user across Spaces.
+- Web-app quirks (`WEB_APPS.md`) — Chromium/WebKit/Electron/Tauri,
+  including the minimized-Chrome keyboard-commit caveat and the
+  `set_value` workaround.
+- Trajectory recording (`RECORDING.md`) — optional per-session
+  recording + replay for demos and regressions.
+- Canvas/viewport apps (Blender, Unity, GHOST, Qt, wxWidgets) —
+  HID-tap fallback when AX is empty.
+
+See `SKILL.md` for the main body.
+
+## Prerequisites
+
+1. **macOS 14 or newer** — the driver depends on SkyLight private SPIs
+   that were stabilized in Sonoma.
+2. **`cua-driver` CLI + `CuaDriver.app`** — installable one-liner:
+   ```bash
+   /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/trycua/cua/main/libs/cua-driver/scripts/install.sh)"
+   ```
+   Or from a clone of `trycua/cua`:
+   ```bash
+   cd libs/cua-driver
+   scripts/install-local.sh   # builds + installs + symlinks for dev use
+   ```
+   The driver runs as an `.app` bundle because macOS TCC grants are
+   tied to a stable bundle id (`com.trycua.driver`). The CLI symlink
+   lets Claude invoke tools via plain shell.
+3. **TCC grants on `CuaDriver.app`** — **Accessibility** and
+   **Screen Recording** in System Settings → Privacy & Security.
+   Verify with:
+   ```bash
+   cua-driver check_permissions
+   ```
+   Both fields must be `true`. If not, the app appears in the
+   relevant panes of System Settings after first use; toggle it on
+   there.
+
+## Install
+
+The skill is two drop-in directories.
+
+**Personal scope** (all Claude Code sessions on your machine):
+
+```bash
+mkdir -p ~/.claude/skills
+cp -R Skills/cua-driver ~/.claude/skills/
+```
+
+Or symlink if you want edits-in-place:
+
+```bash
+ln -s "$PWD/Skills/cua-driver" ~/.claude/skills/cua-driver
+```
+
+**Project scope** (committed alongside a specific repo):
+
+```bash
+mkdir -p .claude/skills
+cp -R /path/to/cua/libs/cua-driver/Skills/cua-driver .claude/skills/
+```
+
+## Invoking the skill
+
+Claude Code auto-invokes the skill when you ask for macOS GUI
+automation — e.g. "open the Downloads folder in Finder", "click the
+Save button in Numbers", "navigate to trycua.com in Chrome". You can
+also invoke it explicitly:
+
+```
+/cua-driver
+```
+
+## Claude Code MCP compatibility mode
+
+For normal skill-driven use, prefer the CLI or the standard MCP server. If you want Claude Code's vision/computer-use-style flow to ground on CuaDriver screenshots, register the compatibility server:
+
+```bash
+claude mcp add --transport stdio cua-computer-use -- cua-driver mcp --claude-code-computer-use-compat
+```
+
+This mode exposes the normal CuaDriver tools and changes only `screenshot`. The compatibility screenshot requires `pid` and `window_id`, captures that window only, and establishes a window-local pixel coordinate frame. It does not call Anthropic APIs or expose Anthropic's native computer-use API tool.
+
+Use MCP for this Claude Code vision/computer-use-style path. CLI screenshots still work as CuaDriver calls, but they do not expose the `mcp__cua-computer-use__screenshot` tool name that Claude Code appears to use as the image-grounding cue.
+
+## Files
+
+- `SKILL.md` — the main skill body (~500 lines). Loaded on first
+  invocation; stays in context for the session.
+- `WEB_APPS.md` — browsers, Electron, Tauri (Chromium + WebKit). Loaded
+  on demand when SKILL.md's pointer is followed.
+- `RECORDING.md` — trajectory recording / replay. Loaded on demand.
+- `TESTS.md` — manual test scripts for end-to-end skill verification.
+
+## Troubleshooting
+
+- `cua-driver: command not found` → re-run the installer or add
+  `.build/CuaDriver.app/Contents/MacOS/` to `$PATH`.
+- `No cached AX state for pid X window_id W` → element_index was
+  reused across turns, or across different windows of the same app.
+  Call `get_window_state({pid, window_id})` first in the same turn,
+  with the same window_id you're about to act against.
+- Empty `tree_markdown` → `capture_mode` is set to `vision`, which
+  skips the AX walk by design. Flip back to the default `som`
+  (`cua-driver config set capture_mode som`) to get the tree.
+  Tiny screenshot → likely a stale window capture. See "Behavior
+  matrix" in SKILL.md for the full mode table.
+- System-alert beep when pressing Return on a minimized Chrome
+  omnibox → the keyboard-commit-on-minimized limitation. Use
+  `set_value` on the field instead, or AX-click a Go/Submit button.
+  See `WEB_APPS.md`.
+
+## Updates
+
+The skill evolves alongside the driver. To update:
+
+```bash
+cd /path/to/cua && git pull
+# if you copied: re-copy
+cp -R libs/cua-driver/Skills/cua-driver ~/.claude/skills/
+# if you symlinked: nothing needed
+```
+
+## License
+
+MIT. Same license as the parent `trycua/cua` repo.

package/bin/darwin-arm64/CuaDriver.app/Contents/Resources/Skills/cua-driver/RECORDING.md

@@ -0,0 +1,113 @@
+# Recording & replaying trajectories
+
+Session-scoped capture of action sequences + pre/post state, suitable
+for demos, regression diffs, and training data. Invoked only when the
+user explicitly asks to record — the skill does not auto-enable this.
+
+`set_recording` turns on a session-scoped trajectory recorder. While
+enabled, every action-tool call (`click`, `right_click`, `scroll`,
+`type_text`, `press_key`, `hotkey`, `set_value`)
+writes a numbered turn folder under a caller-chosen output
+directory. Read-only tools (`get_window_state`, `list_windows`,
+`screenshot`, `list_apps`, permission probes, agent-cursor getters /
+setters, and `set_recording` itself) are not recorded.
+
+## Enable / disable
+
+Two equivalent surfaces: the `set_recording` MCP tool, or the
+friendlier `cua-driver recording` subcommand group (wraps
+`set_recording` + `get_recording_state` with human-readable output).
+
+```
+cua-driver recording start ~/cua-trajectories/run-1
+# … run the workflow …
+cua-driver recording status    # -> enabled / disabled, next_turn, output_dir
+cua-driver recording stop      # -> "Recording disabled (N turns captured in …)"
+```
+
+Raw-tool equivalent:
+
+```
+cua-driver set_recording '{"enabled":true,"output_dir":"~/cua-trajectories/run-1"}'
+cua-driver get_recording_state
+cua-driver set_recording '{"enabled":false}'
+```
+
+The `recording` subcommands require a running daemon (`cua-driver
+serve &`) because recording state is per-process. `output_dir` expands
+`~` and is created (with intermediates) if missing. Turn numbering
+starts at `1` every time recording is (re-)enabled, regardless of any
+existing contents in the directory. State lives in memory only — a
+daemon restart resets to disabled.
+
+## What each turn folder contains
+
+Each action writes to `turn-NNNNN/` (five-digit zero-padded counter):
+
+- `app_state.json` — post-action AX snapshot for the target pid, same
+  shape `get_window_state` returns (tree_markdown, element_count,
+  turn_id, etc.) minus the screenshot fields. The recorder resolves a
+  frontmost window internally (visible + on-current-Space preferred,
+  max-area fallback) since individual action tools carry a
+  window_id but the recorder has no caller-supplied anchor.
+- `screenshot.png` — post-action capture of the same window the
+  recorder just snapshotted. Omitted when the pid has no visible
+  window.
+- `action.json` — the tool name, full input arguments, result
+  summary, pid, click point (when applicable), ISO-8601 timestamp.
+- `click.png` — only for click-family actions (`click`,
+  `right_click`): a copy of `screenshot.png` with a red dot drawn at
+  the click point (screen-absolute point → window-local pixels via
+  the screenshot's `scale_factor`). Absent for other tools and for
+  clicks whose point falls outside the captured window.
+
+## When to use it
+
+- Demos and screen recordings — play the turn folder back to show
+  exactly what the agent saw and what it did.
+- Replay for regression — re-run the same sequence against a future
+  build and diff the new trajectory against the saved one.
+- Training data collection — each turn is a
+  `(state, action, next_state)` triple ready for offline learning.
+
+## When to invoke it
+
+This skill does **not** auto-enable recording. The client invokes
+`set_recording` explicitly when the user asks to capture a session.
+If the user says "record this session" or similar, call
+`set_recording({enabled:true, output_dir:…})` before the first
+action, and `set_recording({enabled:false})` when done.
+
+## Replaying a recorded trajectory
+
+`replay_trajectory({dir})` walks `<dir>/turn-NNNNN/` folders in
+lexical order, reads each `action.json`, and re-invokes the recorded
+tool with its recorded `arguments`. Optional knobs: `delay_ms`
+(pacing between turns, default 500) and `stop_on_error` (halt on
+first failure, default true).
+
+```
+cua-driver recording start ~/cua-trajectories/demo1
+# … run the workflow …
+cua-driver recording stop
+# Later: replay against a new build.
+cua-driver replay_trajectory '{"dir":"~/cua-trajectories/demo1","delay_ms":500}'
+```
+
+Important caveat: **element_index doesn't survive across sessions**.
+Indices are assigned fresh on every `get_window_state` snapshot,
+keyed on `(pid, window_id)`, so a recorded
+`click({pid, window_id, element_index: 14})` from yesterday won't
+resolve today — the pid is usually different, the window_id always
+is. The call returns `Invalid element_index` or `No cached AX
+state`. Pixel clicks (`click({pid, x, y})`) and keyboard tools
+(`press_key`, `hotkey`, `type_text` without element_index) replay cleanly; element-indexed actions require a
+live snapshot that replay doesn't currently re-emit (read-only tools
+like `get_window_state` aren't recorded). For a reliable replay, either
+compose the trajectory from pixel + keyboard primitives, or capture
+it as a regression artifact (compare the failure/success pattern
+across builds) rather than a re-driving script.
+
+If recording is still enabled while replay runs, the replay is
+itself recorded into the current output directory — that's the
+intended regression-diff workflow.