@amaster.ai/pi-computer-use 0.1.2-beta.8 → 0.1.2-beta.80

package/README.md

@@ -1,16 +1,22 @@
 # @amaster.ai/pi-computer-use
 
-pi-coding-agent extension that wraps [cua-driver-rs](https://github.com/trycua/cua/), exposing desktop automation tools with a `computer_use_` prefix.
+![pi-computer-use preview](https://raw.githubusercontent.com/TGYD-helige/pi/master/packages/pi-computer-use/preview.png)
 
-## Features
+Cross-platform computer-use tools for Pi desktop automation. The extension exposes a native MCP tool surface with a `computer_use_` prefix. The bundled runtime comes from the official [Cua Driver Rust 0.28.2 release](https://github.com/trycua/cua/releases/tag/cua-driver-rs-v0.28.2).
 
-- **Zero external dependencies** — pre-compiled cua-driver-rs binaries bundled for all platforms
-- **MCP stdio communication** — spawns `cua-driver mcp` via `StdioClientTransport`, JSON-RPC over stdio
-- **Dynamic tool discovery** — auto-discovers upstream MCP tools and registers with `computer_use_` prefix; falls back to a built-in tool list when cua-driver fails to start
-- **Smart tool filtering** — excludes non-essential tools (agent cursor, recording, config, raw screenshot), exposes 17 action tools + 1 vision tool
-- **Optional visual analysis** — `computer_use_analyze_screenshot` via configurable vision model
-- **Cross-platform permission handling** — detects platform-specific permission issues (macOS TCC, Windows UAC, Linux display server access) and returns actionable guidance
-- **Graceful degradation** — tools are always registered even when cua-driver cannot connect; lazy reconnect is attempted on each tool call
+## What it provides
+
+- One Rust 0.28.2 driver line across macOS, Linux, and Windows
+- 56 version-pinned upstream tools, including sessions, element tokens, accessibility + screenshot state, native input, browser tools, diagnostics, recording, and permission policy support
+- Deferred tool activation: only the core toolset is active by default; extra groups are activated on demand via `computer_use_tools` or `/computer-use-tools`
+- Full MCP text, image, and `structuredContent` forwarding
+- Owned daemon + MCP proxy lifecycle with session-owned reconnect and per-call cancellation
+- A non-prompting Linux/Windows permission probe on session start
+- Once-per-session app-launch approval and confirmation for high-risk operations
+- Bounded text and structured results before they enter Pi's context
+- Optional secondary vision analysis through a configured Pi model
+
+`get_window_state` is the primary perception tool. Cua Driver 0.28 returns the accessibility tree, structured elements with `element_token`, and a screenshot in one response. The standalone `screenshot` tool no longer exists.
 
 ## Install
 
@@ -18,34 +24,56 @@ pi-coding-agent extension that wraps [cua-driver-rs](https://github.com/trycua/c
 bun add @amaster.ai/pi-computer-use
 ```
 
-Requires Node.js >= 20 and `@earendil-works/pi-coding-agent >= 0.74.0`.
+The package selects one signed/precompiled platform runtime through npm `optionalDependencies`. Installation does not download from GitHub, and no separate Cua Driver installation is required. Offline mirrors must include `@amaster.ai/pi-computer-use` and the matching `@amaster.ai/pi-computer-use-cua-driver-*` package.
 
-## Usage
+Upgrading from Cua Driver 0.9 is a deliberate public-tool migration: upstream removed `set_agent_cursor_style` and replaced it with `set_agent_cursor_theme`. The new tool selects an installed `theme_id`; the old custom color/image parameters have no 0.28 equivalent.
 
-Install the package and pi-coding-agent will automatically discover and load the extension. All tools are registered on `session_start`.
+## Configuration
 
-Configure via `.pi/settings.json` (project-level) or `~/.pi/agent/settings.json` (user-level) under the `"pi-computer-use"` key:
+Configure `.pi/settings.json` or `~/.pi/agent/settings.json`:
+
+Project settings are loaded only after project trust is accepted. `${ENV_VAR}` interpolation is supported in user and agent settings, but not in project settings.
 
 ```json
 {
   "pi-computer-use": {
-    "mode": "bundled"
+    "mode": "bundled",
+    "confirmAppLaunch": true,
+    "confirmDangerousActions": true,
+    "toolProfile": "core"
   }
 }
 ```
 
-## Configuration
-
 | Option | Type | Default | Description |
-|--------|------|---------|-------------|
-| `mode` | `'bundled' \| 'path'` | `'bundled'` | Binary resolution strategy |
-| `binaryPath` | `string` | — | Custom cua-driver binary path (requires `mode: 'path'`) |
-| `extraArgs` | `string[]` | — | Extra CLI arguments passed to cua-driver |
-| `visionModel` | `VisionModelConfig` | — | Enable visual screenshot analysis |
+| --- | --- | --- | --- |
+| `mode` | `"bundled" \| "path"` | `"bundled"` | Use the packaged 0.28.2 driver or a compatible custom binary |
+| `binaryPath` | `string` | — | Custom driver path when `mode` is `"path"` |
+| `extraArgs` | `string[]` | — | Additional arguments appended to `cua-driver mcp` |
+| `confirmAppLaunch` | `boolean` | `true` | Ask once per app target before `launch_app` |
+| `confirmDangerousActions` | `boolean` | `true` | Confirm high-risk tools such as `kill_app` and `replay_trajectory`; recording always requires confirmation |
+| `toolProfile` | `"core" \| "full"` | `"core"` | `"core"` activates only the everyday toolset at session start; `"full"` activates all 56 driver tools |
+| `visionModel` | `{ provider, model }` | — | Register `computer_use_analyze_screenshot` |
+
+In non-interactive modes, confirmation-required tools return an error unless the corresponding confirmation setting is explicitly disabled.
+
+### Tool groups
+
+With the default `core` profile, only everyday tools are visible to the model (`list_apps`, `launch_app`, `get_window_state`, `verify_state`, the click/type/scroll family, and `zoom`). The remaining driver tools stay registered but inactive, so they cost no prompt context. The model activates a group itself by calling `computer_use_tools` (without arguments it lists the groups); users can do the same with `/computer-use-tools <group>`:
 
-### Vision Model (Optional)
+| Group | Tools |
+| --- | --- |
+| `browser` | CDP browser automation: `browser_prepare`, `browser_navigate`, `browser_click`, `browser_type`, `browser_dialog`, `browser_set_input_files`, `browser_download`, `browser_pointer`, `get_browser_state`, `page` |
+| `recording` | `start_recording`, `stop_recording`, `get_recording_state`, `replay_trajectory`, `install_ffmpeg` |
+| `session` | `start_session`, `escalate_session`, `get_session`, `get_session_state`, `list_sessions`, `end_session` |
+| `cursor` | `move_cursor`, `set_agent_cursor_enabled`, `set_agent_cursor_motion`, `set_agent_cursor_theme`, `get_agent_cursor_state` |
+| `window` | `list_windows`, `bring_to_front`, `set_window_frame`, `invoke_menu`, `kill_app` |
+| `clipboard` | `clipboard_read`, `clipboard_write` |
+| `diagnostics` | `check_permissions`, `health_report`, `get_config`, `set_config`, `check_for_update`, `get_desktop_state`, `get_screen_size`, `get_cursor_position`, `get_accessibility_tree` |
 
-Enable `computer_use_analyze_screenshot` by referencing a model already configured in Pi's model registry (`models.json`):
+Activated groups stay active for the rest of the session; each new session starts from the configured profile again.
+
+### Optional vision model
 
 ```json
 {
@@ -58,77 +86,44 @@ Enable `computer_use_analyze_screenshot` by referencing a model already configur
 }
 ```
 
-The extension resolves API key, base URL, and headers from the model registry automatically — no need to duplicate credentials here.
-
-## Exposed Tools (17 + 1 vision)
-
-### Input
-
-| Tool | Description |
-|------|-------------|
-| `computer_use_click` | Left-click via element_index or x/y coordinates |
-| `computer_use_double_click` | Double-click at x/y or on an AX element |
-| `computer_use_right_click` | Right-click (context menu) |
-| `computer_use_type_text` | Insert text via AX or CGEvent fallback |
-| `computer_use_press_key` | Press and release a single key |
-| `computer_use_hotkey` | Press a key combination (e.g. Cmd+C) |
-| `computer_use_scroll` | Scroll by line or page in a direction |
-| `computer_use_drag` | Press-drag-release gesture between two points |
-| `computer_use_set_value` | Set value on UI elements (popups, sliders, steppers) |
-
-### Query
-
-| Tool | Description |
-|------|-------------|
-| `computer_use_get_screen_size` | Get display dimensions and scale factor |
-| `computer_use_get_cursor_position` | Get current mouse cursor position |
-| `computer_use_get_accessibility_tree` | Lightweight desktop snapshot (apps, windows, bounds) |
-| `computer_use_get_window_state` | Full AX tree of a window with actionable element indices |
-| `computer_use_list_windows` | List all top-level windows with bounds and z-order |
-| `computer_use_list_apps` | List running and installed apps with state flags |
-
-### App Lifecycle
-
-| Tool | Description |
-|------|-------------|
-| `computer_use_launch_app` | Launch an app in the background without focus steal |
-| `computer_use_kill_app` | Force-terminate a process by pid |
-
-### Vision (requires `visionModel` config)
+`computer_use_analyze_screenshot` requires both `pid` and `window_id`. It calls `get_window_state`, reuses the returned image, and invokes the configured model. Use it only when the primary model cannot resolve visual ambiguity.
 
-| Tool | Description |
-|------|-------------|
-| `computer_use_analyze_screenshot` | Take a screenshot and analyze it with a vision model |
+## Runtime and permissions
 
-## Excluded Tools (16)
+On macOS, `session_start` registers the generated 0.28.2 manifest without starting the driver. The signed app and MCP proxy start lazily on the first computer-use tool call, which requests any missing permissions through `check_permissions({ prompt: true })`. Existing grants do not raise another system dialog. The requested tool still runs and reports its own capability or permission error. Linux and Windows keep eager startup: they discover the exact live `tools/list` surface and call `check_permissions({ prompt: false })`. If discovery fails, the extension registers `computer_use_connect` (and `/computer-use-connect`) so a later retry can install the exact live platform contract without advertising another OS's schemas.
 
-Agent cursor styling, recording/replay, config management, zoom, raw screenshot (use `analyze_screenshot` instead), and browser-specific operations are filtered out.
+Driver startup, reconnect, and the first macOS permission probe are session-owned. Cancelling a tool stops only that caller's wait or MCP request; session shutdown aborts the shared work. macOS and Linux also use a transient pipe-backed lease to stop the owned daemon after an abrupt host exit; no persistent service or system scheduler is installed.
 
-## Permissions
+- **Bundled macOS:** launches the signed `CuaDriver.app` through LaunchServices,
+  so Accessibility and Screen Recording grants belong to `com.trycua.driver`.
+- **Custom macOS binary:** uses Cua Driver embedded mode and inherits the host
+  application's TCC responsibility chain.
+- **Linux/Windows:** starts an extension-owned daemon in the interactive user
+  session and tears it down on session shutdown.
 
-On `session_start`, the extension checks permissions via cua-driver's `check_permissions` tool. Platform-specific guidance is provided:
+## Supported targets
 
-| Platform | Accessibility | Screen Capture |
-|----------|--------------|----------------|
-| macOS | System Settings → Privacy & Security → Accessibility | System Settings → Privacy & Security → Screen & System Audio Recording |
-| Windows | Run as Administrator / UI Automation access | Check DRM or security policy |
-| Linux | AT-SPI accessibility service | PipeWire portal or X11 access |
+| Platform | Runtime package |
+| --- | --- |
+| macOS ARM64 / x64 | `@amaster.ai/pi-computer-use-cua-driver-darwin-universal` |
+| Linux x64 | `@amaster.ai/pi-computer-use-cua-driver-linux-x64` |
+| Linux ARM64 | `@amaster.ai/pi-computer-use-cua-driver-linux-arm64` |
+| Windows x64 | `@amaster.ai/pi-computer-use-cua-driver-win32-x64` |
+| Windows ARM64 | `@amaster.ai/pi-computer-use-cua-driver-win32-arm64` |
 
-When cua-driver fails to connect (missing permissions, binary not found, etc.):
-1. User is notified with a platform-appropriate warning
-2. Tools are still registered using a built-in fallback schema
-3. On each tool call, lazy reconnect is attempted; if it still fails, a friendly error with permission instructions is returned
+## Canonical workflow
 
-## Supported Platforms
+1. `computer_use_launch_app` or `computer_use_list_apps`
+2. `computer_use_get_window_state`
+3. Act using `element_token`/`element_index`, falling back to pixels for
+   custom-drawn surfaces
+4. Re-run `computer_use_get_window_state` and verify the change
+5. When the task needs more than the core toolset (browser automation,
+   recording, escalated sessions, ...), call `computer_use_tools` with the
+   matching group first
 
-| Platform | Binary |
-|----------|--------|
-| macOS ARM64 | `bin/darwin-arm64/cua-driver` |
-| macOS x64 | `bin/darwin-x64/cua-driver` |
-| Linux x64 | `bin/linux-x64/cua-driver` |
-| Windows x64 | `bin/win32-x64/cua-driver.exe` |
-| Windows ARM64 | `bin/win32-arm64/cua-driver.exe` |
+Linux and Windows tool descriptions and schemas come from the exact live driver. macOS uses the generated manifest for the bundled driver release.
 
 ## License
 
-Apache-2.0
+Apache-2.0 for this extension package. Bundled Cua Driver assets retain the upstream MIT license and release metadata in each platform package.

package/dist/config.d.ts

@@ -12,6 +12,16 @@ export interface ComputerUseConfig {
     extraArgs?: string[];
     /** Vision model for screenshot analysis */
     visionModel?: VisionModelConfig;
+    /** Ask once per app target before launch_app. Default: true */
+    confirmAppLaunch?: boolean;
+    /** Confirm high-risk tools such as kill_app and replay_trajectory. Default: true */
+    confirmDangerousActions?: boolean;
+    /**
+     * Which driver tools are active at session start: 'core' activates only the
+     * everyday toolset (extra groups via the computer_use_tools tool or the
+     * /computer-use-tools command), 'full' activates every tool. Default: 'core'
+     */
+    toolProfile?: 'core' | 'full';
 }
 export declare function resolveConfig(config?: ComputerUseConfig): ComputerUseConfig;
 export declare function loadConfigFromFile(options?: PiSettingsOptions): ComputerUseConfig;

package/dist/config.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAkB,KAAK,iBAAiB,EAAE,MAAM,gCAAgC,CAAC;AAGxF,MAAM,WAAW,iBAAiB;IAChC,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;CACf;AAED,MAAM,WAAW,iBAAiB;IAChC,2EAA2E;IAC3E,IAAI,CAAC,EAAE,SAAS,GAAG,MAAM,CAAC;IAC1B,+DAA+D;IAC/D,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,8CAA8C;IAC9C,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;IACrB,2CAA2C;IAC3C,WAAW,CAAC,EAAE,iBAAiB,CAAC;CACjC;AAMD,wBAAgB,aAAa,CAAC,MAAM,CAAC,EAAE,iBAAiB,GAAG,iBAAiB,CAE3E;AAED,wBAAgB,kBAAkB,CAAC,OAAO,CAAC,EAAE,iBAAiB,GAAG,iBAAiB,CAKjF"}
+{"version":3,"file":"config.d.ts","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAkB,KAAK,iBAAiB,EAAE,MAAM,gCAAgC,CAAC;AAExF,MAAM,WAAW,iBAAiB;IAChC,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,MAAM,CAAC;CACf;AAED,MAAM,WAAW,iBAAiB;IAChC,2EAA2E;IAC3E,IAAI,CAAC,EAAE,SAAS,GAAG,MAAM,CAAC;IAC1B,+DAA+D;IAC/D,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,8CAA8C;IAC9C,SAAS,CAAC,EAAE,MAAM,EAAE,CAAC;IACrB,2CAA2C;IAC3C,WAAW,CAAC,EAAE,iBAAiB,CAAC;IAChC,+DAA+D;IAC/D,gBAAgB,CAAC,EAAE,OAAO,CAAC;IAC3B,oFAAoF;IACpF,uBAAuB,CAAC,EAAE,OAAO,CAAC;IAClC;;;;OAIG;IACH,WAAW,CAAC,EAAE,MAAM,GAAG,MAAM,CAAC;CAC/B;AASD,wBAAgB,aAAa,CAAC,MAAM,CAAC,EAAE,iBAAiB,GAAG,iBAAiB,CAE3E;AAED,wBAAgB,kBAAkB,CAAC,OAAO,CAAC,EAAE,iBAAiB,GAAG,iBAAiB,CAIjF"}

package/dist/config.js

@@ -1,14 +1,15 @@
 import { loadPiSettings } from '@amaster.ai/pi-shared/settings';
-import { getAgentDir } from '@earendil-works/pi-coding-agent';
 const DEFAULTS = {
     mode: 'bundled',
+    confirmAppLaunch: true,
+    confirmDangerousActions: true,
+    toolProfile: 'core',
 };
 export function resolveConfig(config) {
     return { ...DEFAULTS, ...config };
 }
 export function loadConfigFromFile(options) {
     return loadPiSettings('pi-computer-use', {
-        agentDir: getAgentDir(),
         ...options,
     });
 }

package/dist/config.js.map

@@ -1 +1 @@
-{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAA0B,MAAM,gCAAgC,CAAC;AACxF,OAAO,EAAE,WAAW,EAAE,MAAM,iCAAiC,CAAC;AAkB9D,MAAM,QAAQ,GAA+B;IAC3C,IAAI,EAAE,SAAS;CAChB,CAAC;AAEF,MAAM,UAAU,aAAa,CAAC,MAA0B;IACtD,OAAO,EAAE,GAAG,QAAQ,EAAE,GAAG,MAAM,EAAE,CAAC;AACpC,CAAC;AAED,MAAM,UAAU,kBAAkB,CAAC,OAA2B;IAC5D,OAAO,cAAc,CAAoB,iBAAiB,EAAE;QAC1D,QAAQ,EAAE,WAAW,EAAE;QACvB,GAAG,OAAO;KACX,CAAC,CAAC;AACL,CAAC"}
+{"version":3,"file":"config.js","sourceRoot":"","sources":["../src/config.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,cAAc,EAA0B,MAAM,gCAAgC,CAAC;AA4BxF,MAAM,QAAQ,GAA+B;IAC3C,IAAI,EAAE,SAAS;IACf,gBAAgB,EAAE,IAAI;IACtB,uBAAuB,EAAE,IAAI;IAC7B,WAAW,EAAE,MAAM;CACpB,CAAC;AAEF,MAAM,UAAU,aAAa,CAAC,MAA0B;IACtD,OAAO,EAAE,GAAG,QAAQ,EAAE,GAAG,MAAM,EAAE,CAAC;AACpC,CAAC;AAED,MAAM,UAAU,kBAAkB,CAAC,OAA2B;IAC5D,OAAO,cAAc,CAAoB,iBAAiB,EAAE;QAC1D,GAAG,OAAO;KACX,CAAC,CAAC;AACL,CAAC"}