@agent-sh/computer-use-linux 0.4.10 → 0.6.0

package/skills/computer-use-linux/SKILL.md

@@ -1,14 +1,15 @@
 ---
 name: computer-use-linux
-description: "Linux desktop observation and control via the computer-use-linux MCP server: accessibility trees, screenshots, window targeting, and input synthesis (click, type, scroll). Works with any MCP host."
+description: "Linux desktop observation and control via native Pi tools or the computer-use-linux MCP server: accessibility trees, screenshots, window targeting, and input synthesis (click, type, scroll)."
 author: agent-sh
 license: MIT
 platforms: [linux]
+compatibility: "Native Pi tools require Pi 0.84.4+ and Node.js 22.19+; the standalone CLI/MCP server supports Node.js 18+."
 ---
 
 # computer-use-linux
 
-Use `computer-use-linux` when an agent needs to observe or operate a local Linux desktop through MCP: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions.
+Use `computer-use-linux` when an agent needs to observe or operate a local Linux desktop: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions.
 
 ## When to Use
 
@@ -23,6 +24,12 @@ Do not use this for remote browsers, websites, or headless automation when a bro
 
 ## Install
 
+Pi users need only the package:
+
+```bash
+pi install npm:@agent-sh/computer-use-linux
+```
+
 Preferred install:
 
 ```bash
@@ -49,6 +56,31 @@ If `doctor` selects ydotool as the input backend, also enable its per-user daemo
 
 On GNOME Wayland, log out and back in after `setup-window-targeting` if the GNOME Shell extension was newly installed.
 
+For MCP hosts with `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`, call the optional
+`complete_interaction` tool once after finishing desktop interaction. A skipped
+cue is not a task failure. This notification does not guarantee exclusive
+desktop ownership or that other clients have stopped sending input.
+This applies only to directly spawned MCP hosts, not the native Pi extension.
+
+`setup_accessibility` verifies the saved GNOME `toolkit-accessibility` key
+separately from runtime AT-SPI. Inspect its warning and readback before assuming
+new apps can expose trees. Other accessibility tools may change the key later;
+setup does not hold it enabled continuously.
+
+### Optional foreground accessibility guard
+
+Skip unless: the user explicitly wants GNOME's saved `toolkit-accessibility`
+setting kept enabled while desktop automation runs.
+
+Run `computer-use-linux guard-accessibility` in a foreground terminal. It
+registers a passive AT-SPI window-activation listener and watches/reasserts the
+saved key with readback. The setting affects all apps for the current user.
+`mcp`, setup, and `get_app_state` never start this guard automatically.
+Stop with Ctrl-C or SIGTERM before intentionally disabling accessibility.
+Stopping ends writes and removes its listener without disabling other clients
+or restoring a previous saved value. Apps launched during a reset/reassertion
+race may still need restarting; do not claim a complete GNOME toggle fix.
+
 ## Configure Your Agent
 
 The `computer-use-linux` binary is an MCP server. Configure it as a stdio MCP server in your agent of choice:
@@ -69,20 +101,43 @@ If the binary is not on `PATH`, use the absolute path (typically `~/.local/bin/c
 
 ## Procedure
 
-1. Start every desktop-control session with `doctor`.
-2. If `can_build_accessibility_tree` is false, run `setup` and restart the target app.
-3. If `can_query_windows` is false on GNOME Wayland, run `setup-window-targeting` and ask the user to log out and back in if setup says the shell extension needs a reload.
-4. Before targeted input, call `list_windows` or `focused_window` and verify the intended window by title, app id, pid, or wm class.
-5. Prefer semantic targeting from `get_app_state`: use element indices or role/name/text/states selectors.
-6. Use coordinates only when the UI surface has no useful accessibility tree.
-7. For text input, prefer `type_text` with a target selector (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `tty`, `terminal_pid`, `terminal_command`, or `terminal_cwd`) rather than relying on current focus.
-8. After mutating actions, re-check state with `get_app_state`, `focused_window`, or an app-specific readback.
+1. In Pi, call `computer_use_linux_tools` with the exact tools or capability you need. Enabled tools use the `computer_use_linux_<name>` prefix, appear starting on the next model turn, and remain active for the session.
+2. Begin every desktop-control turn with `get_app_state`; use `include_screenshot: false` when the accessibility tree is sufficient. Its compact readiness block identifies missing setup.
+3. Use `doctor` only when you need the full diagnostic report.
+4. If `can_build_accessibility_tree` is false, run `setup_accessibility` and restart the target app.
+5. If `can_query_windows` is false on GNOME Wayland, run `setup_window_targeting` and ask the user to log out and back in if setup says the shell extension needs a reload.
+6. Before targeted input, call `list_windows` or `focused_window` and verify the intended window by title, app id, pid, or wm class.
+7. Prefer semantic targeting from `get_app_state`: use element indices or role/name/text/states selectors.
+8. Use coordinates only when the UI surface has no useful accessibility tree.
+9. For text input, prefer `type_text` with a target selector (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `tty`, `terminal_pid`, `terminal_command`, or `terminal_cwd`) rather than relying on current focus.
+10. After mutating actions, re-check state with `get_app_state`, `focused_window`, or an app-specific readback.
+
+Plain left element/index/selector `click` prefers native AT-SPI `click`,
+`press`, or `toggle` over toolkit bounds, avoiding coordinate
+conversion when available. This preference does not replace a coordinate click
+with an arbitrary action name. Explicit `x`/`y`, right clicks, and double/multiple
+clicks retain pointer semantics.
+Use `perform_action` explicitly for entry `activate` or slider `jump`; `click`
+never substitutes those actions, including when bounds are unavailable.
+
+### Screenshot-relative coordinates
+
+Skip unless: a coordinate `click` or `scroll` uses `relative: true`.
+
+Select a target window and use its clipped screenshot crop origin. Divide
+preview `x`/`y` by screenshot `scale` first. Widget-local and raw GDK surface
+coordinates are not interchangeable with that origin; missing window targets
+are rejected. For calibration, use the repository's
+`examples/coordinate_probe.py`: select the green square from the screenshot
+and require a delivered-event `hit: true`. Do not pass widget-local `(85, 85)`
+directly to a window-relative click.
 
 ## Pitfalls
 
 - Already-running GTK, Qt, and Electron apps may need a restart after AT-SPI is enabled.
 - GNOME may show a portal prompt on the first screenshot or `get_app_state` call with screenshots enabled.
 - Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
+- Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.
 - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.
 - When ydotool is selected, `ydotoold` should run as a per-user service with its socket under `/run/user/$UID`, not as a system-wide service.
 - The optional ydotool backend requires version 1.0.3 or newer; `doctor` rejects older or semantically incompatible CLIs even when `ydotoold` is running.

package/skills/computer-use-linux/references/pi-setup.md

@@ -1,57 +1,128 @@
 ---
 name: pi-setup
-description: "Pi coding agent setup for the computer-use-linux MCP server."
+description: "Pi coding agent setup for native computer-use-linux tools."
 ---
 
 # Pi Setup
 
-## Prerequisites
+## Install
 
-You need both of these installed:
+Install one package:
 
 ```bash
-pi install npm:pi-mcp-adapter
 pi install npm:@agent-sh/computer-use-linux
 ```
 
-> **Note:** `pi-mcp-adapter` is a separate extension that provides MCP protocol support in pi. The `@agent-sh/computer-use-linux` package auto-registers the desktop MCP server into pi-mcp-adapter's config.
+Restart Pi or run `/reload`. No separate MCP adapter or MCP configuration is
+required.
 
-After installing both, restart pi or run `/reload`.
+The integration is designed for current Pi releases with additive dynamic tool
+loading. Update Pi and installed packages when needed:
 
-## Verify
+```bash
+pi update --all
+```
 
-The `mcp()` proxy tool should now be available. Call it from any prompt:
+## How tool loading works
 
-```bash
-mcp({ search: "doctor" })
+Pi initially sees one small loader:
+
+```text
+computer_use_linux_tools
 ```
 
-Or start with the readiness check (note the server-prefixed tool name):
+Enable exact tools:
 
-```bash
-mcp({ tool: "computer_use_linux_doctor" })
+```text
+computer_use_linux_tools({ tools: ["doctor", "list_windows"] })
 ```
 
-Search for available tools:
+Or search by capability:
 
-```bash
-mcp({ server: "computer-use-linux" })
+```text
+computer_use_linux_tools({ query: "inspect a window and type text" })
+```
+
+The selected native tools appear starting on the next model turn with their
+full upstream schemas and remain active for the session:
+
+```text
+computer_use_linux_doctor({})
+computer_use_linux_list_windows({})
+computer_use_linux_type_text({ text: "hello", title: "Notes" })
 ```
 
+Pi does not start the desktop process when the loader runs. The first real tool
+call starts one computer-use-linux process, and the package reuses it for the
+session so `get_app_state` element indices and portal sessions remain valid.
+
+## Safe operating loop
+
+1. Enable `get_app_state`, `list_windows`, and any likely action tools.
+2. Call `computer_use_linux_get_app_state`, using
+   `include_screenshot: false` when accessibility data is enough.
+3. Inspect the returned readiness block; enable/call `doctor` only for full
+   diagnostics.
+4. Identify the target with `computer_use_linux_list_windows` or
+   `computer_use_linux_focused_window`.
+5. Enable and call the required action tool.
+6. Re-observe after the UI changes.
+
+Native Computer Use tools execute sequentially. Cancellation is forwarded to
+the MCP request. If the server process exits, the failed call is never replayed
+automatically; call `get_app_state` again before another element-based action.
+
+## Migration from the adapter-based package
+
+Older releases required `pi-mcp-adapter` and wrote a
+`computer-use-linux` entry into `mcp.json` under the configured Pi agent
+directory.
+
+The native integration does not write MCP configuration. It reads only the
+legacy entry location to show a migration notice. After updating:
+
+1. Remove only the `computer-use-linux` entry from the Pi agent `mcp.json` if
+   it is still present.
+2. Keep `pi-mcp-adapter` if you use it for other MCP servers; otherwise remove
+   it with `pi remove npm:pi-mcp-adapter`.
+3. Run `/reload`.
+
+The package reports a non-destructive migration notice when it detects the
+legacy entry.
+
 ## If the binary is not found
 
 The extension looks for `computer-use-linux` in this order:
 
-1. `COMPUTER_USE_LINUX_BIN` environment variable
-2. The npm-bundled binary (from `npm install -g @agent-sh/computer-use-linux`)
-3. `$PATH`
+1. `COMPUTER_USE_LINUX_BIN`
+2. The binary downloaded inside the installed npm package
+
+Reinstall the package if the bundled binary is missing:
+
+```bash
+pi remove npm:@agent-sh/computer-use-linux
+pi install npm:@agent-sh/computer-use-linux
+```
 
-If none are found, install it:
+Users with a separate build can set:
 
 ```bash
-npm install -g @agent-sh/computer-use-linux
-# or
-cargo install computer-use-linux
+export COMPUTER_USE_LINUX_BIN=/absolute/path/to/computer-use-linux
 ```
 
-Then restart pi or run `/reload`.
+## Verification
+
+Enable and call the readiness tool:
+
+```text
+computer_use_linux_tools({ tools: ["doctor"] })
+computer_use_linux_doctor({})
+```
+
+Ready output has:
+
+- `can_register_mcp_tools: true`
+- `can_build_accessibility_tree: true`
+- `can_query_windows: true`
+- `can_send_development_input: true`
+- `blockers: []`