@agent-sh/computer-use-linux 0.4.10 → 0.6.0

package/README.md

@@ -61,6 +61,23 @@ Screenshot payloads are size-bounded by default before they are returned to the
 - `press_key` — keys / chords; can focus a window or terminal first
 - `type_text` — literal text input, optionally targeted at a window or terminal
 
+For a plain left `click` by element index or selector, a recognized native
+AT-SPI `click`, `press`, or `toggle` action takes precedence
+over bounds. Entry `activate` and slider `jump` must be requested explicitly
+with `perform_action`; they are never a substitute for a pointer click, even
+when bounds are missing.
+This avoids pointer conversion for GTK3 HiDPI
+extents and GTK4 zero-origin bounds when the element exposes such an action.
+The preference does not substitute an arbitrary action name for a coordinate
+click. Explicit `x`/`y`, right clicks, and double/multiple clicks retain pointer
+semantics. Re-check application state after either kind of activation.
+
+For coordinate `click` or `scroll` with `relative: true`, select a target window
+and measure from its clipped screenshot crop origin. Divide preview `x` and
+`y` by the returned screenshot `scale` before passing them. These are not raw
+GDK surface or widget-local coordinates; decorations and clipping can change
+the origin. A missing window target is rejected.
+
 Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report.
 
 **Semantic actions**
@@ -75,6 +92,8 @@ Targeted `press_key`/`type_text` results append focused-element feedback from AT
 
 **Conditional host execution**
 
+- `complete_interaction` - optional desktop completion notification, registered only with `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`. Repeated calls can create repeated notifications; it does not provide desktop exclusivity.
+
 - `run_shell` — same-user `/bin/sh -c` execution without login-profile loading, registered only when the server operator starts the MCP process with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. It is deliberately absent by default and is not a sandbox.
 
 ### MCP safety contract
@@ -99,6 +118,7 @@ The binary also exposes the same capabilities from the CLI for scripting and deb
 computer-use-linux mcp                                  # stdio MCP server
 computer-use-linux doctor                               # JSON readiness report
 computer-use-linux setup                                # enable AT-SPI
+computer-use-linux guard-accessibility                  # explicit foreground GNOME accessibility guard
 computer-use-linux setup-window-targeting               # install GNOME Shell extension
 computer-use-linux apps
 computer-use-linux state [APP_NAME]
@@ -243,20 +263,34 @@ Restart Claude Desktop. The tools should appear in the tools list.
 ### Pi Coding Agent
 
 ```bash
-pi install npm:pi-mcp-adapter
 pi install npm:@agent-sh/computer-use-linux
 ```
 
-Restart pi or run `/reload`. The MCP proxy tool `mcp()` will have the desktop tools available:
+Restart Pi or run `/reload`. The package exposes one small loader initially;
+the real tools keep their upstream schemas and are enabled only when Computer
+Use is needed:
+
+Native tools require Pi 0.84.4 or newer (Node.js 22.19 or newer). The
+standalone npm CLI wrapper continues to support Node.js 18 or newer.
+
+```
+computer_use_linux_tools({ tools: ["doctor", "list_windows"] })
+computer_use_linux_doctor({})
+computer_use_linux_list_windows({})
+```
+
+You can also search by capability:
 
 ```
-mcp({ server: "computer-use-linux" })             # list all tools
-mcp({ search: "windows" })                         # search for window tools
-mcp({ tool: "computer_use_linux_doctor" })         # run readiness check
-mcp({ tool: "computer_use_linux_list_windows" })   # list desktop windows
+computer_use_linux_tools({ query: "observe a window and click a control" })
 ```
 
-The extension auto-registers the computer-use-linux MCP server into pi-mcp-adapter's config. If the binary is not found, check the [Pi setup guide](skills/computer-use-linux/references/pi-setup.md).
+No separate MCP adapter or manual MCP configuration is required. Pi starts one
+computer-use-linux process lazily on the first real tool call, reuses it for the
+session so accessibility snapshots remain valid, serializes desktop actions,
+and closes it on reload, session switch, or exit. See the
+[Pi setup guide](skills/computer-use-linux/references/pi-setup.md) for migration
+from older adapter-based installs.
 
 ### Hermes Agent
 
@@ -344,6 +378,7 @@ Most setups need none of these — `doctor` and the installers pick sensible def
 
 | Variable | Effect |
 | --- | --- |
+| `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE` | Set exactly to `1` to expose the optional `complete_interaction` notification tool. Requires `notify-send` and a desktop notification service; disabled by default. |
 | `COMPUTER_USE_LINUX_COSMIC_HELPER` | Path to the `computer-use-linux-cosmic` helper when it isn't next to the binary or on `PATH`. |
 | `CU_DISABLE_ABS_POINTER` | Disable the uinput absolute pointer and click through `ydotool` instead for setups where the abs-pointer device misbehaves. |
 | `COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through the RemoteDesktop portal on Wayland, skipping auto-detection. |
@@ -394,6 +429,44 @@ If you're running this on a shared workstation, set `ydotoold`'s socket permissi
 
 ## Troubleshooting
 
+To receive an explicit completion cue, start the MCP server with
+`COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`. This exposes `complete_interaction`,
+a parameter-free tool the agent calls once after finishing its desktop work.
+It submits a notification through `notify-send` with a two-second execution
+limit and bounded process cleanup. Missing services, errors, or timeouts return
+`cue: "skipped"`; notification settings may suppress a submitted cue. This does
+not reserve the desktop or prove that other clients have stopped sending input.
+No sound or additional desktop settings are enabled by this option.
+This option currently applies only to directly spawned MCP hosts. The native Pi
+extension does not forward the flag or include this optional tool in its catalog.
+
+`setup` and `setup_accessibility` write and read back GNOME's
+`org.gnome.desktop.interface toolkit-accessibility` setting, even if AT-SPI
+is already enabled at runtime. A runtime-only success is reported as a warning:
+newly launched GTK apps may still have no tree. Enabling the saved key may
+require restarting target apps. Other accessibility tools can change that key
+later; setup does not continuously override user settings.
+
+For an explicit foreground guard while using desktop automation, run:
+
+```bash
+computer-use-linux guard-accessibility
+```
+
+The guard registers a passive AT-SPI window-activation listener and watches
+GNOME's saved `toolkit-accessibility` key. It re-enables the key after a reset
+and verifies it by readback, with periodic checks as well as change monitoring.
+This affects all applications using the current user's GNOME setting, not just
+the target app. It does not start a screen reader or change focus.
+
+`mcp`, `setup`, `setup_accessibility`, and `get_app_state` never start this guard.
+Stop it with Ctrl-C or SIGTERM **before** intentionally disabling accessibility.
+Stopping ends setting writes and removes its listener without disabling other
+accessibility clients or restoring an old saved value. A reset and reassertion
+are not atomic: an app launched in that interval may still need restarting.
+This is an opt-in mitigation, not a guarantee that every GNOME toggle sequence
+preserves application accessibility trees.
+
 `computer-use-linux doctor` is the source of truth. Common failure modes and fixes:
 
 - **`accessibility.at_spi_bus.ok = false`** — AT-SPI registry isn't running or the toolkit bridge is off. Fix: `computer-use-linux setup` (or call the `setup_accessibility` MCP tool). Restart the apps you want to drive.
@@ -405,9 +478,30 @@ If you're running this on a shared workstation, set `ydotoold`'s socket permissi
 - **KWin / Hyprland / i3 / COSMIC / X11 windowing is unavailable** — check `doctor.windowing.backends`. KWin needs session-bus scripting; Hyprland needs `hyprctl`; i3 needs `i3-msg` and its IPC socket; generic X11 needs `wmctrl` and `xprop`. COSMIC needs `computer-use-linux-cosmic`, which the standard installers provide automatically; if you copied binaries by hand, copy the helper too or set `COMPUTER_USE_LINUX_COSMIC_HELPER`.
 - **Screenshots return black frames on multi-monitor setups** — known portal / compositor edge case. Use `get_app_state` with `include_screenshot: false` and rely on AT-SPI until the portal backend is healthy.
 - **`type_text` types into the wrong window** — pass an explicit target (`window_id`, `pid`, `wm_class`, `title`, or for terminals `tty` / `terminal_pid` / `terminal_command` / `terminal_cwd`). Without a target, input goes to whatever window currently has compositor focus.
+- **Wayland pointer actions miss an unfocused window** — injected pointer input is subject to the compositor's input-focus rules. Call `activate_window` for the target before `click`, `drag`, or coordinate `scroll`; a pointer can land at the requested coordinate without the unfocused surface receiving the action.
 
 If `doctor` is green and a specific tool still misbehaves, file an issue with the JSON output of `doctor` and the failing tool's request payload.
 
+For coordinate calibration, launch [the GTK4 probe](examples/coordinate_probe.py)
+in your test desktop and take a targeted screenshot. Choose the center of its
+green 10x10 square from that screenshot, convert by `scale`, and click relative
+to the same target window. The probe's delivered-event `hit: true` is the
+acceptance condition. Do not pass its widget-local `(85, 85)` directly as a
+window-relative click: margins and decorations belong to different spaces.
+
+[The semantic-click regression](scripts/semantic_click_test.py) runs against a
+built binary in an isolated graphical display and checks actual GTK3 button
+activation through MCP at scales 1 and 2. This does not establish correctness
+of Mutter 46 EWMH move/resize, all mixed-output layouts, or GNOME 50.4 pointer
+clicks.
+
+Native Wayland qualification on GNOME 50.1 at 133.3% display scale used the
+same green target for both modes: crop-relative `(122, 145)` and desktop
+`(2140, 193)` both delivered widget coordinates approximately `(84.81, 84.78)`
+with `hit: true` through uinput. These are one capture's measured coordinates,
+not reusable offsets. Raw surface coordinates must still be transformed before
+comparing them with a widget-local target.
+
 ## Related
 
 - [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) — the sibling MCP that gives an agent its **own** isolated Linux desktop (a hidden Xvfb display with its own apps and browser) instead of driving yours. It is the inverse of this project: `computer-use-linux` automates the desktop you are already on; `agent-workspace-linux` sandboxes the agent in a separate one. Use them together.

package/npm/README.md

@@ -9,9 +9,21 @@ mutating and can change real application state. The MCP tool list includes
 `ToolAnnotations` so hosts can distinguish read-only observation from mutating
 desktop actions.
 
+Plain left `click` by element index or selector prefers a native AT-SPI
+`click`, `press`, or `toggle` action over toolkit bounds,
+avoiding pointer-coordinate conversion when that action is available. Explicit
+`activate`/`jump` requests belong in `perform_action`, not `click`, even when
+bounds are missing. Explicit
+`x`/`y`, right clicks, and multi-clicks retain pointer semantics. For coordinate
+`click` or `scroll` with `relative: true`, use the clipped target-window
+screenshot crop origin and divide preview coordinates by screenshot `scale`.
+Do not use widget-local or raw GDK surface coordinates. A window target is
+required.
+
 ```bash
 npm install -g @agent-sh/computer-use-linux
 computer-use-linux doctor
+pi install npm:@agent-sh/computer-use-linux
 hermes skills tap add agent-sh/computer-use-linux
 hermes skills install agent-sh/computer-use-linux/computer-use-linux
 hermes mcp add computer-use-linux --command computer-use-linux --args mcp
@@ -19,6 +31,28 @@ hermes mcp test computer-use-linux
 hermes mcp configure computer-use-linux
 ```
 
+For an optional MCP completion notification, set
+`COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1` in the server environment and have the
+agent call `complete_interaction` after its desktop work. `notify-send` must be
+installed with an available desktop notification service. The cue is best effort
+and does not provide exclusive desktop ownership.
+This cue is available only to directly spawned MCP hosts; the native Pi extension
+does not yet forward the flag or include the tool in its catalog.
+
+If accessibility is disabled, run `computer-use-linux setup`. Setup writes and
+reads back GNOME's `toolkit-accessibility` setting and warns if only runtime
+accessibility is available. Restart target apps if their trees remain empty.
+
+To explicitly keep that saved setting enabled during desktop automation, run
+`computer-use-linux guard-accessibility` in a foreground terminal. It registers
+a passive AT-SPI window-activation listener and reasserts the current user's
+GNOME `toolkit-accessibility` key after resets, verifying each write by readback.
+This affects other apps using the same user setting. MCP, setup, and
+`get_app_state` never start it automatically. Stop with Ctrl-C or SIGTERM before
+disabling accessibility; stopping removes its listener and ends writes without
+disabling other clients or restoring an old value. An app launched during a
+reset may still need restarting, so this is not a complete GNOME toggle fix.
+
 The generated Hermes config should look like this:
 
 ```yaml
@@ -35,6 +69,11 @@ GitHub release for this package version and verifies the `.sha256` asset before
 installing it. It also installs the matching `computer-use-linux-cosmic` helper
 used for COSMIC desktop window targeting.
 
+When installed through Pi, the package supplies native, dynamically loaded
+`computer_use_linux_*` tools. No separate MCP adapter or manual MCP
+configuration is required. Native tools require Pi 0.84.4 or newer; the
+standalone CLI wrapper retains Node.js 18 support.
+
 If you already built or installed the binary yourself, set
 `COMPUTER_USE_LINUX_BIN=/path/to/computer-use-linux` to make the wrapper use
 that executable instead.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.4.10",
+  "version": "0.6.0",
   "description": "Linux desktop-control MCP server: AT-SPI accessibility trees, Wayland/X11 input, screenshots, and compositor window targeting.",
   "license": "MIT",
   "type": "commonjs",
@@ -40,18 +40,41 @@
     "npm/README.md",
     "npm/bin/computer-use-linux.js",
     "npm/install.js",
-    "pi/"
+    "pi/extension/index.ts",
+    "pi/extension/generated-tools.ts",
+    "pi/extension/mcp-client.bundle.cjs",
+    "pi/extension/THIRD_PARTY_NOTICES.txt"
   ],
   "pi": {
     "extensions": ["./pi/extension/index.ts"],
     "skills": ["./skills/computer-use-linux/SKILL.md"]
   },
   "scripts": {
+    "build:pi": "npm run build --prefix pi",
+    "check:pi-bundle": "npm run build:check --prefix pi",
     "postinstall": "node npm/install.js",
-    "pack:check": "npm pack --dry-run",
+    "pack:check": "node scripts/check_npm_package.mjs",
+    "test:pi": "npm test --prefix pi",
+    "typecheck:pi": "npm run typecheck --prefix pi",
     "test:wrapper": "node npm/bin/computer-use-linux.js --help"
   },
   "engines": {
     "node": ">=18"
+  },
+  "peerDependencies": {
+    "@earendil-works/pi-ai": "*",
+    "@earendil-works/pi-coding-agent": "*",
+    "typebox": "*"
+  },
+  "peerDependenciesMeta": {
+    "@earendil-works/pi-ai": {
+      "optional": true
+    },
+    "@earendil-works/pi-coding-agent": {
+      "optional": true
+    },
+    "typebox": {
+      "optional": true
+    }
   }
 }