@agent-sh/computer-use-linux 0.5.0 → 0.6.0

package/README.md

@@ -61,6 +61,23 @@ Screenshot payloads are size-bounded by default before they are returned to the
 - `press_key` — keys / chords; can focus a window or terminal first
 - `type_text` — literal text input, optionally targeted at a window or terminal
 
+For a plain left `click` by element index or selector, a recognized native
+AT-SPI `click`, `press`, or `toggle` action takes precedence
+over bounds. Entry `activate` and slider `jump` must be requested explicitly
+with `perform_action`; they are never a substitute for a pointer click, even
+when bounds are missing.
+This avoids pointer conversion for GTK3 HiDPI
+extents and GTK4 zero-origin bounds when the element exposes such an action.
+The preference does not substitute an arbitrary action name for a coordinate
+click. Explicit `x`/`y`, right clicks, and double/multiple clicks retain pointer
+semantics. Re-check application state after either kind of activation.
+
+For coordinate `click` or `scroll` with `relative: true`, select a target window
+and measure from its clipped screenshot crop origin. Divide preview `x` and
+`y` by the returned screenshot `scale` before passing them. These are not raw
+GDK surface or widget-local coordinates; decorations and clipping can change
+the origin. A missing window target is rejected.
+
 Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report.
 
 **Semantic actions**
@@ -75,6 +92,8 @@ Targeted `press_key`/`type_text` results append focused-element feedback from AT
 
 **Conditional host execution**
 
+- `complete_interaction` - optional desktop completion notification, registered only with `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`. Repeated calls can create repeated notifications; it does not provide desktop exclusivity.
+
 - `run_shell` — same-user `/bin/sh -c` execution without login-profile loading, registered only when the server operator starts the MCP process with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. It is deliberately absent by default and is not a sandbox.
 
 ### MCP safety contract
@@ -99,6 +118,7 @@ The binary also exposes the same capabilities from the CLI for scripting and deb
 computer-use-linux mcp                                  # stdio MCP server
 computer-use-linux doctor                               # JSON readiness report
 computer-use-linux setup                                # enable AT-SPI
+computer-use-linux guard-accessibility                  # explicit foreground GNOME accessibility guard
 computer-use-linux setup-window-targeting               # install GNOME Shell extension
 computer-use-linux apps
 computer-use-linux state [APP_NAME]
@@ -358,6 +378,7 @@ Most setups need none of these — `doctor` and the installers pick sensible def
 
 | Variable | Effect |
 | --- | --- |
+| `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE` | Set exactly to `1` to expose the optional `complete_interaction` notification tool. Requires `notify-send` and a desktop notification service; disabled by default. |
 | `COMPUTER_USE_LINUX_COSMIC_HELPER` | Path to the `computer-use-linux-cosmic` helper when it isn't next to the binary or on `PATH`. |
 | `CU_DISABLE_ABS_POINTER` | Disable the uinput absolute pointer and click through `ydotool` instead for setups where the abs-pointer device misbehaves. |
 | `COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through the RemoteDesktop portal on Wayland, skipping auto-detection. |
@@ -408,6 +429,44 @@ If you're running this on a shared workstation, set `ydotoold`'s socket permissi
 
 ## Troubleshooting
 
+To receive an explicit completion cue, start the MCP server with
+`COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`. This exposes `complete_interaction`,
+a parameter-free tool the agent calls once after finishing its desktop work.
+It submits a notification through `notify-send` with a two-second execution
+limit and bounded process cleanup. Missing services, errors, or timeouts return
+`cue: "skipped"`; notification settings may suppress a submitted cue. This does
+not reserve the desktop or prove that other clients have stopped sending input.
+No sound or additional desktop settings are enabled by this option.
+This option currently applies only to directly spawned MCP hosts. The native Pi
+extension does not forward the flag or include this optional tool in its catalog.
+
+`setup` and `setup_accessibility` write and read back GNOME's
+`org.gnome.desktop.interface toolkit-accessibility` setting, even if AT-SPI
+is already enabled at runtime. A runtime-only success is reported as a warning:
+newly launched GTK apps may still have no tree. Enabling the saved key may
+require restarting target apps. Other accessibility tools can change that key
+later; setup does not continuously override user settings.
+
+For an explicit foreground guard while using desktop automation, run:
+
+```bash
+computer-use-linux guard-accessibility
+```
+
+The guard registers a passive AT-SPI window-activation listener and watches
+GNOME's saved `toolkit-accessibility` key. It re-enables the key after a reset
+and verifies it by readback, with periodic checks as well as change monitoring.
+This affects all applications using the current user's GNOME setting, not just
+the target app. It does not start a screen reader or change focus.
+
+`mcp`, `setup`, `setup_accessibility`, and `get_app_state` never start this guard.
+Stop it with Ctrl-C or SIGTERM **before** intentionally disabling accessibility.
+Stopping ends setting writes and removes its listener without disabling other
+accessibility clients or restoring an old saved value. A reset and reassertion
+are not atomic: an app launched in that interval may still need restarting.
+This is an opt-in mitigation, not a guarantee that every GNOME toggle sequence
+preserves application accessibility trees.
+
 `computer-use-linux doctor` is the source of truth. Common failure modes and fixes:
 
 - **`accessibility.at_spi_bus.ok = false`** — AT-SPI registry isn't running or the toolkit bridge is off. Fix: `computer-use-linux setup` (or call the `setup_accessibility` MCP tool). Restart the apps you want to drive.
@@ -419,9 +478,30 @@ If you're running this on a shared workstation, set `ydotoold`'s socket permissi
 - **KWin / Hyprland / i3 / COSMIC / X11 windowing is unavailable** — check `doctor.windowing.backends`. KWin needs session-bus scripting; Hyprland needs `hyprctl`; i3 needs `i3-msg` and its IPC socket; generic X11 needs `wmctrl` and `xprop`. COSMIC needs `computer-use-linux-cosmic`, which the standard installers provide automatically; if you copied binaries by hand, copy the helper too or set `COMPUTER_USE_LINUX_COSMIC_HELPER`.
 - **Screenshots return black frames on multi-monitor setups** — known portal / compositor edge case. Use `get_app_state` with `include_screenshot: false` and rely on AT-SPI until the portal backend is healthy.
 - **`type_text` types into the wrong window** — pass an explicit target (`window_id`, `pid`, `wm_class`, `title`, or for terminals `tty` / `terminal_pid` / `terminal_command` / `terminal_cwd`). Without a target, input goes to whatever window currently has compositor focus.
+- **Wayland pointer actions miss an unfocused window** — injected pointer input is subject to the compositor's input-focus rules. Call `activate_window` for the target before `click`, `drag`, or coordinate `scroll`; a pointer can land at the requested coordinate without the unfocused surface receiving the action.
 
 If `doctor` is green and a specific tool still misbehaves, file an issue with the JSON output of `doctor` and the failing tool's request payload.
 
+For coordinate calibration, launch [the GTK4 probe](examples/coordinate_probe.py)
+in your test desktop and take a targeted screenshot. Choose the center of its
+green 10x10 square from that screenshot, convert by `scale`, and click relative
+to the same target window. The probe's delivered-event `hit: true` is the
+acceptance condition. Do not pass its widget-local `(85, 85)` directly as a
+window-relative click: margins and decorations belong to different spaces.
+
+[The semantic-click regression](scripts/semantic_click_test.py) runs against a
+built binary in an isolated graphical display and checks actual GTK3 button
+activation through MCP at scales 1 and 2. This does not establish correctness
+of Mutter 46 EWMH move/resize, all mixed-output layouts, or GNOME 50.4 pointer
+clicks.
+
+Native Wayland qualification on GNOME 50.1 at 133.3% display scale used the
+same green target for both modes: crop-relative `(122, 145)` and desktop
+`(2140, 193)` both delivered widget coordinates approximately `(84.81, 84.78)`
+with `hit: true` through uinput. These are one capture's measured coordinates,
+not reusable offsets. Raw surface coordinates must still be transformed before
+comparing them with a widget-local target.
+
 ## Related
 
 - [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) — the sibling MCP that gives an agent its **own** isolated Linux desktop (a hidden Xvfb display with its own apps and browser) instead of driving yours. It is the inverse of this project: `computer-use-linux` automates the desktop you are already on; `agent-workspace-linux` sandboxes the agent in a separate one. Use them together.

package/npm/README.md

@@ -9,6 +9,17 @@ mutating and can change real application state. The MCP tool list includes
 `ToolAnnotations` so hosts can distinguish read-only observation from mutating
 desktop actions.
 
+Plain left `click` by element index or selector prefers a native AT-SPI
+`click`, `press`, or `toggle` action over toolkit bounds,
+avoiding pointer-coordinate conversion when that action is available. Explicit
+`activate`/`jump` requests belong in `perform_action`, not `click`, even when
+bounds are missing. Explicit
+`x`/`y`, right clicks, and multi-clicks retain pointer semantics. For coordinate
+`click` or `scroll` with `relative: true`, use the clipped target-window
+screenshot crop origin and divide preview coordinates by screenshot `scale`.
+Do not use widget-local or raw GDK surface coordinates. A window target is
+required.
+
 ```bash
 npm install -g @agent-sh/computer-use-linux
 computer-use-linux doctor
@@ -20,6 +31,28 @@ hermes mcp test computer-use-linux
 hermes mcp configure computer-use-linux
 ```
 
+For an optional MCP completion notification, set
+`COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1` in the server environment and have the
+agent call `complete_interaction` after its desktop work. `notify-send` must be
+installed with an available desktop notification service. The cue is best effort
+and does not provide exclusive desktop ownership.
+This cue is available only to directly spawned MCP hosts; the native Pi extension
+does not yet forward the flag or include the tool in its catalog.
+
+If accessibility is disabled, run `computer-use-linux setup`. Setup writes and
+reads back GNOME's `toolkit-accessibility` setting and warns if only runtime
+accessibility is available. Restart target apps if their trees remain empty.
+
+To explicitly keep that saved setting enabled during desktop automation, run
+`computer-use-linux guard-accessibility` in a foreground terminal. It registers
+a passive AT-SPI window-activation listener and reasserts the current user's
+GNOME `toolkit-accessibility` key after resets, verifying each write by readback.
+This affects other apps using the same user setting. MCP, setup, and
+`get_app_state` never start it automatically. Stop with Ctrl-C or SIGTERM before
+disabling accessibility; stopping removes its listener and ends writes without
+disabling other clients or restoring an old value. An app launched during a
+reset may still need restarting, so this is not a complete GNOME toggle fix.
+
 The generated Hermes config should look like this:
 
 ```yaml

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.5.0",
+  "version": "0.6.0",
   "description": "Linux desktop-control MCP server: AT-SPI accessibility trees, Wayland/X11 input, screenshots, and compositor window targeting.",
   "license": "MIT",
   "type": "commonjs",

package/pi/extension/generated-tools.ts

@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.5.0";
-export const GENERATED_TOOL_CATALOG_HASH = "3dd52d53c100e240fe2789dd1818ef80fd30202331c0aca2718212e2a79e47ff";
-export const GENERATED_SHELL_TOOL_CATALOG_HASH = "13406c87ad78dbdea7e5480e3e482be14c97687fa92320fde8e16f2efbbd3339";
+export const GENERATED_SERVER_VERSION = "0.6.0";
+export const GENERATED_TOOL_CATALOG_HASH = "9107297df765ce3b907540db172cc7823b7664586255db1064a40a44483d6fd6";
+export const GENERATED_SHELL_TOOL_CATALOG_HASH = "e0bc41db617ea6d10d812cbddce6ef78036e44251360f5fac9c207dd0d5ce392";
 export const GENERATED_MCP_TOOLS =
 [
   {
@@ -102,7 +102,7 @@ export const GENERATED_MCP_TOOLS =
       "openWorldHint": true,
       "readOnlyHint": false
     },
-    "description": "Click an element by index, semantic selector, or desktop coordinate pixels from screenshot metadata.",
+    "description": "Click an element by index, semantic selector, or desktop coordinate pixels from screenshot metadata. Plain left activation prefers a native AT-SPI click/press/toggle action, avoiding toolkit coordinate scaling. Entry activate and slider jump actions are not substituted for pointer clicks. Explicit coordinates, right clicks, and multi-clicks retain pointer semantics.",
     "inputSchema": {
       "$schema": "https://json-schema.org/draft/2020-12/schema",
       "properties": {
@@ -153,7 +153,7 @@ export const GENERATED_MCP_TOOLS =
         },
         "relative": {
           "default": null,
-          "description": "Interpret `x`/`y` as relative to the targeted window's top-left corner\n(the same coordinate space as a window-cropped `screenshot`). Requires a\nwindow target; ignored otherwise.",
+          "description": "Interpret `x`/`y` from the clipped window screenshot crop origin, in\ncoordinate pixels before preview resizing. Divide preview pixels by its\nscale first. This is not a toolkit widget or raw GDK surface origin.\nRequires a window target; missing targets are rejected.",
           "type": [
             "boolean",
             "null"
@@ -1043,7 +1043,7 @@ export const GENERATED_MCP_TOOLS =
         },
         "relative": {
           "default": null,
-          "description": "Interpret `x`/`y` as relative to the targeted window's top-left corner\n(the same coordinate space as a window-cropped `screenshot`). Requires a\nwindow target; ignored otherwise.",
+          "description": "Interpret `x`/`y` from the clipped window screenshot crop origin, in\ncoordinate pixels before preview resizing. Divide preview pixels by its\nscale first. This is not a toolkit widget or raw GDK surface origin.\nRequires a window target; missing targets are rejected.",
           "type": [
             "boolean",
             "null"

package/skills/computer-use-linux/SKILL.md

@@ -56,6 +56,31 @@ If `doctor` selects ydotool as the input backend, also enable its per-user daemo
 
 On GNOME Wayland, log out and back in after `setup-window-targeting` if the GNOME Shell extension was newly installed.
 
+For MCP hosts with `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`, call the optional
+`complete_interaction` tool once after finishing desktop interaction. A skipped
+cue is not a task failure. This notification does not guarantee exclusive
+desktop ownership or that other clients have stopped sending input.
+This applies only to directly spawned MCP hosts, not the native Pi extension.
+
+`setup_accessibility` verifies the saved GNOME `toolkit-accessibility` key
+separately from runtime AT-SPI. Inspect its warning and readback before assuming
+new apps can expose trees. Other accessibility tools may change the key later;
+setup does not hold it enabled continuously.
+
+### Optional foreground accessibility guard
+
+Skip unless: the user explicitly wants GNOME's saved `toolkit-accessibility`
+setting kept enabled while desktop automation runs.
+
+Run `computer-use-linux guard-accessibility` in a foreground terminal. It
+registers a passive AT-SPI window-activation listener and watches/reasserts the
+saved key with readback. The setting affects all apps for the current user.
+`mcp`, setup, and `get_app_state` never start this guard automatically.
+Stop with Ctrl-C or SIGTERM before intentionally disabling accessibility.
+Stopping ends writes and removes its listener without disabling other clients
+or restoring a previous saved value. Apps launched during a reset/reassertion
+race may still need restarting; do not claim a complete GNOME toggle fix.
+
 ## Configure Your Agent
 
 The `computer-use-linux` binary is an MCP server. Configure it as a stdio MCP server in your agent of choice:
@@ -87,6 +112,26 @@ If the binary is not on `PATH`, use the absolute path (typically `~/.local/bin/c
 9. For text input, prefer `type_text` with a target selector (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `tty`, `terminal_pid`, `terminal_command`, or `terminal_cwd`) rather than relying on current focus.
 10. After mutating actions, re-check state with `get_app_state`, `focused_window`, or an app-specific readback.
 
+Plain left element/index/selector `click` prefers native AT-SPI `click`,
+`press`, or `toggle` over toolkit bounds, avoiding coordinate
+conversion when available. This preference does not replace a coordinate click
+with an arbitrary action name. Explicit `x`/`y`, right clicks, and double/multiple
+clicks retain pointer semantics.
+Use `perform_action` explicitly for entry `activate` or slider `jump`; `click`
+never substitutes those actions, including when bounds are unavailable.
+
+### Screenshot-relative coordinates
+
+Skip unless: a coordinate `click` or `scroll` uses `relative: true`.
+
+Select a target window and use its clipped screenshot crop origin. Divide
+preview `x`/`y` by screenshot `scale` first. Widget-local and raw GDK surface
+coordinates are not interchangeable with that origin; missing window targets
+are rejected. For calibration, use the repository's
+`examples/coordinate_probe.py`: select the green square from the screenshot
+and require a delivered-event `hit: true`. Do not pass widget-local `(85, 85)`
+directly to a window-relative click.
+
 ## Pitfalls
 
 - Already-running GTK, Qt, and Electron apps may need a restart after AT-SPI is enabled.