@agent-sh/computer-use-linux 0.7.1 → 0.7.3

package/README.md

@@ -26,7 +26,7 @@ The Rust crate is published as [`computer-use-linux`](https://crates.io/crates/c
 
 Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive `xdotool` against an X11 root window or shell out to OCR over screenshots. Four things set this one apart:
 
-- **Wayland actually works.** Pointer actions can use the `org.freedesktop.portal.RemoteDesktop` interface on Wayland, with `ydotool` / `ydotoold` (uinput) as the deterministic fallback. Literal text prefers `wtype` on compatible Wayland compositors when portal keyboard input is unavailable, preserving Unicode and the active layout before falling back to ydotool. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, and fall back to spawning `gnome-screenshot` for background/systemd contexts where both DBus paths are denied.
+- **Wayland actually works.** Pointer actions can use the `org.freedesktop.portal.RemoteDesktop` interface on Wayland, with `ydotool` / `ydotoold` (uinput) as the deterministic fallback. Literal text prefers `wtype` on compatible Wayland compositors when portal keyboard input is unavailable, preserving Unicode and the active layout before falling back to ydotool. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, then on a native X11 session a root-window `GetImage` (device pixels, no toolkit scaling), and finally spawn `gnome-screenshot` for background/systemd contexts where the DBus paths are denied.
 - **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, i3 IPC, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed.
 - **Semantic selectors, not pixel coordinates.** Tools like `click`, `perform_action`, and `set_value` accept `role` / `name` / `text` / `states` selectors backed by AT-SPI. Pixel coordinates remain available as a fallback for rendering-only surfaces (canvas, games, X clients without ATK).
 - **One JSON readiness report.** `computer-use-linux doctor` returns a structured document covering platform, portals, AT-SPI, windowing, input, and a `readiness` summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.
@@ -139,7 +139,7 @@ Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). Other compositor
 | i3 | `i3-msg`; optional `xprop` for PID hydration | Lists and focuses i3 windows over the active i3 IPC socket. |
 | COSMIC Wayland | `computer-use-linux-cosmic` helper | Installed automatically by `./install.sh`, `cargo install`, and npm. For custom/manual layouts, put the helper next to the main binary, on `PATH`, or point `COMPUTER_USE_LINUX_COSMIC_HELPER` at it. |
 | Sway / generic wlroots | no dedicated backend yet | AT-SPI, screenshots, and global `ydotool` input can still work; exact window list/focus is currently unavailable unless another backend applies. |
-| Generic X11 / XFCE / other EWMH WMs | `wmctrl` plus `xprop` | Lists, focuses, moves, and resizes windows; keyboard input prefers `xdotool`/XTEST. |
+| Generic X11 / XFCE / other EWMH WMs | `wmctrl` plus `xprop` | Lists, focuses, moves, and resizes windows; keyboard input prefers `xdotool`/XTEST. Window origins are read from the X server, since `wmctrl -lG` counts the frame offset twice. |
 
 If you run on a desktop not covered above, or a covered backend does not come up cleanly, please open an issue with the output of `computer-use-linux doctor` so we can extend the matrix honestly.
 
@@ -351,7 +351,7 @@ Spawn the binary with `["mcp"]` as the argv tail. It speaks JSON-RPC over stdio
    computer-use-linux doctor | jq .readiness
    ```
 
-   Aim for `can_register_mcp_tools`, `can_build_accessibility_tree`, `can_send_development_input`, and `can_query_windows` all `true`. The `blockers` array should be empty.
+   Aim for `can_register_mcp_tools`, `can_build_accessibility_tree`, `can_send_development_input`, `can_query_windows`, and `can_capture_screenshots` all `true`. The `blockers` array should be empty. `can_capture_screenshots` means a route was detected, not that a test capture succeeded.
 
 2. **If `accessibility.at_spi_bus.ok = false`** — run `computer-use-linux setup` (or call the `setup_accessibility` MCP tool). This sets:
    - `org.gnome.desktop.interface toolkit-accessibility true`
@@ -385,7 +385,7 @@ Most setups need none of these — `doctor` and the installers pick sensible def
 | `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through `ydotool`, skipping the portal and KDE clipboard paths; pointer forcing also skips native-X11 `xdotool` coordinate clicks. |
 | `COMPUTER_USE_LINUX_FORCE_XDOTOOL_KEYBOARD` | Prefer `xdotool`/XTEST keyboard input when `DISPLAY` is available. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_KEYBOARD=1` takes precedence. |
 | `COMPUTER_USE_LINUX_XDOTOOL_TYPE_DELAY_MS` | Per-character delay for `xdotool type` in milliseconds (default `12`). `0` is faster but can deliver characters out of order on some X servers. |
-| `COMPUTER_USE_LINUX_SCREENSHOT_BACKEND` | Force a single screenshot backend, skipping the fallback chain. Accepts `gnome-shell`, `portal`, or `gnome-screenshot`. Pin `gnome-screenshot` for background/systemd contexts where the GNOME Shell and portal DBus paths are denied. |
+| `COMPUTER_USE_LINUX_SCREENSHOT_BACKEND` | Force a single screenshot backend, skipping the fallback chain. Accepts `gnome-shell`, `portal`, `x11`, or `gnome-screenshot`. `x11` works only on a native X11 session. Pin `gnome-screenshot` for background/systemd contexts where the GNOME Shell and portal DBus paths are denied. |
 | `COMPUTER_USE_LINUX_ENABLE_SHELL` | Set exactly to `1` before starting the MCP server to register the destructive `run_shell` tool. Unset by default. Do not enable for untrusted or unattended MCP hosts. |
 
 **Build-time identity overrides** (set while compiling a downstream embedded

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.7.1",
+  "version": "0.7.3",
   "description": "Linux desktop-control MCP server: AT-SPI accessibility trees, Wayland/X11 input, screenshots, and compositor window targeting.",
   "license": "MIT",
   "type": "commonjs",

package/pi/extension/generated-tools.ts

@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.7.1";
-export const GENERATED_TOOL_CATALOG_HASH = "768e9650a96630f6b01672b7e94489d3ea5661544752175112879a5368f8dca9";
-export const GENERATED_SHELL_TOOL_CATALOG_HASH = "4030cfe80339e7006fdf944ad172731d8c2955f6aa1f9ba22c6dd80805355a26";
+export const GENERATED_SERVER_VERSION = "0.7.3";
+export const GENERATED_TOOL_CATALOG_HASH = "a28811e528ada60c60d5e493d3680249428a65037cbd741ab0efa4711dd91837";
+export const GENERATED_SHELL_TOOL_CATALOG_HASH = "3e5700e7be824d965170c202204cb512ee41b7162088050a8d7f479d617d15d3";
 export const GENERATED_MCP_TOOLS =
 [
   {
@@ -90,7 +90,6 @@ export const GENERATED_MCP_TOOLS =
           ]
         }
       },
-      "title": "ActivateWindowParams",
       "type": "object"
     },
     "name": "activate_window"
@@ -219,7 +218,6 @@ export const GENERATED_MCP_TOOLS =
           ]
         }
       },
-      "title": "ClickParams",
       "type": "object"
     },
     "name": "click"
@@ -272,7 +270,6 @@ export const GENERATED_MCP_TOOLS =
         "end_x",
         "end_y"
       ],
-      "title": "DragParams",
       "type": "object"
     },
     "name": "drag"
@@ -487,7 +484,6 @@ export const GENERATED_MCP_TOOLS =
           ]
         }
       },
-      "title": "GetAppStateParams",
       "type": "object"
     },
     "name": "get_app_state"
@@ -612,7 +608,6 @@ export const GENERATED_MCP_TOOLS =
         "x",
         "y"
       ],
-      "title": "MoveWindowParams",
       "type": "object"
     },
     "name": "move_window"
@@ -679,7 +674,6 @@ export const GENERATED_MCP_TOOLS =
           ]
         }
       },
-      "title": "ActionParams",
       "type": "object"
     },
     "name": "perform_action"
@@ -768,7 +762,6 @@ export const GENERATED_MCP_TOOLS =
       "required": [
         "key"
       ],
-      "title": "PressKeyParams",
       "type": "object"
     },
     "name": "press_key"
@@ -865,7 +858,6 @@ export const GENERATED_MCP_TOOLS =
         "width",
         "height"
       ],
-      "title": "ResizeWindowParams",
       "type": "object"
     },
     "name": "resize_window"
@@ -1002,7 +994,6 @@ export const GENERATED_MCP_TOOLS =
           ]
         }
       },
-      "title": "ScreenshotParams",
       "type": "object"
     },
     "name": "screenshot"
@@ -1102,7 +1093,6 @@ export const GENERATED_MCP_TOOLS =
       "required": [
         "direction"
       ],
-      "title": "ScrollParams",
       "type": "object"
     },
     "name": "scroll"
@@ -1168,7 +1158,6 @@ export const GENERATED_MCP_TOOLS =
       "required": [
         "value"
       ],
-      "title": "SetValueParams",
       "type": "object"
     },
     "name": "set_value"
@@ -1285,7 +1274,6 @@ export const GENERATED_MCP_TOOLS =
       "required": [
         "text"
       ],
-      "title": "TypeTextParams",
       "type": "object"
     },
     "name": "type_text"
@@ -1337,7 +1325,6 @@ export const GENERATED_OPTIONAL_MCP_TOOLS =
       "required": [
         "command"
       ],
-      "title": "RunShellParams",
       "type": "object"
     },
     "name": "run_shell"

package/pi/extension/index.ts

@@ -18,10 +18,11 @@ import {
 	constants,
 	existsSync,
 	readFileSync,
+	statSync,
 } from "node:fs";
 import { createRequire } from "node:module";
 import { homedir } from "node:os";
-import { join } from "node:path";
+import { delimiter, join } from "node:path";
 import { Type, type TSchema } from "typebox";
 import {
 	GENERATED_MCP_TOOLS,
@@ -154,6 +155,8 @@ function isRecord(value: unknown): value is Record<string, unknown> {
 
 function executable(path: string): boolean {
 	try {
+		// A directory on PATH also passes X_OK; only a regular file can be spawned.
+		if (!statSync(path).isFile()) return false;
 		accessSync(path, constants.X_OK);
 		return true;
 	} catch {
@@ -161,6 +164,23 @@ function executable(path: string): boolean {
 	}
 }
 
+/**
+ * First executable `name` in a PATH-style list, or `null`. Empty segments are
+ * skipped rather than read as the current directory.
+ */
+export function findExecutableOnPath(
+	pathValue: string | undefined,
+	name: string,
+	isExecutable: (path: string) => boolean = executable,
+): string | null {
+	for (const dir of (pathValue ?? "").split(delimiter)) {
+		if (!dir) continue;
+		const candidate = join(dir, name);
+		if (isExecutable(candidate)) return candidate;
+	}
+	return null;
+}
+
 function runtimeEnvironment(): Record<string, string> {
 	const allowed = new Set([
 		"PATH",
@@ -247,6 +267,15 @@ function defaultFindBinary(): BinaryLaunch | null {
 		return { binaryPath: bundledBinary, env };
 	}
 
+	// A temporary extension (`pi -e npm:@agent-sh/computer-use-linux`) is staged
+	// without the postinstall-downloaded platform binary, so fall back to a
+	// global npm or cargo install. The MCP client still checks the server
+	// version and tool catalog, so a mismatched install fails with that reason.
+	const onPath = findExecutableOnPath(process.env.PATH, "computer-use-linux");
+	if (onPath) {
+		return { binaryPath: onPath, env };
+	}
+
 	return null;
 }
 
@@ -504,7 +533,7 @@ export function createComputerUseLinuxExtension(
 			if (!resolved) {
 				throw new Error(
 					"computer-use-linux binary was not found. Reinstall " +
-						`${PACKAGE_NAME} or set COMPUTER_USE_LINUX_BIN.`,
+						`${PACKAGE_NAME}, install computer-use-linux on PATH, or set COMPUTER_USE_LINUX_BIN.`,
 				);
 			}
 			const { ComputerUseMcpClient } = loadClientModule();
@@ -622,7 +651,7 @@ export function createComputerUseLinuxExtension(
 
 			if (!resolveLaunch() && ctx.hasUI) {
 				ctx.ui.notify(
-					`${PACKAGE_NAME}: binary not found; reinstall the package or set COMPUTER_USE_LINUX_BIN.`,
+					`${PACKAGE_NAME}: binary not found; reinstall the package, install computer-use-linux on PATH, or set COMPUTER_USE_LINUX_BIN.`,
 					"warning",
 				);
 			}

package/skills/computer-use-linux/SKILL.md

@@ -172,6 +172,7 @@ Ready output should have:
 - `can_build_accessibility_tree: true`
 - `can_query_windows: true`
 - `can_send_development_input: true`
+- `can_capture_screenshots: true`
 - `blockers: []`
 
 Then test with your agent by calling the `doctor` tool or asking the agent to list desktop windows.

package/skills/computer-use-linux/references/pi-setup.md

@@ -102,6 +102,12 @@ The extension looks for `computer-use-linux` in this order:
 
 1. `COMPUTER_USE_LINUX_BIN`
 2. The binary downloaded inside the installed npm package
+3. `computer-use-linux` on `PATH`, from `npm install -g` or `cargo install`
+
+A temporary extension (`pi -e npm:@agent-sh/computer-use-linux`) is staged
+without the downloaded binary, so it relies on the `PATH` step. The extension
+still checks that the server version and tool catalog match, so a mismatched
+install fails with that reason instead of starting.
 
 Reinstall the package if the bundled binary is missing:
 
@@ -131,4 +137,5 @@ Ready output has:
 - `can_build_accessibility_tree: true`
 - `can_query_windows: true`
 - `can_send_development_input: true`
+- `can_capture_screenshots: true`
 - `blockers: []`