@agent-sh/computer-use-linux 0.7.53 → 0.7.54
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +5 -3
- package/package.json +1 -1
- package/pi/extension/generated-tools.ts +1 -1
- package/pi/extension/index.ts +1 -0
package/README.md
CHANGED
|
@@ -53,7 +53,7 @@ The Rust crate is published as [`computer-use-linux`](https://crates.io/crates/c
|
|
|
53
53
|
Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive `xdotool` against an X11 root window or shell out to OCR over screenshots. It differs in five ways:
|
|
54
54
|
|
|
55
55
|
- **Wayland actually works.** On Wayland, pointer actions use `/dev/uinput` or `org.freedesktop.portal.RemoteDesktop`. ydotool sends only untargeted wheel events there, and a coordinate action with neither route is refused. Literal text prefers KDE clipboard integration on Plasma, `wtype` on compatible Wayland compositors, and portal keysyms on other Wayland sessions, including GNOME with ydotool installed. Portal text startup or conversion errors fail the call without replaying text through ydotool. Raw ydotool text accepts printable ASCII, tab and newline only, and its physical US key positions can produce different text under the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, then on a native X11 session a root-window `GetImage` (device pixels, no toolkit scaling), and finally spawn `gnome-screenshot` for background/systemd contexts where the DBus paths are denied.
|
|
56
|
-
- **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed. Terminal windows are enriched with their controlling TTY and foreground process so agents can target "the terminal running `pytest`" directly. Every `window_id` fits JavaScript's exact integer range (at most 2^53-1), so JSON clients can pass it back unchanged; a larger value is rejected with a prompt to re-run `list_windows`.
|
|
56
|
+
- **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, the niri IPC socket, the `umbriel` CLI, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed. Terminal windows are enriched with their controlling TTY and foreground process so agents can target "the terminal running `pytest`" directly. Every `window_id` fits JavaScript's exact integer range (at most 2^53-1), so JSON clients can pass it back unchanged; a larger value is rejected with a prompt to re-run `list_windows`.
|
|
57
57
|
- **Semantic selectors first, pixel coordinates when needed.** Tools like `click`, `perform_action`, and `set_value` accept `role` / `name` / `text` / `states` selectors or `element_index` values backed by AT-SPI, and every mutating tool can return a post-action screenshot and updated accessibility tree in the same round trip. Pixel coordinates remain available for rendering-only surfaces (canvas, games, X clients without ATK).
|
|
58
58
|
- **Visible to humans, invisible to screenshots.** While an agent acts, `computer-use-linux-indicator` renders a click-through overlay with a per-agent cursor glide, click ripple, keycaps/typing banner, screen-edge glow, and status pill, and hides automatically under a shared capture lock before any screenshot is read.
|
|
59
59
|
- **One JSON readiness report.** `computer-use-linux doctor` returns a structured document covering platform, portals, AT-SPI, windowing, input, and a `readiness` summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.
|
|
@@ -205,6 +205,7 @@ Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). niri window list
|
|
|
205
205
|
| KDE Plasma / KWin | temporary KWin DBus scripting | Lists and focuses windows through Plasma 5 or 6 `org.kde.KWin` scripting APIs when the session bus exposes them. IDs are derived from the KWin window UUID, stay the same for a window across calls, and fit JavaScript integers. |
|
|
206
206
|
| Hyprland | `hyprctl clients -j`; focus through the Lua `hl.dsp.focus` dispatcher, then `focuswindow` | Requires `hyprctl` in the desktop session. With `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, targeted keyboard input uses `send_key_state` and targeted capture uses `grim -T`, so neither focuses the window. |
|
|
207
207
|
| niri | `niri msg` with a direct JSON IPC fallback | Lists toplevels and focuses exact window IDs. When the host does not pass `NIRI_SOCKET`, the server reads it from its parent processes or the systemd user environment; without it, discovery requires an unambiguous socket matching `WAYLAND_DISPLAY`. The fallback works without the `niri` binary. Missing positions remain `null`; bounds are omitted if output scaling is unknown or mixed. |
|
|
208
|
+
| Umbriel | `umbriel windows --json` and `umbriel msg window-focus` | Lists toplevels with pids and focuses exact window IDs through the `umbriel` CLI, which needs `UMBRIEL_SOCKET` (passed by the host, read from the session's environment, or found as `umbriel-$WAYLAND_DISPLAY.sock` in `XDG_RUNTIME_DIR`). IDs are the first 52 bits of Umbriel's window id, so they fit JavaScript integers. With one enabled output, bounds are scaled to capture pixels with `umbriel outputs --json`. With several, bounds are left out: xdg-desktop-portal-umbriel captures one output, not their union, so where a window sits in the capture is unknown. Windows on a workspace that is not shown are `hidden`. Validated on Umbriel 0.1.0 at scale 1.8. |
|
|
208
209
|
| i3, Sway | `i3-msg`, or `swaymsg` when i3 is not installed; on i3, optional `xprop` for PID hydration | Lists and focuses windows over the active i3 or Sway IPC socket. On Sway this covers native Wayland and Xwayland windows, with pids from the tree, and bounds scaled to capture pixels when every output has the same scale (left out with mixed scales). The socket is the first of `I3SOCK` and `SWAYSOCK` that names a socket file, then the newest i3 socket, then the newest Sway socket in `XDG_RUNTIME_DIR`. X11 windows keep their X11 id; a native Wayland window on Sway gets 2^32 plus its container id, so the two never collide. |
|
|
209
210
|
| labwc with Selkies IPC | `$XDG_RUNTIME_DIR/labwc.sock` | Requires the Selkies IPC patch; stock labwc has no such socket. Lists windows and focuses unique app-id/title matches through `wlr-foreign-toplevel-management`. IDs include pid, app id and title, fit JavaScript integers, and change when the title changes. Ambiguous windows are rejected. Bounds are omitted because the IPC does not expose output scaling. |
|
|
210
211
|
| COSMIC Wayland | `computer-use-linux-cosmic` helper | Installed automatically by `./install.sh`, `cargo install`, and npm. For custom/manual layouts, put the helper next to the main binary, on `PATH`, or point `COMPUTER_USE_LINUX_COSMIC_HELPER` at it. The on-screen indicator was validated here (three outputs, mixed fractional scaling). |
|
|
@@ -551,8 +552,9 @@ files.
|
|
|
551
552
|
- **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. Missing or failed xdotool commands return an error without a system-wide fallback. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.
|
|
552
553
|
- **Drag motion.** XTEST, uinput and portal drags use intermediate points paced 30 ms apart. Native X11 uses XTEST on `DISPLAY`; uinput and portal paths use the same trajectory. A failed press or motion still attempts button release, and the gesture is not replayed through another backend. Portal paths require shared endpoints and clamp intermediate points to the nearest shared monitor when the trajectory crosses a monitor gap. Verify the destination after dispatch.
|
|
553
554
|
- **Native X11 transport.** Native X11 queries (window origins, root screenshots, `doctor`) connect and read replies under one shared 2-second deadline enforced in the transport itself, including the handshake and every reply. A stalled X server closes the query connection instead of leaving a blocked worker behind. Unix sockets, literal IP addresses, and `localhost` need no resolver. A remote hostname in `DISPLAY` needs `getent ahosts` for bounded name resolution; if it is missing or fails, the X11 route is reported as unavailable.
|
|
554
|
-
- **Window registry**: `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
|
|
555
|
+
- **Window registry**: `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, the `umbriel` CLI, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
|
|
555
556
|
- **niri backend**: prefers `niri msg`, falling back to direct JSON IPC over the same session socket. `is_minimized` maps to `hidden`. Bounds use `layout.window_size` (or `tile_size`) and, when available, the tile position plus the window's offset and the output's logical origin. Uniform output scaling converts these to device pixels; unknown or mixed scaling omits bounds. Missing window positions stay `null`, so listing and focusing still work without enabling coordinate targeting. An accepted focus action is verified by querying the focused window again.
|
|
557
|
+
- **Umbriel backend**: runs the `umbriel` CLI. `active` maps to `focused` (Umbriel's own `focused` marks one window per workspace), and a minimized window or one on a workspace that is not shown maps to `hidden`. With one enabled output, logical window geometry is scaled by its scale from its origin. With several enabled outputs bounds are omitted, because the portal captures one output and the capture space of a window is unknown; listing and exact focus still work.
|
|
556
558
|
- **GNOME extension fallback**: recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`. `setup-window-targeting` and `install.sh` install it in `$XDG_DATA_HOME/gnome-shell/extensions` (`~/.local/share/gnome-shell/extensions` when `XDG_DATA_HOME` is unset or empty), where GNOME Shell loads user extensions. When the Shell already loaded a user copy, `setup-window-targeting` updates the copy the Shell reports. When the extension is already enabled, both ask for a new login after an upgrade that GNOME Shell would run: a changed `extension.js`, or a changed `metadata.json` field the Shell acts on, such as `shell-version`. The Shell only checks that `name` and `description` are present, and only displays them, `url`, `version-name` and `donations`, so a change to those alone needs none. `install.sh` checks the four string fields and asks for a login when `donations` changes.
|
|
557
559
|
- **COSMIC helper**: `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it. Like `computer-use-linux`, the helper checks its command line first: `-h` or `--help` prints help and exits 0, and a command, flag or argument it does not take exits 2 before it connects to the compositor.
|
|
558
560
|
- **Indicator overlay**: the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening. The indicator takes no arguments: `-h` or `--help` prints help, and any other argument exits 2 before it takes a lock or connects to the compositor.
|
|
@@ -622,7 +624,7 @@ preserves application accessibility trees.
|
|
|
622
624
|
- **`input.ydotool.ok = false` with an unsupported CLI message**: install ydotool 1.0.3 or newer. A running daemon or socket alone is not enough; `doctor` verifies the raw key, wheel, stdin typing, and absolute-movement command family before advertising the backend.
|
|
623
625
|
- **`input.uinput.ok = false`**: `/dev/uinput` isn't accessible to your user. Fix: add yourself to the `input` group (`sudo usermod -aG input $USER`) and re-login. On distros that ship `uinput` as a kernel module without auto-loading it, add `uinput` to `/etc/modules-load.d/`. Direct uinput supplies absolute pointer input only, so `doctor` also requires a keyboard-capable portal, wtype (on compatible Wayland compositors), xdotool, or ydotool backend.
|
|
624
626
|
- **Portal calls hang or time out**: `xdg-desktop-portal` or its backend (`-gnome`, `-gtk`, `-kde`, `-wlr`) crashed. Fix: check `journalctl --user -u xdg-desktop-portal -u xdg-desktop-portal-gnome --since '5 min ago'` and restart the relevant unit.
|
|
625
|
-
- **KWin / Hyprland / niri / i3 / Sway / COSMIC / X11 windowing is unavailable**: check `doctor.windowing.backends`. KWin needs session-bus scripting; Hyprland needs `hyprctl`; niri needs `NIRI_SOCKET` (passed by the host, or read from the server's parent processes or the systemd user environment), or an unambiguous matching `niri.*.sock` in `XDG_RUNTIME_DIR`; i3 and Sway need `i3-msg` or `swaymsg` and the IPC socket; generic X11 needs `wmctrl` and `xprop`. COSMIC needs `computer-use-linux-cosmic`, which the standard installers provide automatically; if you copied binaries by hand, copy the helper too or set `COMPUTER_USE_LINUX_COSMIC_HELPER`.
|
|
627
|
+
- **KWin / Hyprland / niri / Umbriel / i3 / Sway / COSMIC / X11 windowing is unavailable**: check `doctor.windowing.backends`. KWin needs session-bus scripting; Hyprland needs `hyprctl`; niri needs `NIRI_SOCKET` (passed by the host, or read from the server's parent processes or the systemd user environment), or an unambiguous matching `niri.*.sock` in `XDG_RUNTIME_DIR`; Umbriel needs the `umbriel` binary and `UMBRIEL_SOCKET`, or `umbriel-$WAYLAND_DISPLAY.sock` in `XDG_RUNTIME_DIR`; i3 and Sway need `i3-msg` or `swaymsg` and the IPC socket; generic X11 needs `wmctrl` and `xprop`. COSMIC needs `computer-use-linux-cosmic`, which the standard installers provide automatically; if you copied binaries by hand, copy the helper too or set `COMPUTER_USE_LINUX_COSMIC_HELPER`.
|
|
626
628
|
- **niri windows list correctly but targeted screenshots or relative clicks fail**: niri may omit window positions, leaving `bounds.x`/`bounds.y` as `null`. Unknown or mixed output scaling also leaves bounds unavailable. Focus the window first (`activate_window`) and use full-screen coordinates after checking a fresh screenshot.
|
|
627
629
|
- **Screenshots return black frames on multi-monitor setups**: a known portal / compositor edge case. Use `get_app_state` with `include_screenshot: false` and rely on AT-SPI until the portal backend is healthy.
|
|
628
630
|
- **`type_text` types into the wrong window**: pass an explicit target (`window_id`, `pid`, `wm_class`, `title`, or for terminals `tty` / `terminal_pid` / `terminal_command` / `terminal_cwd`). Without a target, input goes to whatever window currently has compositor focus.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.54",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,7 +7,7 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.54";
|
|
11
11
|
export const GENERATED_TOOL_CATALOG_HASH = "03d66a3e0e8ee39a68fe551e1bfe6cfe99463ef8e3cdbbf53d8d6b501a8435ff";
|
|
12
12
|
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "12bfcef2bf761a9990ba46734fd6a93a54c04a39209bcf17e62eecc7441da87a";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|