@agent-sh/computer-use-linux 0.7.45 → 0.7.47
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/package.json +1 -1
- package/pi/extension/generated-tools.ts +15 -15
- package/pi/extension/index.ts +1 -0
package/README.md
CHANGED
|
@@ -531,9 +531,9 @@ files.
|
|
|
531
531
|
- **Native X11 transport.** Native X11 queries (window origins, root screenshots, `doctor`) connect and read replies under one shared 2-second deadline enforced in the transport itself, including the handshake and every reply. A stalled X server closes the query connection instead of leaving a blocked worker behind. Unix sockets, literal IP addresses, and `localhost` need no resolver. A remote hostname in `DISPLAY` needs `getent ahosts` for bounded name resolution; if it is missing or fails, the X11 route is reported as unavailable.
|
|
532
532
|
- **Window registry**: `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
|
|
533
533
|
- **niri backend**: prefers `niri msg`, falling back to direct JSON IPC over the same session socket. `is_minimized` maps to `hidden`. Bounds use `layout.window_size` (or `tile_size`) and, when available, the tile position plus the window's offset and the output's logical origin. Uniform output scaling converts these to device pixels; unknown or mixed scaling omits bounds. Missing window positions stay `null`, so listing and focusing still work without enabling coordinate targeting. An accepted focus action is verified by querying the focused window again.
|
|
534
|
-
- **GNOME extension fallback**: recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`.
|
|
535
|
-
- **COSMIC helper**: `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it.
|
|
536
|
-
- **Indicator overlay**: the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening.
|
|
534
|
+
- **GNOME extension fallback**: recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`. `setup-window-targeting` and `install.sh` install it in `$XDG_DATA_HOME/gnome-shell/extensions` (`~/.local/share/gnome-shell/extensions` when `XDG_DATA_HOME` is unset or empty), where GNOME Shell loads user extensions. When the Shell already loaded a user copy, `setup-window-targeting` updates the copy the Shell reports. When the extension is already enabled, both ask for a new login after an upgrade that GNOME Shell would run: a changed `extension.js`, or a changed `metadata.json` field the Shell acts on, such as `shell-version`. The Shell only checks that `name` and `description` are present, and only displays them, `url`, `version-name` and `donations`, so a change to those alone needs none. `install.sh` checks the four string fields and asks for a login when `donations` changes.
|
|
535
|
+
- **COSMIC helper**: `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it. Like `computer-use-linux`, the helper checks its command line first: `-h` or `--help` prints help and exits 0, and a command, flag or argument it does not take exits 2 before it connects to the compositor.
|
|
536
|
+
- **Indicator overlay**: the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening. The indicator takes no arguments: `-h` or `--help` prints help, and any other argument exits 2 before it takes a lock or connects to the compositor.
|
|
537
537
|
- **Terminal enrichment**: `list_windows` cross-references each terminal window with its controlling TTY and the foreground process on that TTY, so `type_text` / `press_key` can target "the terminal where `pytest` is running" without the host ever knowing the window id.
|
|
538
538
|
|
|
539
539
|
## Security
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.47",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
11
|
-
export const GENERATED_TOOL_CATALOG_HASH = "
|
|
12
|
-
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.47";
|
|
11
|
+
export const GENERATED_TOOL_CATALOG_HASH = "5220d9a11046797491e8899ea59c66d1d5d450c1457086867d8ae9784c297acd";
|
|
12
|
+
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "5cb7f910f9317b8b249fad1b994f1d771bd0854ddcad3a9b1f4b040ed2c050b5";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|
|
14
14
|
[
|
|
15
15
|
{
|
|
@@ -128,7 +128,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
128
128
|
},
|
|
129
129
|
"terminal_command": {
|
|
130
130
|
"default": null,
|
|
131
|
-
"description": "
|
|
131
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
132
132
|
"type": [
|
|
133
133
|
"string",
|
|
134
134
|
"null"
|
|
@@ -410,7 +410,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
410
410
|
},
|
|
411
411
|
"x": {
|
|
412
412
|
"default": null,
|
|
413
|
-
"description": "Point x in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.",
|
|
413
|
+
"description": "Point x in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.\nWith coordinate_space=\"atspi\", a global AT-SPI screen x instead.",
|
|
414
414
|
"format": "int32",
|
|
415
415
|
"type": [
|
|
416
416
|
"integer",
|
|
@@ -419,7 +419,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
419
419
|
},
|
|
420
420
|
"y": {
|
|
421
421
|
"default": null,
|
|
422
|
-
"description": "Point y in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.",
|
|
422
|
+
"description": "Point y in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.\nWith coordinate_space=\"atspi\", a global AT-SPI screen y instead.",
|
|
423
423
|
"format": "int32",
|
|
424
424
|
"type": [
|
|
425
425
|
"integer",
|
|
@@ -726,7 +726,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
726
726
|
},
|
|
727
727
|
"terminal_command": {
|
|
728
728
|
"default": null,
|
|
729
|
-
"description": "Terminal command substring. Resolves a window
|
|
729
|
+
"description": "Terminal command substring, ignoring ASCII case. Resolves a window\ntarget and scopes the tree when possible.",
|
|
730
730
|
"type": [
|
|
731
731
|
"string",
|
|
732
732
|
"null"
|
|
@@ -965,7 +965,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
965
965
|
},
|
|
966
966
|
"terminal_command": {
|
|
967
967
|
"default": null,
|
|
968
|
-
"description": "
|
|
968
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
969
969
|
"type": [
|
|
970
970
|
"string",
|
|
971
971
|
"null"
|
|
@@ -1146,7 +1146,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1146
1146
|
},
|
|
1147
1147
|
"pid": {
|
|
1148
1148
|
"default": null,
|
|
1149
|
-
"description": "Process id of the target app.",
|
|
1149
|
+
"description": "Process id of the target app: an element another process owns is\nrefused. With pid alone, no window is needed. When pid picks the window\n(with app_id, wm_class or window_title, or for\nscreenshot_after=\"window\"), it must match exactly one.",
|
|
1150
1150
|
"minimum": 0,
|
|
1151
1151
|
"type": [
|
|
1152
1152
|
"integer",
|
|
@@ -1250,7 +1250,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1250
1250
|
"openWorldHint": true,
|
|
1251
1251
|
"readOnlyHint": false
|
|
1252
1252
|
},
|
|
1253
|
-
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1253
|
+
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). With PIXELFLUX_CU and PIXELFLUX_CU_TOKEN set, chords go to pixelflux. An unreachable pixelflux falls through to the routes below; a pixelflux error does not. On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1254
1254
|
"inputSchema": {
|
|
1255
1255
|
"$defs": {
|
|
1256
1256
|
"ScreenshotOutputFormat": {
|
|
@@ -1363,7 +1363,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1363
1363
|
},
|
|
1364
1364
|
"terminal_command": {
|
|
1365
1365
|
"default": null,
|
|
1366
|
-
"description": "
|
|
1366
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
1367
1367
|
"type": [
|
|
1368
1368
|
"string",
|
|
1369
1369
|
"null"
|
|
@@ -1549,7 +1549,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1549
1549
|
},
|
|
1550
1550
|
"terminal_command": {
|
|
1551
1551
|
"default": null,
|
|
1552
|
-
"description": "
|
|
1552
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
1553
1553
|
"type": [
|
|
1554
1554
|
"string",
|
|
1555
1555
|
"null"
|
|
@@ -2057,7 +2057,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2057
2057
|
},
|
|
2058
2058
|
"pid": {
|
|
2059
2059
|
"default": null,
|
|
2060
|
-
"description": "Process id of the target app.",
|
|
2060
|
+
"description": "Process id of the target app: an element another process owns is\nrefused. With pid alone, no window is needed. When pid picks the window\n(with app_id, wm_class or window_title, or for\nscreenshot_after=\"window\"), it must match exactly one.",
|
|
2061
2061
|
"minimum": 0,
|
|
2062
2062
|
"type": [
|
|
2063
2063
|
"integer",
|
|
@@ -2124,7 +2124,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2124
2124
|
]
|
|
2125
2125
|
},
|
|
2126
2126
|
"value": {
|
|
2127
|
-
"description": "A number sets an element with the
|
|
2127
|
+
"description": "A value that parses as a floating-point number (such as 5, -2.5, 1e3,\ninf or NaN, with no surrounding spaces) sets an element with the\nAT-SPI Value interface (slider, spin button). Otherwise the text\nreplaces an editable text element's contents. Text on a Value-only\nelement is refused.",
|
|
2128
2128
|
"type": "string"
|
|
2129
2129
|
},
|
|
2130
2130
|
"window_id": {
|
|
@@ -2305,7 +2305,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2305
2305
|
},
|
|
2306
2306
|
"terminal_command": {
|
|
2307
2307
|
"default": null,
|
|
2308
|
-
"description": "
|
|
2308
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
2309
2309
|
"type": [
|
|
2310
2310
|
"string",
|
|
2311
2311
|
"null"
|