@agent-sh/computer-use-linux 0.7.44 → 0.7.46
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +3 -3
- package/package.json +1 -1
- package/pi/extension/generated-tools.ts +24 -12
package/README.md
CHANGED
|
@@ -531,9 +531,9 @@ files.
|
|
|
531
531
|
- **Native X11 transport.** Native X11 queries (window origins, root screenshots, `doctor`) connect and read replies under one shared 2-second deadline enforced in the transport itself, including the handshake and every reply. A stalled X server closes the query connection instead of leaving a blocked worker behind. Unix sockets, literal IP addresses, and `localhost` need no resolver. A remote hostname in `DISPLAY` needs `getent ahosts` for bounded name resolution; if it is missing or fails, the X11 route is reported as unavailable.
|
|
532
532
|
- **Window registry**: `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
|
|
533
533
|
- **niri backend**: prefers `niri msg`, falling back to direct JSON IPC over the same session socket. `is_minimized` maps to `hidden`. Bounds use `layout.window_size` (or `tile_size`) and, when available, the tile position plus the window's offset and the output's logical origin. Uniform output scaling converts these to device pixels; unknown or mixed scaling omits bounds. Missing window positions stay `null`, so listing and focusing still work without enabling coordinate targeting. An accepted focus action is verified by querying the focused window again.
|
|
534
|
-
- **GNOME extension fallback**: recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`.
|
|
535
|
-
- **COSMIC helper**: `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it.
|
|
536
|
-
- **Indicator overlay**: the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening.
|
|
534
|
+
- **GNOME extension fallback**: recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`. When the extension is already enabled, `setup-window-targeting` and `install.sh` ask for a new login after an upgrade that GNOME Shell would run: a changed `extension.js`, or a changed `metadata.json` field the Shell acts on, such as `shell-version`. The Shell only checks that the name and description are present, and only displays them and the link, so a change to those alone needs none.
|
|
535
|
+
- **COSMIC helper**: `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it. Like `computer-use-linux`, the helper checks its command line first: `-h` or `--help` prints help and exits 0, and a command, flag or argument it does not take exits 2 before it connects to the compositor.
|
|
536
|
+
- **Indicator overlay**: the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening. The indicator takes no arguments: `-h` or `--help` prints help, and any other argument exits 2 before it takes a lock or connects to the compositor.
|
|
537
537
|
- **Terminal enrichment**: `list_windows` cross-references each terminal window with its controlling TTY and the foreground process on that TTY, so `type_text` / `press_key` can target "the terminal where `pytest` is running" without the host ever knowing the window id.
|
|
538
538
|
|
|
539
539
|
## Security
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.46",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
11
|
-
export const GENERATED_TOOL_CATALOG_HASH = "
|
|
12
|
-
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.46";
|
|
11
|
+
export const GENERATED_TOOL_CATALOG_HASH = "5220d9a11046797491e8899ea59c66d1d5d450c1457086867d8ae9784c297acd";
|
|
12
|
+
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "5cb7f910f9317b8b249fad1b994f1d771bd0854ddcad3a9b1f4b040ed2c050b5";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|
|
14
14
|
[
|
|
15
15
|
{
|
|
@@ -128,7 +128,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
128
128
|
},
|
|
129
129
|
"terminal_command": {
|
|
130
130
|
"default": null,
|
|
131
|
-
"description": "
|
|
131
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
132
132
|
"type": [
|
|
133
133
|
"string",
|
|
134
134
|
"null"
|
|
@@ -410,6 +410,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
410
410
|
},
|
|
411
411
|
"x": {
|
|
412
412
|
"default": null,
|
|
413
|
+
"description": "Point x in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.\nWith coordinate_space=\"atspi\", a global AT-SPI screen x instead.",
|
|
413
414
|
"format": "int32",
|
|
414
415
|
"type": [
|
|
415
416
|
"integer",
|
|
@@ -418,6 +419,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
418
419
|
},
|
|
419
420
|
"y": {
|
|
420
421
|
"default": null,
|
|
422
|
+
"description": "Point y in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.\nWith coordinate_space=\"atspi\", a global AT-SPI screen y instead.",
|
|
421
423
|
"format": "int32",
|
|
422
424
|
"type": [
|
|
423
425
|
"integer",
|
|
@@ -464,10 +466,12 @@ export const GENERATED_MCP_TOOLS =
|
|
|
464
466
|
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
465
467
|
"properties": {
|
|
466
468
|
"end_x": {
|
|
469
|
+
"description": "End x in raw screenshot capture pixels.",
|
|
467
470
|
"format": "int32",
|
|
468
471
|
"type": "integer"
|
|
469
472
|
},
|
|
470
473
|
"end_y": {
|
|
474
|
+
"description": "End y in raw screenshot capture pixels.",
|
|
471
475
|
"format": "int32",
|
|
472
476
|
"type": "integer"
|
|
473
477
|
},
|
|
@@ -549,10 +553,12 @@ export const GENERATED_MCP_TOOLS =
|
|
|
549
553
|
"description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
|
|
550
554
|
},
|
|
551
555
|
"start_x": {
|
|
556
|
+
"description": "Start x in raw screenshot capture pixels: the coordinate_width space.\nDivide preview pixels by the screenshot scale first.",
|
|
552
557
|
"format": "int32",
|
|
553
558
|
"type": "integer"
|
|
554
559
|
},
|
|
555
560
|
"start_y": {
|
|
561
|
+
"description": "Start y in raw screenshot capture pixels.",
|
|
556
562
|
"format": "int32",
|
|
557
563
|
"type": "integer"
|
|
558
564
|
}
|
|
@@ -720,7 +726,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
720
726
|
},
|
|
721
727
|
"terminal_command": {
|
|
722
728
|
"default": null,
|
|
723
|
-
"description": "Terminal command substring. Resolves a window
|
|
729
|
+
"description": "Terminal command substring, ignoring ASCII case. Resolves a window\ntarget and scopes the tree when possible.",
|
|
724
730
|
"type": [
|
|
725
731
|
"string",
|
|
726
732
|
"null"
|
|
@@ -959,7 +965,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
959
965
|
},
|
|
960
966
|
"terminal_command": {
|
|
961
967
|
"default": null,
|
|
962
|
-
"description": "
|
|
968
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
963
969
|
"type": [
|
|
964
970
|
"string",
|
|
965
971
|
"null"
|
|
@@ -1057,6 +1063,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1057
1063
|
"properties": {
|
|
1058
1064
|
"action": {
|
|
1059
1065
|
"default": null,
|
|
1066
|
+
"description": "AT-SPI action to invoke: its name or description, ignoring ASCII case,\nor its 0-based index. Default: the element's first action.",
|
|
1060
1067
|
"type": [
|
|
1061
1068
|
"string",
|
|
1062
1069
|
"null"
|
|
@@ -1072,6 +1079,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1072
1079
|
},
|
|
1073
1080
|
"element_identifier": {
|
|
1074
1081
|
"default": null,
|
|
1082
|
+
"description": "A node's object_ref from get_app_state or get_tree_page. It names the\nAT-SPI object itself, not a snapshot index, and wins over element_index\nand the selectors.",
|
|
1075
1083
|
"type": [
|
|
1076
1084
|
"string",
|
|
1077
1085
|
"null"
|
|
@@ -1138,7 +1146,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1138
1146
|
},
|
|
1139
1147
|
"pid": {
|
|
1140
1148
|
"default": null,
|
|
1141
|
-
"description": "Process id of the target app.",
|
|
1149
|
+
"description": "Process id of the target app: an element another process owns is\nrefused. With pid alone, no window is needed. When pid picks the window\n(with app_id, wm_class or window_title, or for\nscreenshot_after=\"window\"), it must match exactly one.",
|
|
1142
1150
|
"minimum": 0,
|
|
1143
1151
|
"type": [
|
|
1144
1152
|
"integer",
|
|
@@ -1242,7 +1250,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1242
1250
|
"openWorldHint": true,
|
|
1243
1251
|
"readOnlyHint": false
|
|
1244
1252
|
},
|
|
1245
|
-
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1253
|
+
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). With PIXELFLUX_CU and PIXELFLUX_CU_TOKEN set, chords go to pixelflux. An unreachable pixelflux falls through to the routes below; a pixelflux error does not. On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1246
1254
|
"inputSchema": {
|
|
1247
1255
|
"$defs": {
|
|
1248
1256
|
"ScreenshotOutputFormat": {
|
|
@@ -1355,7 +1363,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1355
1363
|
},
|
|
1356
1364
|
"terminal_command": {
|
|
1357
1365
|
"default": null,
|
|
1358
|
-
"description": "
|
|
1366
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
1359
1367
|
"type": [
|
|
1360
1368
|
"string",
|
|
1361
1369
|
"null"
|
|
@@ -1541,7 +1549,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1541
1549
|
},
|
|
1542
1550
|
"terminal_command": {
|
|
1543
1551
|
"default": null,
|
|
1544
|
-
"description": "
|
|
1552
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
1545
1553
|
"type": [
|
|
1546
1554
|
"string",
|
|
1547
1555
|
"null"
|
|
@@ -1928,6 +1936,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1928
1936
|
},
|
|
1929
1937
|
"x": {
|
|
1930
1938
|
"default": null,
|
|
1939
|
+
"description": "Point x to scroll at in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.\nWithout x/y or element_index, the target window's centre, or the pointer.",
|
|
1931
1940
|
"format": "int32",
|
|
1932
1941
|
"type": [
|
|
1933
1942
|
"integer",
|
|
@@ -1936,6 +1945,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1936
1945
|
},
|
|
1937
1946
|
"y": {
|
|
1938
1947
|
"default": null,
|
|
1948
|
+
"description": "Point y to scroll at in raw screenshot capture pixels: the coordinate_width and\ncoordinate_height space. Divide preview pixels by the screenshot scale\nfirst. With relative=true, measured from the window screenshot's crop origin.",
|
|
1939
1949
|
"format": "int32",
|
|
1940
1950
|
"type": [
|
|
1941
1951
|
"integer",
|
|
@@ -1980,6 +1990,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1980
1990
|
},
|
|
1981
1991
|
"element_identifier": {
|
|
1982
1992
|
"default": null,
|
|
1993
|
+
"description": "A node's object_ref from get_app_state or get_tree_page. It names the\nAT-SPI object itself, not a snapshot index, and wins over element_index\nand the selectors.",
|
|
1983
1994
|
"type": [
|
|
1984
1995
|
"string",
|
|
1985
1996
|
"null"
|
|
@@ -2046,7 +2057,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2046
2057
|
},
|
|
2047
2058
|
"pid": {
|
|
2048
2059
|
"default": null,
|
|
2049
|
-
"description": "Process id of the target app.",
|
|
2060
|
+
"description": "Process id of the target app: an element another process owns is\nrefused. With pid alone, no window is needed. When pid picks the window\n(with app_id, wm_class or window_title, or for\nscreenshot_after=\"window\"), it must match exactly one.",
|
|
2050
2061
|
"minimum": 0,
|
|
2051
2062
|
"type": [
|
|
2052
2063
|
"integer",
|
|
@@ -2113,6 +2124,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2113
2124
|
]
|
|
2114
2125
|
},
|
|
2115
2126
|
"value": {
|
|
2127
|
+
"description": "A value that parses as a floating-point number (such as 5, -2.5, 1e3,\ninf or NaN, with no surrounding spaces) sets an element with the\nAT-SPI Value interface (slider, spin button). Otherwise the text\nreplaces an editable text element's contents. Text on a Value-only\nelement is refused.",
|
|
2116
2128
|
"type": "string"
|
|
2117
2129
|
},
|
|
2118
2130
|
"window_id": {
|
|
@@ -2293,7 +2305,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2293
2305
|
},
|
|
2294
2306
|
"terminal_command": {
|
|
2295
2307
|
"default": null,
|
|
2296
|
-
"description": "
|
|
2308
|
+
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
2297
2309
|
"type": [
|
|
2298
2310
|
"string",
|
|
2299
2311
|
"null"
|