@agent-sh/computer-use-linux 0.7.57 → 0.7.59

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -173,13 +173,27 @@ On Hyprland with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a window target wit
173
173
  | X11 (EWMH window managers) | Keys; pointer only without a window manager | XInput 2 multi-pointer is part of the X server. Checked with xterm, GTK 3 and Chrome under openbox, and the pointer without a window manager. |
174
174
  | sway 1.10+ | Keys and pointer | [ext-transient-seat-v1](https://wayland.app/protocols/ext-transient-seat-v1) plus foreign-toplevel activation on a named seat ([sway view.c](https://github.com/swaywm/sway/blob/master/sway/tree/view.c)). GTK 3, GTK 4 and Chromium bind only the seats that exist when they start ([gdkdisplay-wayland.c](https://gitlab.gnome.org/GNOME/gtk/-/blob/gtk-3-24/gdk/wayland/gdkdisplay-wayland.c), [wayland_seat.cc](https://source.chromium.org/chromium/chromium/src/+/main:ui/ozone/platform/wayland/host/wayland_seat.cc)), so they never see the agent seat and have no route; foot and other clients that bind new seats get the keys. Xwayland windows have no route. Older sway has no transient seats. |
175
175
  | Hyprland | Keys (`sendkeystate`, US layout) | One seat; `sendkeystate` is its per-window key dispatcher. No second pointer. |
176
- | GNOME | None | One seat, and Mutter has no per-window input request. |
177
- | KDE Plasma (KWin) | None | One `SeatInterface` and no per-window input request. |
176
+ | GNOME | Pointer through a virtual pen (below); no keys | One seat, and Mutter has no per-window input request. A tablet has its own cursor. |
177
+ | KDE Plasma (KWin) | None | One `SeatInterface` and no per-window input request. KWin 6.2+ gives a tablet its own cursor too, but maps a tablet with no configured output to the active output, which follows the user's pointer. |
178
178
  | COSMIC | None | cosmic-comp creates one seat and has no transient-seat protocol. |
179
179
  | niri | None | One seat and no transient-seat protocol. |
180
180
  | labwc, river-classic, other wlroots compositors | None | One seat (labwc activates foreign toplevels on it; river-classic lists multiple seats as a TODO). |
181
181
 
182
- Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`.
182
+ Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`. `capabilities.background_pointer` shows the GNOME pen route and its limits, or why there is none.
183
+
184
+ On GNOME Wayland (and KDE Plasma, below), background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. GNOME 48 and later default the lower button to a right click and the upper one to a middle click, so before each action the server sets its own pen's button actions to match the buttons it presses, and resets them when the pen goes away. A drag is the pen tip pressed at the start, moved and lifted at the end. The pen leaves proximity after each action: libinput pairs a pen with a touchscreen and ignores that screen's touches while the pen is near, and that screen can be the user's. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input.
185
+
186
+ Limits:
187
+
188
+ - A pen tap or a touch focuses the window it lands on, as a click does, so the user's typing goes there after it. The result says when the keyboard focus moved.
189
+ - The window check sees windows, not GNOME Shell's own UI. The call is refused while the screen is locked, while no window has focus (the overview may be open), and when the target is not on the workspace of the focused window, but a Shell dialog, notification banner or open menu over the point is not seen.
190
+ - Mutter sends tablet events only to clients that support the tablet protocol (GTK 3 and 4, Qt, Chromium, Firefox and Xwayland do). Other clients never see the pen, and the call cannot tell: re-observe the window. This is also why the pen is not the default route: the uinput pointer reaches every client.
191
+ - A drag stays inside one window. A touch pan scrolls only where the app pans on touch.
192
+ - The pen leaves proximity right after each action, so hover effects such as tooltips end with it.
193
+ - Mutter maps an unconfigured tablet across the whole layout, and an unconfigured touchscreen to the builtin panel. When a device maps elsewhere than the point needs, the server writes that device's `output` setting (`org.gnome.desktop.peripherals.tablet` or `.touchscreen`, at the path keyed by the devices' own vendor and product ids, `6375:7065` and `6375:7463`), waits for mutter's `org.gnome.Mutter.InputMapping` to confirm the new mapping, and resets the setting when the device goes away. No other device's setting is touched.
194
+ - The devices appear on first use and go away after ten idle minutes or when the server exits.
195
+
196
+ On KDE Plasma Wayland with KWin 6.3 or later the same route runs as `kwin-pen` and `kwin-touch`. KWin gives a tablet its own cursor from 6.3 (it tracks the tablet cursor itself and moves it whether or not the client under it supports tablets). KWin maps a tablet with no setting to the active output, which follows the user's pointer, and a touchscreen to the internal panel or the first output. So before input the server sets the pen's `mapToWorkspace` and the touchscreen's `outputName` (the output under the point) through `org.kde.KWin.InputDevice`, reads them back, and once the devices are gone removes what KWin wrote for them into `kcminputrc` (the groups keyed by these devices' vendor, product and name) with `kwriteconfig6`. The window check goes through KWin scripting: the point must be on the target, on the desktop and activity the user sees, not minimized or hidden, with no window, panel, popup or notification stacked above it. The call is refused while KWin shows the desktop or runs a full-screen effect such as the Overview (`org.kde.kwin.Effects.activeEffects`), and when another setting for the device moves where input lands (calibration, input or output area, rotation). A scroll reads the output's rotation with `kscreen-doctor` and refuses a rotated or flipped output; without `kscreen-doctor` there is no touch route. There is no route at all when KWin's `SyncWithMouse` tablet option (`kcminputrc` `[Tablet]`) is on, since it moves the user's pointer to where the pen was, when KWin runs with `KWIN_WAYLAND_EMULATE_TABLET`, which sends pen input over apps without tablet support through the user's pointer, or without `kreadconfig6` and `kwriteconfig6`.
183
197
 
184
198
  Some clients mishandle a second `wl_seat` (Zed, [zed-industries/zed#66735](https://github.com/zed-industries/zed/issues/66735)); that is one more reason the sway seat exists only during a call. A menu the agent opens on sway closes when its call ends, since the seat that opened it goes away. While sway is in a binding mode other than `default`, background keys have no route, since the mode's bindings would take them.
185
199
 
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@agent-sh/computer-use-linux",
3
- "version": "0.7.57",
3
+ "version": "0.7.59",
4
4
  "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
5
5
  "license": "MIT",
6
6
  "type": "commonjs",
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
7
7
  annotations: Record<string, unknown>;
8
8
  }
9
9
 
10
- export const GENERATED_SERVER_VERSION = "0.7.57";
11
- export const GENERATED_TOOL_CATALOG_HASH = "2bb4a324370516666dd67bc2c676007de98ebfb3c8179796578e51eaa4275728";
12
- export const GENERATED_SHELL_TOOL_CATALOG_HASH = "dfc420c6297a2f61f92e6379083c558cab420d1fdbafccc00e7f8390adef847b";
10
+ export const GENERATED_SERVER_VERSION = "0.7.59";
11
+ export const GENERATED_TOOL_CATALOG_HASH = "de04450abc876a3712849d85daad7b80ed6656e1ddf21e2bfd16e94712b4247c";
12
+ export const GENERATED_SHELL_TOOL_CATALOG_HASH = "26a00c32301af2704aeaf97fe3cb58f9429e255305ad4e17efd50db506cd6717";
13
13
  export const GENERATED_MCP_TOOLS =
14
14
  [
15
15
  {
@@ -279,7 +279,7 @@ export const GENERATED_MCP_TOOLS =
279
279
  }
280
280
  ],
281
281
  "default": null,
282
- "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). Needs x and y in screenshot capture pixels over the\nwindow the call targets, without relative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
282
+ "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
283
283
  },
284
284
  "element_index": {
285
285
  "default": null,
@@ -501,9 +501,24 @@ export const GENERATED_MCP_TOOLS =
501
501
  "openWorldHint": true,
502
502
  "readOnlyHint": false
503
503
  },
504
- "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no window target or relative coordinates. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
504
+ "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME and KDE Plasma Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
505
505
  "inputSchema": {
506
506
  "$defs": {
507
+ "KeyboardDelivery": {
508
+ "description": "How type_text and press_key deliver input.",
509
+ "oneOf": [
510
+ {
511
+ "const": "focus",
512
+ "description": "Focus the target window, then type into it.",
513
+ "type": "string"
514
+ },
515
+ {
516
+ "const": "background",
517
+ "description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
518
+ "type": "string"
519
+ }
520
+ ]
521
+ },
507
522
  "ScreenshotOutputFormat": {
508
523
  "enum": [
509
524
  "png",
@@ -514,6 +529,26 @@ export const GENERATED_MCP_TOOLS =
514
529
  },
515
530
  "$schema": "https://json-schema.org/draft/2020-12/schema",
516
531
  "properties": {
532
+ "app_id": {
533
+ "default": null,
534
+ "description": "Application id, ignoring ASCII case: the full desktop id, or its last\ndot-separated segment.",
535
+ "type": [
536
+ "string",
537
+ "null"
538
+ ]
539
+ },
540
+ "delivery": {
541
+ "anyOf": [
542
+ {
543
+ "$ref": "#/$defs/KeyboardDelivery"
544
+ },
545
+ {
546
+ "type": "null"
547
+ }
548
+ ],
549
+ "default": null,
550
+ "description": "\"background\" drags with a virtual pen on GNOME and KDE Plasma Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
551
+ },
517
552
  "end_x": {
518
553
  "description": "End x in raw screenshot capture pixels.",
519
554
  "format": "int32",
@@ -566,6 +601,15 @@ export const GENERATED_MCP_TOOLS =
566
601
  "null"
567
602
  ]
568
603
  },
604
+ "pid": {
605
+ "default": null,
606
+ "description": "Process id. It must match exactly one window.",
607
+ "minimum": 0,
608
+ "type": [
609
+ "integer",
610
+ "null"
611
+ ]
612
+ },
569
613
  "quality": {
570
614
  "description": "JPEG quality from 1 to 95 (default 80) for post-action screenshots. Ignored for png.",
571
615
  "maximum": 95,
@@ -617,6 +661,32 @@ export const GENERATED_MCP_TOOLS =
617
661
  "description": "Start y in raw screenshot capture pixels.",
618
662
  "format": "int32",
619
663
  "type": "integer"
664
+ },
665
+ "window_id": {
666
+ "default": null,
667
+ "description": "Window id from a recent list_windows: the window a background drag\nruns in. The first target field set decides, in this order:\nwindow_id, pid, app_id, wm_class, window_title. A drag with\n\"focus\" delivery ignores the target.",
668
+ "maximum": 9007199254740991,
669
+ "minimum": 0,
670
+ "type": [
671
+ "integer",
672
+ "null"
673
+ ]
674
+ },
675
+ "window_title": {
676
+ "default": null,
677
+ "description": "Window title substring, ignoring ASCII case.",
678
+ "type": [
679
+ "string",
680
+ "null"
681
+ ]
682
+ },
683
+ "wm_class": {
684
+ "default": null,
685
+ "description": "Window manager class, an exact match ignoring ASCII case.",
686
+ "type": [
687
+ "string",
688
+ "null"
689
+ ]
620
690
  }
621
691
  },
622
692
  "required": [
@@ -2024,7 +2094,7 @@ export const GENERATED_MCP_TOOLS =
2024
2094
  }
2025
2095
  ],
2026
2096
  "default": null,
2027
- "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). Needs x and y in screenshot capture pixels over the\nwindow the call targets, without relative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
2097
+ "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME and KDE Plasma Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
2028
2098
  },
2029
2099
  "direction": {
2030
2100
  "description": "up, down, left or right (case-insensitive).",
@@ -236,7 +236,7 @@ directly to a window-relative click.
236
236
  - `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive.
237
237
  - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
238
238
  - On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
239
- - To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
239
+ - To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland and KDE Plasma Wayland (KWin 6.3+), `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, and a scroll is a touch pan (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
240
240
  - Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
241
241
  - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.
242
242
  - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.