@agent-sh/computer-use-linux 0.7.56 → 0.7.58

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -76,8 +76,8 @@ The crate was extracted from [`codex-desktop-linux`](https://github.com/avifenes
76
76
  | **Input** | `click` | Click by `element_index`, semantic selector (`role`/`name`/`text`/`states`), or desktop/window-relative coordinates (`left`/`right`/`middle`, `click_count`). |
77
77
  | | `drag` | Desktop coordinate drag from `(start_x, start_y)` to `(end_x, end_y)`. |
78
78
  | | `scroll` | Page-based vertical/horizontal scroll on an element, window, or pixel coordinate. |
79
- | | `press_key` | Single keys or chords (`Ctrl+L`, `Alt+F4`, `Return`), optionally focusing a target window or terminal first. |
80
- | | `type_text` | Literal text entry, optionally targeted at a window or terminal (`tty`, `terminal_pid`, `terminal_command`, `terminal_cwd`). |
79
+ | | `press_key` | Single keys or chords (`Ctrl+L`, `Alt+F4`, `Return`), optionally focusing a target window or terminal first, or with `delivery: "background"` to an element, browser tab or terminal without moving keyboard focus. |
80
+ | | `type_text` | Literal text entry, optionally targeted at a window or terminal (`tty`, `terminal_pid`, `terminal_command`, `terminal_cwd`), or with `delivery: "background"` to an element, browser tab or terminal without moving keyboard focus. |
81
81
  | **Semantic actions** | `perform_action` | Invoke any AT-SPI action exposed by an element (`Press`, `Activate`, `Toggle`, `Jump`, …); defaults to the primary action. |
82
82
  | | `set_value` | Write directly to a settable accessibility element (text fields, sliders, spinners). |
83
83
  | **Window management** | `activate_window` | Focus a window by `window_id`, `pid`, `app_id`, `wm_class`, `title`, or terminal selectors. |
@@ -152,6 +152,49 @@ On X11, text input sends LF and CRLF line endings as Return, with one newline pe
152
152
 
153
153
  On GNOME, a read-only keymap preflight rejects the whole string before input if a character is absent or requires a modifier level Mutter cannot synthesize. The background reader cannot establish the active layout group, so every configured group must support the text. This can reject text that works in one of several groups. Then the text goes into the focused editable field of the verified target window through AT-SPI EditableText: it is inserted at the caret, replaces a selection, and counts as typed (`ok: true`) only when the field reads it back with the rest of its text unchanged (in a field over 4096 characters, the 32 on each side of the insert). The field must be in the AT-SPI window that matches the target and that window must be active, so a field left focused in another window of the app is not used. A password field, focus outside an editable field, and text with a newline, tab or other control character keep the refusal; send keys with `press_key`, or fill the field with `set_value`. The check parses the keymap in pure Rust, with no system XKB library, and stops after a two-second keymap read deadline. It does not cover held modifiers or a keymap changed concurrently with typing.
154
154
 
155
+ ### Background keyboard delivery
156
+
157
+ `type_text` and `press_key` take `delivery: "background"` to reach a target without moving the user's keyboard focus ([#480](https://github.com/agent-sh/computer-use-linux/issues/480)). The target picks the route. Naming an element or a `browser_tab` selects background delivery on its own, and `delivery: "focus"` with one is refused. The result's `background` field says what happened: `{"requested": true, "used": true, "route": "..."}` when a background route delivered it.
158
+
159
+ When no route serves the target because the desktop or app lacks one (below, "no route"), the call is not blocked. The first such call in a server session sends nothing and returns a notice: `ok: false` without `isError`, `background: {"requested": true, "used": false, "delivered": false, "reason": "...", "retry_to_continue": true}`, and a message with the reason (and the setting that would open a route, where there is one), that normal input moves the user's keyboard focus and pointer, and that [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) gives the agent its own session. Repeating the call sends it through normal focused input, as without `delivery`, and so does every later call with no route, with only `background: {"requested": true, "used": false}` in the result. For an element, focused input first gives the element keyboard focus with AT-SPI `GrabFocus` and then types into its window (the target fields, or the snapshot's window); for a `browser_tab`, it targets the one window whose title contains the tab's name (with window fields, that window must show it). With Hyprland background mode on, normal input to a window is the same sendkeystate route, so a call with no route there stays an error. A wrong, ambiguous or stale target (an unmatched or ambiguous element, tab, tty or window, a replaced snapshot, a terminal whose pty belongs to another devpts instance, a point outside the target window or screen), the X server's Ctrl+Alt+F-key and Ctrl+Alt+BackSpace chords on the agent seat, and a failure after input started stay errors.
160
+
161
+ - **Element (AT-SPI).** Pass `element_index` with `snapshot_id`, `element_identifier`, or `role`/`name`/`states`, plus the app target to check the snapshot against, as for `set_value`. `type_text` inserts through EditableText at the element's caret, or at its end when the caret is not in it, and succeeds only when the field reads the text back with the rest unchanged. A selection is left alone and the caret is not moved. A line feed or tab goes in only on a multi-line element; other control characters have no route. A password field takes the text but cannot be read back, and the result says so, as with `set_value`. `press_key` runs the AT-SPI action the key means: Enter or space on a button or menu item, space on a check box, radio button or toggle, Enter on a link, and Enter on a single-line field that has an `activate` action. Other keys and chords with modifiers have no route there.
162
+ - **Browser tab (DevTools).** Start Chrome, Chromium or an Electron app yourself with `--remote-debugging-port` and its own `--user-data-dir`, and set `COMPUTER_USE_LINUX_DEVTOOLS` to the port or to the path of its `DevToolsActivePort` file. The server connects only to that port on 127.0.0.1, only when its listener belongs to the same user, and never starts or restarts a browser. `browser_tab` names the tab: a target id, or text in one tab's URL or title. Text goes in with `Input.insertText` at the tab's focused element, as typing would, and an input or textarea must read it back. Keys go with `Input.dispatchKeyEvent`. The tab needs a focused field for text. Shortcuts the browser handles outside the page, such as Ctrl+T, do not run this way. A field inside a frame from another origin is refused.
163
+ - **Terminal.** Pass `tty`, `terminal_pid`, `terminal_command`, `terminal_cwd`, or a terminal window. The server finds the program on that tty and tries, in order: kitty remote control (the `KITTY_LISTEN_ON` socket kitty gives its shells; needs `allow_remote_control` and `listen_on` in kitty.conf), `wezterm cli`, tmux (a pane on the tty, or the active pane of a client attached on it), GNU screen (the current window of a client on the tty, when no other client may be attached to its session; a screen window's own tty is refused, since window numbers can be swapped), TIOCSTI where `dev.tty.legacy_tiocsti` is 1 and the server has CAP_SYS_ADMIN, and the terminal's pty master through `pidfd_getfd` where ptrace rules allow it. The result names the channel it used. Text reaches kitty, wezterm and tmux on stdin and screen through a private file and its `^_` register, which is cleared after the paste, never as a process argument. A tty taken from a window must have its pty master held by that window's terminal, so a container's pts with the same number is refused. When nothing works, there is no route, and the reason names the terminal found and the setting that would open one. GNOME Terminal, Ptyxis and other VTE terminals and Ghostty have no remote input command, and Konsole's D-Bus route is not supported yet, so a program in them needs tmux or screen. Writing to `/dev/pts/N` only prints on the screen, so the server never does it.
164
+
165
+ On Hyprland with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a window target with `delivery: "background"` goes through `send_key_state` as in background mode, and so does a terminal window that no terminal channel reaches.
166
+
167
+ - **Whole window (agent seat).** A window target that is not an element, a browser tab or a terminal some channel above reaches goes through the agent's own keyboard focus, on desktops that ship one. On X11 the server adds an XInput 2 master pointer and keyboard named `computer-use-linux agent <pid>`, focuses the target with `XISetFocus` for that keyboard only, and types through its XTEST device; the user's master keeps its focus and cursor. Keys come from the user's keymap (first two levels), so a character it lacks has no route, and neither does a key or chord another client grabs on the root window (a window manager shortcut such as openbox's Alt+F4), since through the agent seat the grab would act on the user's session. The check takes and drops a passive grab of each key back to back; a key the user presses in that instant goes nowhere. On sway 1.10 and later the server creates an `ext-transient-seat-v1` seat, focuses the target on it with `zwlr_foreign_toplevel_handle_v1.activate`, and types through a virtual keyboard with its own keymap; `seat0` keeps its focus and its workspace (a target on a workspace the user is not viewing still gets the keys), and sway removes the seat when the server disconnects. Xwayland windows have no route, because Xwayland feeds every seat into one X keyboard that follows the X focus, and neither do scratchpad windows, because activating one shows it on the user's workspace; if seat0's focus moves anyway, no keys are sent. Alt and Super chords, and chords the sway config binds (its included files too), have no route, because sway runs its bindings for the agent seat too. The agent keymap uses keycodes above any real keyboard's, so `bindcode` and `bindsym --to-code` bindings never match an agent key. On Hyprland the route is `sendkeystate`, as in Hyprland background mode below, without the environment variable. On X11 the master appears on first use and goes away after ten idle minutes or when the server exits. On sway the transient seat lives for one call (each call waits 100 ms after creating it, so clients can bind the new seat before input arrives; a client busy for longer misses the input), so no other program sees it between calls: wtype, for one, binds the last seat it sees, and a password or autotype helper using it would otherwise type into the agent's window. The X11 master's name carries a window owned by the server's X connection, so a master left by a killed server is removed by the next one. When input fails partway, the result says it may have been partly delivered and the seat is rebuilt.
168
+
169
+ `click` and `scroll` take `delivery: "background"` too, with `x` and `y` in screenshot capture pixels and a window target: the agent seat's pointer moves and clicks, and the user's pointer stays. The point must be over that window (on sway, on a visible workspace), or nothing is pressed. On X11 this needs a session with no window manager, because a window manager answers a click from any pointer by focusing that window for the user (openbox did); with one running, background clicks have no route and background keys still work. On sway the click also focuses the window on the agent seat only.
170
+
171
+ | Desktop | Agent seat | Why |
172
+ | --- | --- | --- |
173
+ | X11 (EWMH window managers) | Keys; pointer only without a window manager | XInput 2 multi-pointer is part of the X server. Checked with xterm, GTK 3 and Chrome under openbox, and the pointer without a window manager. |
174
+ | sway 1.10+ | Keys and pointer | [ext-transient-seat-v1](https://wayland.app/protocols/ext-transient-seat-v1) plus foreign-toplevel activation on a named seat ([sway view.c](https://github.com/swaywm/sway/blob/master/sway/tree/view.c)). GTK 3, GTK 4 and Chromium bind only the seats that exist when they start ([gdkdisplay-wayland.c](https://gitlab.gnome.org/GNOME/gtk/-/blob/gtk-3-24/gdk/wayland/gdkdisplay-wayland.c), [wayland_seat.cc](https://source.chromium.org/chromium/chromium/src/+/main:ui/ozone/platform/wayland/host/wayland_seat.cc)), so they never see the agent seat and have no route; foot and other clients that bind new seats get the keys. Xwayland windows have no route. Older sway has no transient seats. |
175
+ | Hyprland | Keys (`sendkeystate`, US layout) | One seat; `sendkeystate` is its per-window key dispatcher. No second pointer. |
176
+ | GNOME | Pointer through a virtual pen (below); no keys | One seat, and Mutter has no per-window input request. A tablet has its own cursor. |
177
+ | KDE Plasma (KWin) | None | One `SeatInterface` and no per-window input request. KWin 6.2+ gives a tablet its own cursor too, but maps a tablet with no configured output to the active output, which follows the user's pointer. |
178
+ | COSMIC | None | cosmic-comp creates one seat and has no transient-seat protocol. |
179
+ | niri | None | One seat and no transient-seat protocol. |
180
+ | labwc, river-classic, other wlroots compositors | None | One seat (labwc activates foreign toplevels on it; river-classic lists multiple seats as a TODO). |
181
+
182
+ Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`. `capabilities.background_pointer` shows the GNOME pen route and its limits, or why there is none.
183
+
184
+ On GNOME Wayland, background `click`, `drag` and `scroll` go through a virtual pen and a virtual touchscreen instead of the user's pointer ([#480](https://github.com/agent-sh/computer-use-linux/issues/480); the idea and the tested approach are [@Ondra-Zik](https://github.com/Ondra-Zik)'s). Mutter gives a tablet tool its own cursor and sends its events through the tablet protocol, not `wl_pointer`, so the user's mouse stays where it is. Clicks are pen taps (left), the upper barrel button (right) and the lower one (middle); `click_count` repeats them. A drag is the pen tip pressed at the start, moved and lifted at the end. A scroll is a one-finger touch pan, 60 logical pixels per wheel notch (5 notches per page), split into strokes that stay inside the window and 48 pixels from the monitor's edges, where GNOME Shell starts its edge swipes; the finger rests before it lifts, so the app does not fling further. Before any input, the server checks through the GNOME Shell extension that the point is on the target window, on the workspace the user sees, and that no window stacked above it covers the point; both ends of a drag and every pan stroke must be on the window, and a pan always starts at the point. The results report the route as `gnome-pen` or `gnome-touch`. The devices need write access to `/dev/uinput`, as the uinput pointer does; without it, or without the extension, the call gets the notice and then focused input.
185
+
186
+ Limits:
187
+
188
+ - A pen tap or a touch focuses the window it lands on, as a click does, so the user's typing goes there after it. The result says when the keyboard focus moved.
189
+ - The window check sees windows, not GNOME Shell's own UI. The call is refused while the screen is locked, while no window has focus (the overview may be open), and when the target is not on the workspace of the focused window, but a Shell dialog, notification banner or open menu over the point is not seen.
190
+ - Mutter sends tablet events only to clients that support the tablet protocol (GTK 3 and 4, Qt, Chromium, Firefox and Xwayland do). Other clients never see the pen, and the call cannot tell: re-observe the window. This is also why the pen is not the default route: the uinput pointer reaches every client.
191
+ - A drag stays inside one window. A touch pan scrolls only where the app pans on touch.
192
+ - The pen hovers at its last point, with its own cursor, for 8 seconds after an action, then leaves proximity; hover effects such as tooltips end then.
193
+ - Mutter maps an unconfigured tablet across the whole layout, and an unconfigured touchscreen to the builtin panel. When a device maps elsewhere than the point needs, the server writes that device's `output` setting (`org.gnome.desktop.peripherals.tablet` or `.touchscreen`, at the path keyed by the devices' own vendor and product ids, `6375:7065` and `6375:7463`), waits for mutter's `org.gnome.Mutter.InputMapping` to confirm the new mapping, and resets the setting when the device goes away. No other device's setting is touched.
194
+ - The devices appear on first use and go away after ten idle minutes or when the server exits.
195
+
196
+ Some clients mishandle a second `wl_seat` (Zed, [zed-industries/zed#66735](https://github.com/zed-industries/zed/issues/66735)); that is one more reason the sway seat exists only during a call. A menu the agent opens on sway closes when its call ends, since the seat that opened it goes away. While sway is in a binding mode other than `default`, background keys have no route, since the mode's bindings would take them.
197
+
155
198
  ### On-screen indicator
156
199
 
157
200
  While an agent acts, the server drives `computer-use-linux-indicator`, a click-through overlay that shows what it is doing: a software cursor tinted per agent that glides to each coordinate `click`, `drag`, or `scroll` target and ripples on click (AT-SPI actions such as `perform_action`, `set_value`, and element clicks show in the pill without moving it, since no pointer moves); keycaps for `press_key` and a typing line for `type_text`; a glow along every screen edge; and a status pill such as *Claude is using your computer · click*. Pointer actions wait 350 ms so the cursor lands before the real input. While `screenshot` or `get_app_state` captures the screen, the overlay draws nothing, so agents never see it: not their own, not one shown for another agent, and not one that another agent starts mid-capture. This holds with `COMPUTER_USE_LINUX_INDICATOR=0` too, which only stops a server from showing its own actions. The capture waits for the compositor only when something was on screen. The agent name comes from the MCP client's `clientInfo`.
@@ -348,6 +391,38 @@ claude mcp list
348
391
 
349
392
  If `computer-use-linux` is not on `PATH`, pass the absolute path (e.g. `~/.local/bin/computer-use-linux`). Inside a Claude Code session, run `/mcp` to confirm the tools are loaded.
350
393
 
394
+ ### GitHub Copilot CLI
395
+
396
+ Install the binary first using one of the methods above, then register it with
397
+ Copilot CLI from your logged-in Linux desktop session:
398
+
399
+ ```bash
400
+ copilot mcp add computer-use-linux \
401
+ --env COMPUTER_USE_LINUX_ENABLE_SHELL=0 \
402
+ -- computer-use-linux mcp
403
+ copilot mcp get computer-use-linux
404
+ ```
405
+
406
+ This adds a user-scoped MCP entry; it does not install the backend or configure
407
+ desktop permissions. If the binary is not on `PATH`, use its absolute path
408
+ after `--`. Avoid registering a second controller if a plugin already supplies
409
+ this server.
410
+
411
+ Start a new Copilot session and use `/mcp` to confirm that the server's tools
412
+ are loaded. Ask for the `doctor` tool first, then test one non-sensitive app
413
+ with a specific target and visible-result check. Complete portal permission
414
+ dialogs yourself. A ready doctor report is not capture/control permission;
415
+ screenshots and app text can enter the model's context.
416
+
417
+ To remove only this registration:
418
+
419
+ ```bash
420
+ copilot mcp remove computer-use-linux
421
+ ```
422
+
423
+ These commands were checked with Copilot CLI 1.0.112. For other clients, use the
424
+ generic stdio configuration below.
425
+
351
426
  ### Claude Desktop
352
427
 
353
428
  Edit `~/.config/Claude/claude_desktop_config.json`:
@@ -506,6 +581,7 @@ Most setups need none of these: the server and installers detect backends and pa
506
581
  | `CU_DISABLE_ABS_POINTER` | Set exactly to `1` to disable the uinput absolute pointer. Wayland coordinate actions then need the RemoteDesktop portal; they refuse input rather than use ydotool's relative-motion approximation. Native X11 fallback is unchanged. |
507
582
  | `COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER` / `…_KEYBOARD` | Set exactly to `1` to prefer the RemoteDesktop portal on Wayland, skipping auto-detection. Pointer forcing skips the absolute uinput backend. Forced ydotool or XTEST keyboard overrides take precedence; KDE literal text keeps its clipboard route. Portal startup and text conversion failures return errors without replaying through ydotool. |
508
583
  | `COMPUTER_USE_LINUX_PORTAL_SCROLL_INVERT` | Set to `1` to reverse vertical scrolling through the RemoteDesktop portal. Defaults to normal direction on both GNOME and KDE Plasma; horizontal direction is unchanged. |
584
+ | `COMPUTER_USE_LINUX_DEVTOOLS` | The loopback DevTools port of a browser you started with `--remote-debugging-port`, or the absolute path of its `DevToolsActivePort` file. With it, `type_text` and `press_key` with `browser_tab` type into that browser's tabs without focusing it. The server connects only to 127.0.0.1 and never starts a browser. Unset, `browser_tab` calls are refused. See [Background keyboard delivery](#background-keyboard-delivery). |
509
585
  | `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND` | Set exactly to `1` on Hyprland to work on a window without focusing it. `type_text` and `press_key` calls that name a target window send keys to it through Hyprland's `send_key_state` dispatcher instead of focusing it and typing, and `get_app_state` and `screenshot` capture an unfocused target alone with `grim -T` instead of raising it or using the portal. The user's focus, cursor and workspace do not change, so the target can sit on a hidden workspace. Text is mapped for a US layout: other keymaps and characters outside printable ASCII, tab and newline are refused before any key is sent. Browser accelerators such as Ctrl+R are ignored by an unfocused Chromium. Unset, input focuses the target as before. |
510
586
  | `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP` | Set exactly to `1` to ask the RemoteDesktop portal to remember pointer and keyboard grants across processes. The first dialog still appears (on GNOME the remember box starts checked). Later processes reuse the single-use restore token from `Start`, stored mode `0600` under `$XDG_STATE_HOME/computer-use-linux/` (or `~/.local/state/computer-use-linux/`). Pointer and keyboard tokens are separate. Needs interface version 2. Unset, every new process is prompted. No effect when input is not using the portal. |
511
587
  | `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER` / `…_KEYBOARD` | Set exactly to `1` to always route pointer / keyboard through `ydotool`, skipping the portal and KDE clipboard paths; pointer forcing also skips the absolute uinput backend and native-X11 `xdotool` clicks, scrolls and drags. Coordinate pointer actions on Wayland then refuse input because ydotool cannot position them accurately; scrolling at the current pointer remains available. |
package/npm/README.md CHANGED
@@ -14,9 +14,11 @@ Provides:
14
14
  > `ToolAnnotations` so hosts can distinguish read-only observation from mutating
15
15
  > desktop actions.
16
16
 
17
- On labwc, window targeting uses the Selkies IPC patch's `labwc.sock` when it exists. Stock labwc and other wlroots compositors that advertise wlr-foreign-toplevel-management use that protocol instead, which reports no pids or bounds. Activation requires a unique app-id/title match. Window IDs include the title and must be refreshed after it changes. Ambiguous windows are rejected. Window bounds and relative coordinate targeting are unavailable until output scaling can be verified. Window-targeted screenshots, including `screenshot_after: "window"`, fail for the same reason; capture the full screen instead.
18
-
19
- Native X11 clicks, scrolls and drags use xdotool/XTEST on `DISPLAY`, ahead of uinput and portal input. Install xdotool for this route. Missing or failed commands return an error without a system-wide fallback. Drags send paced intermediate motion and always attempt release after a failed press or motion; verify the drop target afterward.
17
+ This page covers the npm install. Tool contracts, compositor notes, environment
18
+ variables and troubleshooting are in the
19
+ [repository README](https://github.com/agent-sh/computer-use-linux#readme),
20
+ which this package also ships as `README.md`. Agents should read
21
+ `skills/computer-use-linux/SKILL.md`.
20
22
 
21
23
  ## Quick start
22
24
 
@@ -40,23 +42,20 @@ Configure in any MCP client:
40
42
  }
41
43
  ```
42
44
 
45
+ If accessibility is disabled, run `computer-use-linux setup`.
46
+
43
47
  ### 2. Pi coding agent (native tools)
44
48
 
45
49
  ```bash
46
50
  pi install npm:@agent-sh/computer-use-linux
47
51
  ```
48
52
 
49
- When installed through Pi, the package supplies native, dynamically loaded
50
- `computer_use_linux_*` tools. When npm skipped the install script, the first
51
- real tool call downloads the binary. No separate MCP adapter or manual MCP
52
- configuration is required. Native tools require Pi 0.84.4 or newer; the
53
- standalone CLI wrapper retains Node.js 18 support. Each native tool call times
54
- out after 60 seconds; set `COMPUTER_USE_LINUX_TIMEOUT_MS` in Pi's environment to
55
- change it.
56
-
57
- In Pi codemode, native tools return `{ text, images, isError }`. Pass image
58
- blocks directly to `image(result.images[0])` and check `isError` before using
59
- a result.
53
+ Pi gets native `computer_use_linux_*` tools with no MCP configuration. They need
54
+ Pi 0.84.4 or newer; the standalone CLI wrapper keeps Node.js 18 support. Each
55
+ call times out after 60 seconds; set `COMPUTER_USE_LINUX_TIMEOUT_MS` in Pi's
56
+ environment to change it. See
57
+ [Pi Coding Agent](https://github.com/agent-sh/computer-use-linux#pi-coding-agent)
58
+ for codemode results.
60
59
 
61
60
  ### 3. Claude Code & Codex CLI plugins
62
61
 
@@ -71,8 +70,8 @@ codex plugin add computer-use-linux@computer-use-linux
71
70
  ```
72
71
 
73
72
  The plugin registers the MCP server and caches private release binaries on first
74
- start (it does not put `computer-use-linux` on `PATH`; also run step 1 if you
75
- want the shell CLI).
73
+ start. It does not put `computer-use-linux` on `PATH`; run step 1 as well if you
74
+ want the shell CLI.
76
75
 
77
76
  ### 4. Hermes Agent
78
77
 
@@ -82,22 +81,25 @@ hermes skills tap add agent-sh/computer-use-linux
82
81
  hermes skills install agent-sh/computer-use-linux/computer-use-linux
83
82
  hermes mcp add computer-use-linux --command computer-use-linux --args mcp
84
83
  hermes mcp test computer-use-linux
85
- hermes mcp configure computer-use-linux
86
84
  ```
87
85
 
88
- The generated Hermes config holds `command`, `args` and `enabled`, plus
89
- `tools.include` when you pick a subset of tools. Hermes defaults to a 300-second
90
- tool-call timeout and a 60-second connect timeout; add `timeout` or
91
- `connect_timeout` under the server to change them.
86
+ See [Hermes Agent](https://github.com/agent-sh/computer-use-linux#hermes-agent)
87
+ for tool subsets and timeouts.
88
+
89
+ ### 5. GitHub Copilot CLI
90
+
91
+ After step 1, register the server from your logged-in desktop session:
92
92
 
93
- ```yaml
94
- mcp_servers:
95
- computer-use-linux:
96
- command: computer-use-linux
97
- args: ["mcp"]
98
- enabled: true
93
+ ```bash
94
+ copilot mcp add computer-use-linux \
95
+ --env COMPUTER_USE_LINUX_ENABLE_SHELL=0 \
96
+ -- computer-use-linux mcp
97
+ copilot mcp get computer-use-linux
99
98
  ```
100
99
 
100
+ See [GitHub Copilot CLI](https://github.com/agent-sh/computer-use-linux#github-copilot-cli)
101
+ before first use.
102
+
101
103
  ## What the package installs
102
104
 
103
105
  The package downloads the matching Linux `x86_64` or `aarch64` binary from the
@@ -113,168 +115,3 @@ If you already built or installed the binary yourself, set
113
115
  `COMPUTER_USE_LINUX_BIN=/path/to/computer-use-linux` to make the wrapper use
114
116
  that executable instead. The Pi extension uses it only when it is an absolute
115
117
  path to an executable.
116
-
117
- ## Post-action observation & completion cue
118
-
119
- Mutating tools (`click`, `drag`, `scroll`, `press_key`, `type_text`,
120
- `perform_action`, `set_value`, `activate_window`, `move_window`, and
121
- `resize_window`) accept `screenshot_after: true | "screen" | "window"`,
122
- `format`/`quality`/`scale`/`max_width`/`max_height`/`max_bytes`, and
123
- `include_tree: true` to return a post-action screenshot and/or compacted AT-SPI
124
- tree in the same tool result.
125
-
126
- `screenshot`, `get_app_state` and `screenshot_after` also take
127
- `screenshot_file_path`, an absolute path where the image is saved instead of
128
- returned as image content. The parent directory must exist, the extension must
129
- match `format`, and the file must not exist: file output creates a new file
130
- and never replaces one. It is refused over HTTP (`mcp --listen`).
131
-
132
- For an optional MCP completion notification, set
133
- `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1` in the server environment and have the
134
- agent call `complete_interaction` after its desktop work. `notify-send` must be
135
- installed with an available desktop notification service. The cue is best effort
136
- and does not provide exclusive desktop ownership.
137
- This cue is available only to directly spawned MCP hosts; the native Pi extension
138
- does not yet forward the flag or include the tool in its catalog.
139
-
140
- ## Desktop setup & compositor notes
141
-
142
- - **Accessibility setup**: If accessibility is disabled, run `computer-use-linux setup`. Setup writes and
143
- reads back GNOME's `toolkit-accessibility` setting and warns if only runtime
144
- accessibility is available. Restart target apps if their trees remain empty.
145
- - **Foreground accessibility guard**: To explicitly keep that saved setting enabled during desktop automation, run
146
- `computer-use-linux guard-accessibility` in a foreground terminal. It registers
147
- a passive AT-SPI window-activation listener and reasserts the current user's
148
- GNOME `toolkit-accessibility` key after resets, verifying each write by readback.
149
- This affects other apps using the same user setting. MCP, setup, and
150
- `get_app_state` never start it automatically. Stop with Ctrl-C or SIGTERM before
151
- disabling accessibility; stopping removes its listener and ends writes without
152
- disabling other clients or restoring an old value. An app launched during a
153
- reset may still need restarting, so this is not a complete GNOME toggle fix.
154
- - **COSMIC Wayland**: Pointer coordinates use per-monitor RemoteDesktop portal streams. Approve the consent dialog before continuing. Available EIS connections wait for device readiness before the first input event. Window bounds support targeted screenshots and relative input, and window IDs fit JavaScript's exact integer range. The toplevel protocol does not expose PID.
155
- - **Window IDs**: Every backend's `window_id` fits JavaScript's exact integer range (at most 2^53-1), including KWin, so JSON clients can pass it back unchanged. A larger value is rejected with a prompt to re-run `list_windows`.
156
- - **Portal keyboard focus**: The receiving window or app target is preserved before consent and its focus is restored and verified afterward. If verification fails, no text or key chord is sent. Untargeted portal keyboard calls require working window introspection. GNOME Introspect can only verify an already focused app; refocus it manually after consent if needed.
157
- - **GNOME Wayland portal persistence**: On GNOME Wayland the remote-control portal asks again for every new server
158
- process. Set `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in that process's
159
- environment to reuse a restore token instead. The first dialog still appears.
160
- Leave it unset to keep the prompt. The repository README records where the
161
- token is stored and who can read it.
162
- - **niri**: When the host does not pass `NIRI_SOCKET`, the server reads it from its
163
- parent processes or the systemd user environment; without it, discovery needs
164
- an unambiguous socket matching `WAYLAND_DISPLAY`. Window listing and
165
- exact focus use `niri msg` with a direct IPC fallback. Missing positions or
166
- unknown/mixed output scaling leave window-relative coordinates unavailable;
167
- focus the target and inspect a fresh full-screen screenshot before using
168
- desktop coordinates.
169
- - **HTTP MCP**: `computer-use-linux mcp --listen ADDR:PORT` serves streamable
170
- HTTP at `/mcp`. It needs a token from the file in `COMPUTER_USE_LINUX_MCP_TOKEN_FILE`
171
- or from `COMPUTER_USE_LINUX_MCP_TOKEN`; clients send `Authorization: Bearer <token>`.
172
- Prefer the file: a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in
173
- `/proc/<pid>/environ` while the server runs.
174
- A bind that is not loopback needs `--allowed-host NAME` for each hostname clients
175
- use. Requests with an `Origin` header are refused, and `--listen` refuses to start
176
- with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, `get_tree_page` and calls that pick
177
- an element by index or selector must pass `snapshot_id`. The port is plain HTTP; put TLS in front
178
- (service mesh, ingress or sidecar). Idle sessions expire after an hour. A
179
- `404 Session not found` means initialize again; the cached tree and its
180
- `snapshot_id` survive re-initializing.
181
-
182
- ## Input & coordinate contracts
183
-
184
- - **Snapshot ids**: `get_app_state`, `include_tree` observations and
185
- `get_tree_page` return the `snapshot_id` of their tree. The server caches
186
- one tree, and each new tree replaces it. Pass `snapshot_id` with an
187
- `element_index` (or a selector, except on `scroll`) to `click`, `scroll`,
188
- `perform_action` or `set_value`, and to `get_tree_page`: a call naming a
189
- replaced snapshot is refused before any input. Without it, the element
190
- resolves in the latest tree. A full capture-space `x`/`y` point is not
191
- checked; a `coordinate_space: "atspi"` click is checked.
192
- - **Element targets**: `click`, `scroll`, `perform_action`, and `set_value`
193
- accept an optional app target (`pid`, `window_id`, `app_id`, `wm_class`,
194
- `window_title`). With one, an `element_index` (or a selector, where the
195
- tool takes one) from another app's `get_app_state` snapshot is refused. When the window backend reports no pid
196
- for the target window (i3 without `xprop`, COSMIC, other wlroots compositors),
197
- the snapshot must have been taken for that window. `perform_action` and
198
- `set_value` do not focus the window; an `element_identifier` is refused only
199
- when its bus owner is known and is not the target pid.
200
- - **Click vs. semantic actions**: Plain left `click` by element index or selector invokes the
201
- element's first AT-SPI action instead of using toolkit bounds when that action
202
- is `click`, `press`, or `toggle`, avoiding pointer-coordinate conversion. Explicit
203
- `activate`/`jump` requests belong in `perform_action`, not `click`, even when
204
- bounds are missing. Explicit
205
- `x`/`y`, right clicks, and multi-clicks retain pointer semantics. Explicit
206
- coordinates default to raw screenshot pixels (`coordinate_space: "capture"`).
207
- On KDE Wayland, `coordinate_space: "atspi"` with `element_index` or a semantic
208
- selector converts a global AT-SPI screen point through the element's KWin window,
209
- decoration, output origin, and display scale, even with a client-local AT-SPI
210
- window frame. It requires both `x`/`y` and element identity
211
- and cannot be combined with `relative: true`. For coordinate
212
- `click` or `scroll` with `relative: true`, use the clipped target-window
213
- screenshot crop origin and divide preview coordinates by screenshot `scale`.
214
- Do not use widget-local or raw GDK surface coordinates. A window target is
215
- required.
216
- - **Selkies containers**: set `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` so screenshots and input use the container's pixelflux computer-use server. No portal prompt. A closed port falls through; a server error does not.
217
- - **X11 layout risk**: xdotool/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian `br`/ABNT2) or changing keymap state, even with `ok: true`. Successful xdotool text dispatches include a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. This also detects a `setxkbmap` layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Inspect the actual text before pressing Enter, submitting it, or treating it as delivered. Use `include_tree: true` when AT-SPI exposes the text, or `screenshot_after: true`; neither automatically compares the field with the request. Do not retry blindly.
218
- - **X11 text input**: X11 text input sends LF and CRLF line endings as Return. Focused terminals
219
- accept keyboard input even when AT-SPI does not expose EditableText. Focus
220
- feedback describes the target; re-observe its contents to confirm input.
221
- - **Hyprland background mode**: Set `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` to type into and
222
- capture a targeted window without focusing it (`send_key_state` and `grim -T`), so the
223
- target can stay on a hidden workspace. Text is mapped for a US layout; other keymaps
224
- and non-ASCII characters are refused before any key is sent.
225
- - **Hyprland `type_text`**: On Hyprland, `type_text` passes one literal wtype argument after `--` for a
226
- classified nonsecret field, avoiding stdin keymap-refresh corruption in Electron.
227
- Argument text is visible in process listings, so confirmed password roles are
228
- rejected. Terminals and ambiguous roles (`Entry`/`Text`, `text box`, unreadable
229
- roles), or a missing/incomplete/timed-out probe, use private stdin for at most
230
- 100 Unicode characters per call. This includes terminals with AT-SPI support;
231
- they can still contain password prompts. Larger ambiguous-field text must be
232
- split into smaller calls or, for a nonterminal field, accessibility enabled.
233
- The input probe has its own 1.5-second budget; the 250 ms indicator probe only
234
- masks text. Both protections apply with the indicator disabled. Other compatible
235
- compositors retain their existing stdin path. Wtype rejects NUL and text over
236
- 131,071 UTF-8 bytes before focusing a window or displaying text. `ok: true`
237
- from a keyboard backend confirms dispatch only and leaves field contents
238
- unverified. Only the GNOME AT-SPI fallback reads the field back: its success
239
- is `ok: true` with a message that starts with "Typed through AT-SPI
240
- EditableText". Its failures have `ok: false` and start with "AT-SPI
241
- EditableText insert was not verified". Re-observe before retrying. Exact window focus alone does not suppress missing-focus
242
- warnings: only a complete scoped tree containing window/menu chrome without
243
- content makes the feedback informational. Such trees get a conditional
244
- Electron/Chromium hint computed before compaction; this is not positive Electron
245
- detection.
246
-
247
- Chromium can mark a `document web` focused alongside its contenteditable
248
- editor. The focus probe keeps searching that document and reports the editor;
249
- a document that holds focus itself still gets a warning, which no longer
250
- claims the text was lost. An editor can carry the `editable` state without the
251
- EditableText interface (`supports_editable_text: false`): `type_text` can
252
- reach it, `set_value` cannot. Compact nodes leave a false
253
- `supports_editable_text` out. Verify its text or a screenshot before
254
- submitting or retrying. Embedded browser documents can use up an app-scoped
255
- snapshot's node budget before the app's own controls; when
256
- `accessibility_tree_truncated` is true, a missing editor proves nothing. Raise
257
- `max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard
258
- cap 64); a busy renderer may give only a few thousand nodes before the
259
- deadline. A snapshot stops reading at 10 seconds
260
- and returns what it read, marked truncated; if the editor is still missing,
261
- hide the embedded browser pane for the snapshot.
262
-
263
- When a result would pass 32,000 bytes, the tree comes back as its leading
264
- nodes plus `accessibility_tree_spill` (`path`, `node_count`, `bytes`,
265
- `inline_node_count`). `path` is a private JSON file holding the whole tree
266
- with the same `element_index` values (under `$XDG_RUNTIME_DIR` when it is
267
- private, otherwise `~/.cache`). Search it instead of requesting the tree
268
- again; hosts that cannot read files page through it with `get_tree_page`. The
269
- file lasts until the next snapshot or until the server exits.
270
-
271
- Tree nodes use a compact encoding: null, empty, zero and false fields are
272
- left out, containers do not carry the U+FFFC placeholder text Chromium gives
273
- them, and `actions` lists action names. `verbose: true` returns the full
274
- encoding with action descriptions and keybindings. `interactive_only: true`
275
- returns only nodes with an action, editable text or a value, plus headings,
276
- status and live regions, windows and dialogs, with containers collapsed and
277
- the same `element_index` values as the full tree. When `title` or
278
- `window_id` names one of several windows under one AT-SPI application root,
279
- as in Chromium, the tree covers that window plus the app's popups and
280
- dialogs, and drops its other named windows.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@agent-sh/computer-use-linux",
3
- "version": "0.7.56",
3
+ "version": "0.7.58",
4
4
  "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
5
5
  "license": "MIT",
6
6
  "type": "commonjs",
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
7
7
  annotations: Record<string, unknown>;
8
8
  }
9
9
 
10
- export const GENERATED_SERVER_VERSION = "0.7.56";
11
- export const GENERATED_TOOL_CATALOG_HASH = "88fdbedb6fab2e9bde4236382de77f35d32542894a77d7ec3ff9a3fde53078ba";
12
- export const GENERATED_SHELL_TOOL_CATALOG_HASH = "dbc5fec64bafdd29cb7dc22464241e3583e59dbfffe052c02493fc9632b2ed8e";
10
+ export const GENERATED_SERVER_VERSION = "0.7.58";
11
+ export const GENERATED_TOOL_CATALOG_HASH = "e447f071f2ccf6c438c5fca9c80dbbe642661b1e9afa6239aad6534b2815ebfb";
12
+ export const GENERATED_SHELL_TOOL_CATALOG_HASH = "edaac05bdf1eaca9f3ae95f7e0e7d9f41797c920ec7331c39e6864c04841fa51";
13
13
  export const GENERATED_MCP_TOOLS =
14
14
  [
15
15
  {
@@ -214,6 +214,21 @@ export const GENERATED_MCP_TOOLS =
214
214
  ],
215
215
  "type": "string"
216
216
  },
217
+ "KeyboardDelivery": {
218
+ "description": "How type_text and press_key deliver input.",
219
+ "oneOf": [
220
+ {
221
+ "const": "focus",
222
+ "description": "Focus the target window, then type into it.",
223
+ "type": "string"
224
+ },
225
+ {
226
+ "const": "background",
227
+ "description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
228
+ "type": "string"
229
+ }
230
+ ]
231
+ },
217
232
  "ScreenshotOutputFormat": {
218
233
  "enum": [
219
234
  "png",
@@ -254,6 +269,18 @@ export const GENERATED_MCP_TOOLS =
254
269
  "default": "capture",
255
270
  "description": "Pixel space for explicit x/y: raw screenshot capture pixels (default),\nor global AT-SPI screen coordinates mapped through the element's KWin window."
256
271
  },
272
+ "delivery": {
273
+ "anyOf": [
274
+ {
275
+ "$ref": "#/$defs/KeyboardDelivery"
276
+ },
277
+ {
278
+ "type": "null"
279
+ }
280
+ ],
281
+ "default": null,
282
+ "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
283
+ },
257
284
  "element_index": {
258
285
  "default": null,
259
286
  "description": "Element index from the latest get_app_state or include_tree snapshot.",
@@ -474,9 +501,24 @@ export const GENERATED_MCP_TOOLS =
474
501
  "openWorldHint": true,
475
502
  "readOnlyHint": false
476
503
  },
477
- "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no window target or relative coordinates. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
504
+ "description": "Drag with the left button from one point to another in screenshot capture pixels. Takes no relative coordinates; a window target matters only with delivery=\"background\", which on GNOME Wayland drags with a virtual pen inside that window and leaves the user's pointer where it is (elsewhere the first such call returns a one-time notice, then later calls drag through the user's pointer). Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
478
505
  "inputSchema": {
479
506
  "$defs": {
507
+ "KeyboardDelivery": {
508
+ "description": "How type_text and press_key deliver input.",
509
+ "oneOf": [
510
+ {
511
+ "const": "focus",
512
+ "description": "Focus the target window, then type into it.",
513
+ "type": "string"
514
+ },
515
+ {
516
+ "const": "background",
517
+ "description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
518
+ "type": "string"
519
+ }
520
+ ]
521
+ },
480
522
  "ScreenshotOutputFormat": {
481
523
  "enum": [
482
524
  "png",
@@ -487,6 +529,26 @@ export const GENERATED_MCP_TOOLS =
487
529
  },
488
530
  "$schema": "https://json-schema.org/draft/2020-12/schema",
489
531
  "properties": {
532
+ "app_id": {
533
+ "default": null,
534
+ "description": "Application id, ignoring ASCII case: the full desktop id, or its last\ndot-separated segment.",
535
+ "type": [
536
+ "string",
537
+ "null"
538
+ ]
539
+ },
540
+ "delivery": {
541
+ "anyOf": [
542
+ {
543
+ "$ref": "#/$defs/KeyboardDelivery"
544
+ },
545
+ {
546
+ "type": "null"
547
+ }
548
+ ],
549
+ "default": null,
550
+ "description": "\"background\" drags with a virtual pen on GNOME Wayland, so the\nuser's pointer stays where it is; both points must be on the target\nwindow, with no window stacked above them. Elsewhere the first such\ncall in a server session sends nothing and returns a notice;\nrepeated, and for every later such call, the drag goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
551
+ },
490
552
  "end_x": {
491
553
  "description": "End x in raw screenshot capture pixels.",
492
554
  "format": "int32",
@@ -539,6 +601,15 @@ export const GENERATED_MCP_TOOLS =
539
601
  "null"
540
602
  ]
541
603
  },
604
+ "pid": {
605
+ "default": null,
606
+ "description": "Process id. It must match exactly one window.",
607
+ "minimum": 0,
608
+ "type": [
609
+ "integer",
610
+ "null"
611
+ ]
612
+ },
542
613
  "quality": {
543
614
  "description": "JPEG quality from 1 to 95 (default 80) for post-action screenshots. Ignored for png.",
544
615
  "maximum": 95,
@@ -590,6 +661,32 @@ export const GENERATED_MCP_TOOLS =
590
661
  "description": "Start y in raw screenshot capture pixels.",
591
662
  "format": "int32",
592
663
  "type": "integer"
664
+ },
665
+ "window_id": {
666
+ "default": null,
667
+ "description": "Window id from a recent list_windows: the window a background drag\nruns in. The first target field set decides, in this order:\nwindow_id, pid, app_id, wm_class, window_title. A drag with\n\"focus\" delivery ignores the target.",
668
+ "maximum": 9007199254740991,
669
+ "minimum": 0,
670
+ "type": [
671
+ "integer",
672
+ "null"
673
+ ]
674
+ },
675
+ "window_title": {
676
+ "default": null,
677
+ "description": "Window title substring, ignoring ASCII case.",
678
+ "type": [
679
+ "string",
680
+ "null"
681
+ ]
682
+ },
683
+ "wm_class": {
684
+ "default": null,
685
+ "description": "Window manager class, an exact match ignoring ASCII case.",
686
+ "type": [
687
+ "string",
688
+ "null"
689
+ ]
593
690
  }
594
691
  },
595
692
  "required": [
@@ -1317,9 +1414,24 @@ export const GENERATED_MCP_TOOLS =
1317
1414
  "openWorldHint": true,
1318
1415
  "readOnlyHint": false
1319
1416
  },
1320
- "description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). With PIXELFLUX_CU and PIXELFLUX_CU_TOKEN set, chords go to pixelflux. An unreachable pixelflux falls through to the routes below; a pixelflux error does not. On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
1417
+ "description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, kp_enter (keypad Enter), escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). With PIXELFLUX_CU and PIXELFLUX_CU_TOKEN set, chords go to pixelflux. An unreachable pixelflux falls through to the routes below; a pixelflux error does not. On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass delivery=\"background\" (the default when an element or browser_tab is named) to leave the user's keyboard focus alone: input goes to a named element through AT-SPI, to a browser tab over a DevTools port the user opened (COMPUTER_USE_LINUX_DEVTOOLS), or to a terminal's program through tmux; when no route serves the target (the reason says which setting or desktop would open one), the first such call in a server session sends nothing and returns a notice (background.retry_to_continue=true); repeating it, and every later such call, goes through normal focused input and reports background.used=false. The result's background field names the route used. On an element, only keys that mean an action map (Enter or space on a button or menu item, space on a check box, radio button or toggle, Enter on a link or a single-line field with an activate action); others have no route there. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
1321
1418
  "inputSchema": {
1322
1419
  "$defs": {
1420
+ "KeyboardDelivery": {
1421
+ "description": "How type_text and press_key deliver input.",
1422
+ "oneOf": [
1423
+ {
1424
+ "const": "focus",
1425
+ "description": "Focus the target window, then type into it.",
1426
+ "type": "string"
1427
+ },
1428
+ {
1429
+ "const": "background",
1430
+ "description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
1431
+ "type": "string"
1432
+ }
1433
+ ]
1434
+ },
1323
1435
  "ScreenshotOutputFormat": {
1324
1436
  "enum": [
1325
1437
  "png",
@@ -1338,6 +1450,43 @@ export const GENERATED_MCP_TOOLS =
1338
1450
  "null"
1339
1451
  ]
1340
1452
  },
1453
+ "browser_tab": {
1454
+ "default": null,
1455
+ "description": "Background delivery to a browser tab over DevTools: a target id, or a\ncase-insensitive substring of one page's URL or title. Needs\nCOMPUTER_USE_LINUX_DEVTOOLS (the loopback port of a browser the user\nstarted with --remote-debugging-port, or its DevToolsActivePort\nfile). Input goes to the tab's focused element.",
1456
+ "type": [
1457
+ "string",
1458
+ "null"
1459
+ ]
1460
+ },
1461
+ "delivery": {
1462
+ "anyOf": [
1463
+ {
1464
+ "$ref": "#/$defs/KeyboardDelivery"
1465
+ },
1466
+ {
1467
+ "type": "null"
1468
+ }
1469
+ ],
1470
+ "default": null,
1471
+ "description": "\"focus\" focuses the target window and types into it. \"background\"\nleaves the user's keyboard focus where it is and delivers through the\nroute the call names: an element (element_index, element_identifier\nor role/name/states), a browser_tab, or a terminal (tty, terminal_pid,\nterminal_command, terminal_cwd, or a terminal window) whose program\nruns in tmux. When no route serves the target, the first such call in\na server session sends nothing and returns a notice; repeated, and\nfor every later such call, the input goes through normal focused\ninput (an element gets keyboard focus first, a browser_tab targets\nthe one window whose title contains it). A wrong or ambiguous target is still an\nerror. Default: \"background\" when an element or browser_tab is named,\nelse \"focus\"."
1472
+ },
1473
+ "element_identifier": {
1474
+ "default": null,
1475
+ "description": "A node's object_ref from get_app_state or get_tree_page, for\nbackground delivery to that element. Wins over element_index and the\nselectors.",
1476
+ "type": [
1477
+ "string",
1478
+ "null"
1479
+ ]
1480
+ },
1481
+ "element_index": {
1482
+ "default": null,
1483
+ "description": "Background delivery to an element: its element_index from the latest\nget_app_state or include_tree snapshot. type_text inserts at its caret\n(or end) through AT-SPI EditableText and reads it back; press_key runs\nthe AT-SPI action the key means (Enter or space on a button or menu\nitem, space on a check box, radio button or toggle, Enter on a link\nor on a single-line field with an activate action).",
1484
+ "minimum": 0,
1485
+ "type": [
1486
+ "integer",
1487
+ "null"
1488
+ ]
1489
+ },
1341
1490
  "format": {
1342
1491
  "anyOf": [
1343
1492
  {
@@ -1384,6 +1533,14 @@ export const GENERATED_MCP_TOOLS =
1384
1533
  "null"
1385
1534
  ]
1386
1535
  },
1536
+ "name": {
1537
+ "default": null,
1538
+ "description": "Background delivery to the element whose accessible name contains\nthis, ignoring case.",
1539
+ "type": [
1540
+ "string",
1541
+ "null"
1542
+ ]
1543
+ },
1387
1544
  "pid": {
1388
1545
  "default": null,
1389
1546
  "description": "Process id. It must match exactly one window.",
@@ -1402,6 +1559,14 @@ export const GENERATED_MCP_TOOLS =
1402
1559
  "null"
1403
1560
  ]
1404
1561
  },
1562
+ "role": {
1563
+ "default": null,
1564
+ "description": "Background delivery to the element whose AT-SPI role contains this,\nignoring case (with name and states, it must match one element).",
1565
+ "type": [
1566
+ "string",
1567
+ "null"
1568
+ ]
1569
+ },
1405
1570
  "scale": {
1406
1571
  "description": "Additional downscale factor from 0.0 to 1.0, applied before max dimensions.",
1407
1572
  "format": "float",
@@ -1435,6 +1600,22 @@ export const GENERATED_MCP_TOOLS =
1435
1600
  "null"
1436
1601
  ]
1437
1602
  },
1603
+ "snapshot_id": {
1604
+ "default": null,
1605
+ "description": "snapshot_id of the result element_index or the selector comes from.\nIf a later snapshot replaced it, the call is refused before input.",
1606
+ "type": [
1607
+ "string",
1608
+ "null"
1609
+ ]
1610
+ },
1611
+ "states": {
1612
+ "default": [],
1613
+ "description": "AT-SPI states the background target element must all have.",
1614
+ "items": {
1615
+ "type": "string"
1616
+ },
1617
+ "type": "array"
1618
+ },
1438
1619
  "terminal_command": {
1439
1620
  "default": null,
1440
1621
  "description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
@@ -1870,6 +2051,21 @@ export const GENERATED_MCP_TOOLS =
1870
2051
  "description": "Scroll an element, window, or point in a direction by a number of pages (5 wheel notches per page). With a window target and no x/y/element_index, scrolls at the centre of the targeted window; with no point or target, at the current pointer. A window target is raised and focused first; nothing is sent when focus cannot be verified. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
1871
2052
  "inputSchema": {
1872
2053
  "$defs": {
2054
+ "KeyboardDelivery": {
2055
+ "description": "How type_text and press_key deliver input.",
2056
+ "oneOf": [
2057
+ {
2058
+ "const": "focus",
2059
+ "description": "Focus the target window, then type into it.",
2060
+ "type": "string"
2061
+ },
2062
+ {
2063
+ "const": "background",
2064
+ "description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
2065
+ "type": "string"
2066
+ }
2067
+ ]
2068
+ },
1873
2069
  "ScreenshotOutputFormat": {
1874
2070
  "enum": [
1875
2071
  "png",
@@ -1888,6 +2084,18 @@ export const GENERATED_MCP_TOOLS =
1888
2084
  "null"
1889
2085
  ]
1890
2086
  },
2087
+ "delivery": {
2088
+ "anyOf": [
2089
+ {
2090
+ "$ref": "#/$defs/KeyboardDelivery"
2091
+ },
2092
+ {
2093
+ "type": "null"
2094
+ }
2095
+ ],
2096
+ "default": null,
2097
+ "description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). On GNOME Wayland a virtual pen (a touch pan for\nscroll) leaves the user's pointer alone, but a click still focuses the\nwindow, and apps without tablet support ignore it. Needs x and y in\nscreenshot capture pixels over the window the call targets, without\nrelative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
2098
+ },
1891
2099
  "direction": {
1892
2100
  "description": "up, down, left or right (case-insensitive).",
1893
2101
  "type": "string"
@@ -2324,9 +2532,24 @@ export const GENERATED_MCP_TOOLS =
2324
2532
  "openWorldHint": true,
2325
2533
  "readOnlyHint": false
2326
2534
  },
2327
- "description": "Type literal text using keyboard input, optionally after focusing a target window or terminal selector. Keyboard success confirms dispatch, not field-content verification; re-observe the target before retrying. On GNOME, when a configured keyboard layout group cannot type a character of the text, the text goes into the focused editable field of the verified target window through AT-SPI EditableText instead: inserted at the caret, replacing a selection, and reported as ok only when the field reads it back with the rest unchanged. Password fields, text with a newline, tab or other control character, and focus outside an editable field of that active window keep the refusal. X11 xdotool can silently substitute symbols under non-US or changing XKB layouts; inspect the text before pressing Enter or submitting it. The wtype backend rejects NUL and text over 131071 UTF-8 bytes before focus or indicators. On Hyprland, confirmed password fields are rejected, and ambiguous fields and terminals use private stdin for at most 100 Unicode characters per call. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
2535
+ "description": "Type literal text using keyboard input, optionally after focusing a target window or terminal selector. Keyboard success confirms dispatch, not field-content verification; re-observe the target before retrying. On GNOME, when a configured keyboard layout group cannot type a character of the text, the text goes into the focused editable field of the verified target window through AT-SPI EditableText instead: inserted at the caret, replacing a selection, and reported as ok only when the field reads it back with the rest unchanged. Password fields, text with a newline, tab or other control character, and focus outside an editable field of that active window keep the refusal. Pass delivery=\"background\" (the default when an element or browser_tab is named) to leave the user's keyboard focus alone: input goes to a named element through AT-SPI, to a browser tab over a DevTools port the user opened (COMPUTER_USE_LINUX_DEVTOOLS), or to a terminal's program through tmux; when no route serves the target (the reason says which setting or desktop would open one), the first such call in a server session sends nothing and returns a notice (background.retry_to_continue=true); repeating it, and every later such call, goes through normal focused input and reports background.used=false. The result's background field names the route used. Into a named element the text goes in at its caret (or end) and must read back; a password field is filled unverified. X11 xdotool can silently substitute symbols under non-US or changing XKB layouts; inspect the text before pressing Enter or submitting it. The wtype backend rejects NUL and text over 131071 UTF-8 bytes before focus or indicators. On Hyprland, confirmed password fields are rejected, and ambiguous fields and terminals use private stdin for at most 100 Unicode characters per call. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
2328
2536
  "inputSchema": {
2329
2537
  "$defs": {
2538
+ "KeyboardDelivery": {
2539
+ "description": "How type_text and press_key deliver input.",
2540
+ "oneOf": [
2541
+ {
2542
+ "const": "focus",
2543
+ "description": "Focus the target window, then type into it.",
2544
+ "type": "string"
2545
+ },
2546
+ {
2547
+ "const": "background",
2548
+ "description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
2549
+ "type": "string"
2550
+ }
2551
+ ]
2552
+ },
2330
2553
  "ScreenshotOutputFormat": {
2331
2554
  "enum": [
2332
2555
  "png",
@@ -2345,6 +2568,43 @@ export const GENERATED_MCP_TOOLS =
2345
2568
  "null"
2346
2569
  ]
2347
2570
  },
2571
+ "browser_tab": {
2572
+ "default": null,
2573
+ "description": "Background delivery to a browser tab over DevTools: a target id, or a\ncase-insensitive substring of one page's URL or title. Needs\nCOMPUTER_USE_LINUX_DEVTOOLS (the loopback port of a browser the user\nstarted with --remote-debugging-port, or its DevToolsActivePort\nfile). Input goes to the tab's focused element.",
2574
+ "type": [
2575
+ "string",
2576
+ "null"
2577
+ ]
2578
+ },
2579
+ "delivery": {
2580
+ "anyOf": [
2581
+ {
2582
+ "$ref": "#/$defs/KeyboardDelivery"
2583
+ },
2584
+ {
2585
+ "type": "null"
2586
+ }
2587
+ ],
2588
+ "default": null,
2589
+ "description": "\"focus\" focuses the target window and types into it. \"background\"\nleaves the user's keyboard focus where it is and delivers through the\nroute the call names: an element (element_index, element_identifier\nor role/name/states), a browser_tab, or a terminal (tty, terminal_pid,\nterminal_command, terminal_cwd, or a terminal window) whose program\nruns in tmux. When no route serves the target, the first such call in\na server session sends nothing and returns a notice; repeated, and\nfor every later such call, the input goes through normal focused\ninput (an element gets keyboard focus first, a browser_tab targets\nthe one window whose title contains it). A wrong or ambiguous target is still an\nerror. Default: \"background\" when an element or browser_tab is named,\nelse \"focus\"."
2590
+ },
2591
+ "element_identifier": {
2592
+ "default": null,
2593
+ "description": "A node's object_ref from get_app_state or get_tree_page, for\nbackground delivery to that element. Wins over element_index and the\nselectors.",
2594
+ "type": [
2595
+ "string",
2596
+ "null"
2597
+ ]
2598
+ },
2599
+ "element_index": {
2600
+ "default": null,
2601
+ "description": "Background delivery to an element: its element_index from the latest\nget_app_state or include_tree snapshot. type_text inserts at its caret\n(or end) through AT-SPI EditableText and reads it back; press_key runs\nthe AT-SPI action the key means (Enter or space on a button or menu\nitem, space on a check box, radio button or toggle, Enter on a link\nor on a single-line field with an activate action).",
2602
+ "minimum": 0,
2603
+ "type": [
2604
+ "integer",
2605
+ "null"
2606
+ ]
2607
+ },
2348
2608
  "format": {
2349
2609
  "anyOf": [
2350
2610
  {
@@ -2387,6 +2647,14 @@ export const GENERATED_MCP_TOOLS =
2387
2647
  "null"
2388
2648
  ]
2389
2649
  },
2650
+ "name": {
2651
+ "default": null,
2652
+ "description": "Background delivery to the element whose accessible name contains\nthis, ignoring case.",
2653
+ "type": [
2654
+ "string",
2655
+ "null"
2656
+ ]
2657
+ },
2390
2658
  "pid": {
2391
2659
  "default": null,
2392
2660
  "description": "Process id. It must match exactly one window.",
@@ -2405,6 +2673,14 @@ export const GENERATED_MCP_TOOLS =
2405
2673
  "null"
2406
2674
  ]
2407
2675
  },
2676
+ "role": {
2677
+ "default": null,
2678
+ "description": "Background delivery to the element whose AT-SPI role contains this,\nignoring case (with name and states, it must match one element).",
2679
+ "type": [
2680
+ "string",
2681
+ "null"
2682
+ ]
2683
+ },
2408
2684
  "scale": {
2409
2685
  "description": "Additional downscale factor from 0.0 to 1.0, applied before max dimensions.",
2410
2686
  "format": "float",
@@ -2438,6 +2714,22 @@ export const GENERATED_MCP_TOOLS =
2438
2714
  "null"
2439
2715
  ]
2440
2716
  },
2717
+ "snapshot_id": {
2718
+ "default": null,
2719
+ "description": "snapshot_id of the result element_index or the selector comes from.\nIf a later snapshot replaced it, the call is refused before input.",
2720
+ "type": [
2721
+ "string",
2722
+ "null"
2723
+ ]
2724
+ },
2725
+ "states": {
2726
+ "default": [],
2727
+ "description": "AT-SPI states the background target element must all have.",
2728
+ "items": {
2729
+ "type": "string"
2730
+ },
2731
+ "type": "array"
2732
+ },
2441
2733
  "terminal_command": {
2442
2734
  "default": null,
2443
2735
  "description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
@@ -242,6 +242,7 @@ const FORWARDED_ENVIRONMENT = [
242
242
  "RUST_LOG",
243
243
  "COMPUTER_USE_LINUX_AGENT_NAME",
244
244
  "COMPUTER_USE_LINUX_COSMIC_HELPER",
245
+ "COMPUTER_USE_LINUX_DEVTOOLS",
245
246
  "COMPUTER_USE_LINUX_ENABLE_SHELL",
246
247
  "COMPUTER_USE_LINUX_FORCE_PORTAL_KEYBOARD",
247
248
  "COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER",
@@ -134,6 +134,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
134
134
 
135
135
  ### Host-specific guides
136
136
 
137
+ - [GitHub Copilot CLI setup](https://github.com/agent-sh/computer-use-linux#github-copilot-cli)
137
138
  - [Hermes setup](references/hermes-setup.md)
138
139
  - [Pi coding agent setup](references/pi-setup.md)
139
140
  - [Selkies container setup](references/selkies-setup.md)
@@ -235,6 +236,7 @@ directly to a window-relative click.
235
236
  - `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive.
236
237
  - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
237
238
  - On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
239
+ - To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. On GNOME Wayland, `click`, `drag` and `scroll` with `delivery: "background"`, a window target and points on that window use a virtual pen and touchscreen: the user's mouse stays put, but a click focuses the window, apps without tablet support ignore the pen (re-observe after the call), a drag stays in one window, a scroll is a touch pan, and hover lasts 8 s (`capabilities.background_pointer`). When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
238
240
  - Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
239
241
  - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.
240
242
  - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.