@agent-sh/computer-use-linux 0.7.56 → 0.7.57
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +66 -2
- package/npm/README.md +29 -192
- package/package.json +1 -1
- package/pi/extension/generated-tools.ts +227 -5
- package/pi/extension/index.ts +1 -0
- package/skills/computer-use-linux/SKILL.md +2 -0
package/README.md
CHANGED
|
@@ -76,8 +76,8 @@ The crate was extracted from [`codex-desktop-linux`](https://github.com/avifenes
|
|
|
76
76
|
| **Input** | `click` | Click by `element_index`, semantic selector (`role`/`name`/`text`/`states`), or desktop/window-relative coordinates (`left`/`right`/`middle`, `click_count`). |
|
|
77
77
|
| | `drag` | Desktop coordinate drag from `(start_x, start_y)` to `(end_x, end_y)`. |
|
|
78
78
|
| | `scroll` | Page-based vertical/horizontal scroll on an element, window, or pixel coordinate. |
|
|
79
|
-
| | `press_key` | Single keys or chords (`Ctrl+L`, `Alt+F4`, `Return`), optionally focusing a target window or terminal first. |
|
|
80
|
-
| | `type_text` | Literal text entry, optionally targeted at a window or terminal (`tty`, `terminal_pid`, `terminal_command`, `terminal_cwd`). |
|
|
79
|
+
| | `press_key` | Single keys or chords (`Ctrl+L`, `Alt+F4`, `Return`), optionally focusing a target window or terminal first, or with `delivery: "background"` to an element, browser tab or terminal without moving keyboard focus. |
|
|
80
|
+
| | `type_text` | Literal text entry, optionally targeted at a window or terminal (`tty`, `terminal_pid`, `terminal_command`, `terminal_cwd`), or with `delivery: "background"` to an element, browser tab or terminal without moving keyboard focus. |
|
|
81
81
|
| **Semantic actions** | `perform_action` | Invoke any AT-SPI action exposed by an element (`Press`, `Activate`, `Toggle`, `Jump`, …); defaults to the primary action. |
|
|
82
82
|
| | `set_value` | Write directly to a settable accessibility element (text fields, sliders, spinners). |
|
|
83
83
|
| **Window management** | `activate_window` | Focus a window by `window_id`, `pid`, `app_id`, `wm_class`, `title`, or terminal selectors. |
|
|
@@ -152,6 +152,37 @@ On X11, text input sends LF and CRLF line endings as Return, with one newline pe
|
|
|
152
152
|
|
|
153
153
|
On GNOME, a read-only keymap preflight rejects the whole string before input if a character is absent or requires a modifier level Mutter cannot synthesize. The background reader cannot establish the active layout group, so every configured group must support the text. This can reject text that works in one of several groups. Then the text goes into the focused editable field of the verified target window through AT-SPI EditableText: it is inserted at the caret, replaces a selection, and counts as typed (`ok: true`) only when the field reads it back with the rest of its text unchanged (in a field over 4096 characters, the 32 on each side of the insert). The field must be in the AT-SPI window that matches the target and that window must be active, so a field left focused in another window of the app is not used. A password field, focus outside an editable field, and text with a newline, tab or other control character keep the refusal; send keys with `press_key`, or fill the field with `set_value`. The check parses the keymap in pure Rust, with no system XKB library, and stops after a two-second keymap read deadline. It does not cover held modifiers or a keymap changed concurrently with typing.
|
|
154
154
|
|
|
155
|
+
### Background keyboard delivery
|
|
156
|
+
|
|
157
|
+
`type_text` and `press_key` take `delivery: "background"` to reach a target without moving the user's keyboard focus ([#480](https://github.com/agent-sh/computer-use-linux/issues/480)). The target picks the route. Naming an element or a `browser_tab` selects background delivery on its own, and `delivery: "focus"` with one is refused. The result's `background` field says what happened: `{"requested": true, "used": true, "route": "..."}` when a background route delivered it.
|
|
158
|
+
|
|
159
|
+
When no route serves the target because the desktop or app lacks one (below, "no route"), the call is not blocked. The first such call in a server session sends nothing and returns a notice: `ok: false` without `isError`, `background: {"requested": true, "used": false, "delivered": false, "reason": "...", "retry_to_continue": true}`, and a message with the reason (and the setting that would open a route, where there is one), that normal input moves the user's keyboard focus and pointer, and that [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) gives the agent its own session. Repeating the call sends it through normal focused input, as without `delivery`, and so does every later call with no route, with only `background: {"requested": true, "used": false}` in the result. For an element, focused input first gives the element keyboard focus with AT-SPI `GrabFocus` and then types into its window (the target fields, or the snapshot's window); for a `browser_tab`, it targets the one window whose title contains the tab's name (with window fields, that window must show it). With Hyprland background mode on, normal input to a window is the same sendkeystate route, so a call with no route there stays an error. A wrong, ambiguous or stale target (an unmatched or ambiguous element, tab, tty or window, a replaced snapshot, a terminal whose pty belongs to another devpts instance, a point outside the target window or screen), the X server's Ctrl+Alt+F-key and Ctrl+Alt+BackSpace chords on the agent seat, and a failure after input started stay errors.
|
|
160
|
+
|
|
161
|
+
- **Element (AT-SPI).** Pass `element_index` with `snapshot_id`, `element_identifier`, or `role`/`name`/`states`, plus the app target to check the snapshot against, as for `set_value`. `type_text` inserts through EditableText at the element's caret, or at its end when the caret is not in it, and succeeds only when the field reads the text back with the rest unchanged. A selection is left alone and the caret is not moved. A line feed or tab goes in only on a multi-line element; other control characters have no route. A password field takes the text but cannot be read back, and the result says so, as with `set_value`. `press_key` runs the AT-SPI action the key means: Enter or space on a button or menu item, space on a check box, radio button or toggle, Enter on a link, and Enter on a single-line field that has an `activate` action. Other keys and chords with modifiers have no route there.
|
|
162
|
+
- **Browser tab (DevTools).** Start Chrome, Chromium or an Electron app yourself with `--remote-debugging-port` and its own `--user-data-dir`, and set `COMPUTER_USE_LINUX_DEVTOOLS` to the port or to the path of its `DevToolsActivePort` file. The server connects only to that port on 127.0.0.1, only when its listener belongs to the same user, and never starts or restarts a browser. `browser_tab` names the tab: a target id, or text in one tab's URL or title. Text goes in with `Input.insertText` at the tab's focused element, as typing would, and an input or textarea must read it back. Keys go with `Input.dispatchKeyEvent`. The tab needs a focused field for text. Shortcuts the browser handles outside the page, such as Ctrl+T, do not run this way. A field inside a frame from another origin is refused.
|
|
163
|
+
- **Terminal.** Pass `tty`, `terminal_pid`, `terminal_command`, `terminal_cwd`, or a terminal window. The server finds the program on that tty and tries, in order: kitty remote control (the `KITTY_LISTEN_ON` socket kitty gives its shells; needs `allow_remote_control` and `listen_on` in kitty.conf), `wezterm cli`, tmux (a pane on the tty, or the active pane of a client attached on it), GNU screen (the current window of a client on the tty, when no other client may be attached to its session; a screen window's own tty is refused, since window numbers can be swapped), TIOCSTI where `dev.tty.legacy_tiocsti` is 1 and the server has CAP_SYS_ADMIN, and the terminal's pty master through `pidfd_getfd` where ptrace rules allow it. The result names the channel it used. Text reaches kitty, wezterm and tmux on stdin and screen through a private file and its `^_` register, which is cleared after the paste, never as a process argument. A tty taken from a window must have its pty master held by that window's terminal, so a container's pts with the same number is refused. When nothing works, there is no route, and the reason names the terminal found and the setting that would open one. GNOME Terminal, Ptyxis and other VTE terminals and Ghostty have no remote input command, and Konsole's D-Bus route is not supported yet, so a program in them needs tmux or screen. Writing to `/dev/pts/N` only prints on the screen, so the server never does it.
|
|
164
|
+
|
|
165
|
+
On Hyprland with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a window target with `delivery: "background"` goes through `send_key_state` as in background mode, and so does a terminal window that no terminal channel reaches.
|
|
166
|
+
|
|
167
|
+
- **Whole window (agent seat).** A window target that is not an element, a browser tab or a terminal some channel above reaches goes through the agent's own keyboard focus, on desktops that ship one. On X11 the server adds an XInput 2 master pointer and keyboard named `computer-use-linux agent <pid>`, focuses the target with `XISetFocus` for that keyboard only, and types through its XTEST device; the user's master keeps its focus and cursor. Keys come from the user's keymap (first two levels), so a character it lacks has no route, and neither does a key or chord another client grabs on the root window (a window manager shortcut such as openbox's Alt+F4), since through the agent seat the grab would act on the user's session. The check takes and drops a passive grab of each key back to back; a key the user presses in that instant goes nowhere. On sway 1.10 and later the server creates an `ext-transient-seat-v1` seat, focuses the target on it with `zwlr_foreign_toplevel_handle_v1.activate`, and types through a virtual keyboard with its own keymap; `seat0` keeps its focus and its workspace (a target on a workspace the user is not viewing still gets the keys), and sway removes the seat when the server disconnects. Xwayland windows have no route, because Xwayland feeds every seat into one X keyboard that follows the X focus, and neither do scratchpad windows, because activating one shows it on the user's workspace; if seat0's focus moves anyway, no keys are sent. Alt and Super chords, and chords the sway config binds (its included files too), have no route, because sway runs its bindings for the agent seat too. The agent keymap uses keycodes above any real keyboard's, so `bindcode` and `bindsym --to-code` bindings never match an agent key. On Hyprland the route is `sendkeystate`, as in Hyprland background mode below, without the environment variable. On X11 the master appears on first use and goes away after ten idle minutes or when the server exits. On sway the transient seat lives for one call (each call waits 100 ms after creating it, so clients can bind the new seat before input arrives; a client busy for longer misses the input), so no other program sees it between calls: wtype, for one, binds the last seat it sees, and a password or autotype helper using it would otherwise type into the agent's window. The X11 master's name carries a window owned by the server's X connection, so a master left by a killed server is removed by the next one. When input fails partway, the result says it may have been partly delivered and the seat is rebuilt.
|
|
168
|
+
|
|
169
|
+
`click` and `scroll` take `delivery: "background"` too, with `x` and `y` in screenshot capture pixels and a window target: the agent seat's pointer moves and clicks, and the user's pointer stays. The point must be over that window (on sway, on a visible workspace), or nothing is pressed. On X11 this needs a session with no window manager, because a window manager answers a click from any pointer by focusing that window for the user (openbox did); with one running, background clicks have no route and background keys still work. On sway the click also focuses the window on the agent seat only.
|
|
170
|
+
|
|
171
|
+
| Desktop | Agent seat | Why |
|
|
172
|
+
| --- | --- | --- |
|
|
173
|
+
| X11 (EWMH window managers) | Keys; pointer only without a window manager | XInput 2 multi-pointer is part of the X server. Checked with xterm, GTK 3 and Chrome under openbox, and the pointer without a window manager. |
|
|
174
|
+
| sway 1.10+ | Keys and pointer | [ext-transient-seat-v1](https://wayland.app/protocols/ext-transient-seat-v1) plus foreign-toplevel activation on a named seat ([sway view.c](https://github.com/swaywm/sway/blob/master/sway/tree/view.c)). GTK 3, GTK 4 and Chromium bind only the seats that exist when they start ([gdkdisplay-wayland.c](https://gitlab.gnome.org/GNOME/gtk/-/blob/gtk-3-24/gdk/wayland/gdkdisplay-wayland.c), [wayland_seat.cc](https://source.chromium.org/chromium/chromium/src/+/main:ui/ozone/platform/wayland/host/wayland_seat.cc)), so they never see the agent seat and have no route; foot and other clients that bind new seats get the keys. Xwayland windows have no route. Older sway has no transient seats. |
|
|
175
|
+
| Hyprland | Keys (`sendkeystate`, US layout) | One seat; `sendkeystate` is its per-window key dispatcher. No second pointer. |
|
|
176
|
+
| GNOME | None | One seat, and Mutter has no per-window input request. |
|
|
177
|
+
| KDE Plasma (KWin) | None | One `SeatInterface` and no per-window input request. |
|
|
178
|
+
| COSMIC | None | cosmic-comp creates one seat and has no transient-seat protocol. |
|
|
179
|
+
| niri | None | One seat and no transient-seat protocol. |
|
|
180
|
+
| labwc, river-classic, other wlroots compositors | None | One seat (labwc activates foreign toplevels on it; river-classic lists multiple seats as a TODO). |
|
|
181
|
+
|
|
182
|
+
Where there is no agent seat, a window-targeted background call has no route (the notice, then focused input); element, browser tab and terminal targets still work. For full isolation, with its own keyboard and pointer, run the app in an [agent-workspace-linux](https://github.com/agent-sh/agent-workspace-linux) workspace; the notice points there. `doctor` lists the routes this machine has in `capabilities.background_input` (`seat` when the agent seat is available) and the seat's route, pointer support or the reason it has none in `capabilities.background_seat`.
|
|
183
|
+
|
|
184
|
+
Some clients mishandle a second `wl_seat` (Zed, [zed-industries/zed#66735](https://github.com/zed-industries/zed/issues/66735)); that is one more reason the sway seat exists only during a call. A menu the agent opens on sway closes when its call ends, since the seat that opened it goes away. While sway is in a binding mode other than `default`, background keys have no route, since the mode's bindings would take them.
|
|
185
|
+
|
|
155
186
|
### On-screen indicator
|
|
156
187
|
|
|
157
188
|
While an agent acts, the server drives `computer-use-linux-indicator`, a click-through overlay that shows what it is doing: a software cursor tinted per agent that glides to each coordinate `click`, `drag`, or `scroll` target and ripples on click (AT-SPI actions such as `perform_action`, `set_value`, and element clicks show in the pill without moving it, since no pointer moves); keycaps for `press_key` and a typing line for `type_text`; a glow along every screen edge; and a status pill such as *Claude is using your computer · click*. Pointer actions wait 350 ms so the cursor lands before the real input. While `screenshot` or `get_app_state` captures the screen, the overlay draws nothing, so agents never see it: not their own, not one shown for another agent, and not one that another agent starts mid-capture. This holds with `COMPUTER_USE_LINUX_INDICATOR=0` too, which only stops a server from showing its own actions. The capture waits for the compositor only when something was on screen. The agent name comes from the MCP client's `clientInfo`.
|
|
@@ -348,6 +379,38 @@ claude mcp list
|
|
|
348
379
|
|
|
349
380
|
If `computer-use-linux` is not on `PATH`, pass the absolute path (e.g. `~/.local/bin/computer-use-linux`). Inside a Claude Code session, run `/mcp` to confirm the tools are loaded.
|
|
350
381
|
|
|
382
|
+
### GitHub Copilot CLI
|
|
383
|
+
|
|
384
|
+
Install the binary first using one of the methods above, then register it with
|
|
385
|
+
Copilot CLI from your logged-in Linux desktop session:
|
|
386
|
+
|
|
387
|
+
```bash
|
|
388
|
+
copilot mcp add computer-use-linux \
|
|
389
|
+
--env COMPUTER_USE_LINUX_ENABLE_SHELL=0 \
|
|
390
|
+
-- computer-use-linux mcp
|
|
391
|
+
copilot mcp get computer-use-linux
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
This adds a user-scoped MCP entry; it does not install the backend or configure
|
|
395
|
+
desktop permissions. If the binary is not on `PATH`, use its absolute path
|
|
396
|
+
after `--`. Avoid registering a second controller if a plugin already supplies
|
|
397
|
+
this server.
|
|
398
|
+
|
|
399
|
+
Start a new Copilot session and use `/mcp` to confirm that the server's tools
|
|
400
|
+
are loaded. Ask for the `doctor` tool first, then test one non-sensitive app
|
|
401
|
+
with a specific target and visible-result check. Complete portal permission
|
|
402
|
+
dialogs yourself. A ready doctor report is not capture/control permission;
|
|
403
|
+
screenshots and app text can enter the model's context.
|
|
404
|
+
|
|
405
|
+
To remove only this registration:
|
|
406
|
+
|
|
407
|
+
```bash
|
|
408
|
+
copilot mcp remove computer-use-linux
|
|
409
|
+
```
|
|
410
|
+
|
|
411
|
+
These commands were checked with Copilot CLI 1.0.112. For other clients, use the
|
|
412
|
+
generic stdio configuration below.
|
|
413
|
+
|
|
351
414
|
### Claude Desktop
|
|
352
415
|
|
|
353
416
|
Edit `~/.config/Claude/claude_desktop_config.json`:
|
|
@@ -506,6 +569,7 @@ Most setups need none of these: the server and installers detect backends and pa
|
|
|
506
569
|
| `CU_DISABLE_ABS_POINTER` | Set exactly to `1` to disable the uinput absolute pointer. Wayland coordinate actions then need the RemoteDesktop portal; they refuse input rather than use ydotool's relative-motion approximation. Native X11 fallback is unchanged. |
|
|
507
570
|
| `COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER` / `…_KEYBOARD` | Set exactly to `1` to prefer the RemoteDesktop portal on Wayland, skipping auto-detection. Pointer forcing skips the absolute uinput backend. Forced ydotool or XTEST keyboard overrides take precedence; KDE literal text keeps its clipboard route. Portal startup and text conversion failures return errors without replaying through ydotool. |
|
|
508
571
|
| `COMPUTER_USE_LINUX_PORTAL_SCROLL_INVERT` | Set to `1` to reverse vertical scrolling through the RemoteDesktop portal. Defaults to normal direction on both GNOME and KDE Plasma; horizontal direction is unchanged. |
|
|
572
|
+
| `COMPUTER_USE_LINUX_DEVTOOLS` | The loopback DevTools port of a browser you started with `--remote-debugging-port`, or the absolute path of its `DevToolsActivePort` file. With it, `type_text` and `press_key` with `browser_tab` type into that browser's tabs without focusing it. The server connects only to 127.0.0.1 and never starts a browser. Unset, `browser_tab` calls are refused. See [Background keyboard delivery](#background-keyboard-delivery). |
|
|
509
573
|
| `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND` | Set exactly to `1` on Hyprland to work on a window without focusing it. `type_text` and `press_key` calls that name a target window send keys to it through Hyprland's `send_key_state` dispatcher instead of focusing it and typing, and `get_app_state` and `screenshot` capture an unfocused target alone with `grim -T` instead of raising it or using the portal. The user's focus, cursor and workspace do not change, so the target can sit on a hidden workspace. Text is mapped for a US layout: other keymaps and characters outside printable ASCII, tab and newline are refused before any key is sent. Browser accelerators such as Ctrl+R are ignored by an unfocused Chromium. Unset, input focuses the target as before. |
|
|
510
574
|
| `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP` | Set exactly to `1` to ask the RemoteDesktop portal to remember pointer and keyboard grants across processes. The first dialog still appears (on GNOME the remember box starts checked). Later processes reuse the single-use restore token from `Start`, stored mode `0600` under `$XDG_STATE_HOME/computer-use-linux/` (or `~/.local/state/computer-use-linux/`). Pointer and keyboard tokens are separate. Needs interface version 2. Unset, every new process is prompted. No effect when input is not using the portal. |
|
|
511
575
|
| `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER` / `…_KEYBOARD` | Set exactly to `1` to always route pointer / keyboard through `ydotool`, skipping the portal and KDE clipboard paths; pointer forcing also skips the absolute uinput backend and native-X11 `xdotool` clicks, scrolls and drags. Coordinate pointer actions on Wayland then refuse input because ydotool cannot position them accurately; scrolling at the current pointer remains available. |
|
package/npm/README.md
CHANGED
|
@@ -14,9 +14,11 @@ Provides:
|
|
|
14
14
|
> `ToolAnnotations` so hosts can distinguish read-only observation from mutating
|
|
15
15
|
> desktop actions.
|
|
16
16
|
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
17
|
+
This page covers the npm install. Tool contracts, compositor notes, environment
|
|
18
|
+
variables and troubleshooting are in the
|
|
19
|
+
[repository README](https://github.com/agent-sh/computer-use-linux#readme),
|
|
20
|
+
which this package also ships as `README.md`. Agents should read
|
|
21
|
+
`skills/computer-use-linux/SKILL.md`.
|
|
20
22
|
|
|
21
23
|
## Quick start
|
|
22
24
|
|
|
@@ -40,23 +42,20 @@ Configure in any MCP client:
|
|
|
40
42
|
}
|
|
41
43
|
```
|
|
42
44
|
|
|
45
|
+
If accessibility is disabled, run `computer-use-linux setup`.
|
|
46
|
+
|
|
43
47
|
### 2. Pi coding agent (native tools)
|
|
44
48
|
|
|
45
49
|
```bash
|
|
46
50
|
pi install npm:@agent-sh/computer-use-linux
|
|
47
51
|
```
|
|
48
52
|
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
change it.
|
|
56
|
-
|
|
57
|
-
In Pi codemode, native tools return `{ text, images, isError }`. Pass image
|
|
58
|
-
blocks directly to `image(result.images[0])` and check `isError` before using
|
|
59
|
-
a result.
|
|
53
|
+
Pi gets native `computer_use_linux_*` tools with no MCP configuration. They need
|
|
54
|
+
Pi 0.84.4 or newer; the standalone CLI wrapper keeps Node.js 18 support. Each
|
|
55
|
+
call times out after 60 seconds; set `COMPUTER_USE_LINUX_TIMEOUT_MS` in Pi's
|
|
56
|
+
environment to change it. See
|
|
57
|
+
[Pi Coding Agent](https://github.com/agent-sh/computer-use-linux#pi-coding-agent)
|
|
58
|
+
for codemode results.
|
|
60
59
|
|
|
61
60
|
### 3. Claude Code & Codex CLI plugins
|
|
62
61
|
|
|
@@ -71,8 +70,8 @@ codex plugin add computer-use-linux@computer-use-linux
|
|
|
71
70
|
```
|
|
72
71
|
|
|
73
72
|
The plugin registers the MCP server and caches private release binaries on first
|
|
74
|
-
start
|
|
75
|
-
want the shell CLI
|
|
73
|
+
start. It does not put `computer-use-linux` on `PATH`; run step 1 as well if you
|
|
74
|
+
want the shell CLI.
|
|
76
75
|
|
|
77
76
|
### 4. Hermes Agent
|
|
78
77
|
|
|
@@ -82,22 +81,25 @@ hermes skills tap add agent-sh/computer-use-linux
|
|
|
82
81
|
hermes skills install agent-sh/computer-use-linux/computer-use-linux
|
|
83
82
|
hermes mcp add computer-use-linux --command computer-use-linux --args mcp
|
|
84
83
|
hermes mcp test computer-use-linux
|
|
85
|
-
hermes mcp configure computer-use-linux
|
|
86
84
|
```
|
|
87
85
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
86
|
+
See [Hermes Agent](https://github.com/agent-sh/computer-use-linux#hermes-agent)
|
|
87
|
+
for tool subsets and timeouts.
|
|
88
|
+
|
|
89
|
+
### 5. GitHub Copilot CLI
|
|
90
|
+
|
|
91
|
+
After step 1, register the server from your logged-in desktop session:
|
|
92
92
|
|
|
93
|
-
```
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
enabled: true
|
|
93
|
+
```bash
|
|
94
|
+
copilot mcp add computer-use-linux \
|
|
95
|
+
--env COMPUTER_USE_LINUX_ENABLE_SHELL=0 \
|
|
96
|
+
-- computer-use-linux mcp
|
|
97
|
+
copilot mcp get computer-use-linux
|
|
99
98
|
```
|
|
100
99
|
|
|
100
|
+
See [GitHub Copilot CLI](https://github.com/agent-sh/computer-use-linux#github-copilot-cli)
|
|
101
|
+
before first use.
|
|
102
|
+
|
|
101
103
|
## What the package installs
|
|
102
104
|
|
|
103
105
|
The package downloads the matching Linux `x86_64` or `aarch64` binary from the
|
|
@@ -113,168 +115,3 @@ If you already built or installed the binary yourself, set
|
|
|
113
115
|
`COMPUTER_USE_LINUX_BIN=/path/to/computer-use-linux` to make the wrapper use
|
|
114
116
|
that executable instead. The Pi extension uses it only when it is an absolute
|
|
115
117
|
path to an executable.
|
|
116
|
-
|
|
117
|
-
## Post-action observation & completion cue
|
|
118
|
-
|
|
119
|
-
Mutating tools (`click`, `drag`, `scroll`, `press_key`, `type_text`,
|
|
120
|
-
`perform_action`, `set_value`, `activate_window`, `move_window`, and
|
|
121
|
-
`resize_window`) accept `screenshot_after: true | "screen" | "window"`,
|
|
122
|
-
`format`/`quality`/`scale`/`max_width`/`max_height`/`max_bytes`, and
|
|
123
|
-
`include_tree: true` to return a post-action screenshot and/or compacted AT-SPI
|
|
124
|
-
tree in the same tool result.
|
|
125
|
-
|
|
126
|
-
`screenshot`, `get_app_state` and `screenshot_after` also take
|
|
127
|
-
`screenshot_file_path`, an absolute path where the image is saved instead of
|
|
128
|
-
returned as image content. The parent directory must exist, the extension must
|
|
129
|
-
match `format`, and the file must not exist: file output creates a new file
|
|
130
|
-
and never replaces one. It is refused over HTTP (`mcp --listen`).
|
|
131
|
-
|
|
132
|
-
For an optional MCP completion notification, set
|
|
133
|
-
`COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1` in the server environment and have the
|
|
134
|
-
agent call `complete_interaction` after its desktop work. `notify-send` must be
|
|
135
|
-
installed with an available desktop notification service. The cue is best effort
|
|
136
|
-
and does not provide exclusive desktop ownership.
|
|
137
|
-
This cue is available only to directly spawned MCP hosts; the native Pi extension
|
|
138
|
-
does not yet forward the flag or include the tool in its catalog.
|
|
139
|
-
|
|
140
|
-
## Desktop setup & compositor notes
|
|
141
|
-
|
|
142
|
-
- **Accessibility setup**: If accessibility is disabled, run `computer-use-linux setup`. Setup writes and
|
|
143
|
-
reads back GNOME's `toolkit-accessibility` setting and warns if only runtime
|
|
144
|
-
accessibility is available. Restart target apps if their trees remain empty.
|
|
145
|
-
- **Foreground accessibility guard**: To explicitly keep that saved setting enabled during desktop automation, run
|
|
146
|
-
`computer-use-linux guard-accessibility` in a foreground terminal. It registers
|
|
147
|
-
a passive AT-SPI window-activation listener and reasserts the current user's
|
|
148
|
-
GNOME `toolkit-accessibility` key after resets, verifying each write by readback.
|
|
149
|
-
This affects other apps using the same user setting. MCP, setup, and
|
|
150
|
-
`get_app_state` never start it automatically. Stop with Ctrl-C or SIGTERM before
|
|
151
|
-
disabling accessibility; stopping removes its listener and ends writes without
|
|
152
|
-
disabling other clients or restoring an old value. An app launched during a
|
|
153
|
-
reset may still need restarting, so this is not a complete GNOME toggle fix.
|
|
154
|
-
- **COSMIC Wayland**: Pointer coordinates use per-monitor RemoteDesktop portal streams. Approve the consent dialog before continuing. Available EIS connections wait for device readiness before the first input event. Window bounds support targeted screenshots and relative input, and window IDs fit JavaScript's exact integer range. The toplevel protocol does not expose PID.
|
|
155
|
-
- **Window IDs**: Every backend's `window_id` fits JavaScript's exact integer range (at most 2^53-1), including KWin, so JSON clients can pass it back unchanged. A larger value is rejected with a prompt to re-run `list_windows`.
|
|
156
|
-
- **Portal keyboard focus**: The receiving window or app target is preserved before consent and its focus is restored and verified afterward. If verification fails, no text or key chord is sent. Untargeted portal keyboard calls require working window introspection. GNOME Introspect can only verify an already focused app; refocus it manually after consent if needed.
|
|
157
|
-
- **GNOME Wayland portal persistence**: On GNOME Wayland the remote-control portal asks again for every new server
|
|
158
|
-
process. Set `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in that process's
|
|
159
|
-
environment to reuse a restore token instead. The first dialog still appears.
|
|
160
|
-
Leave it unset to keep the prompt. The repository README records where the
|
|
161
|
-
token is stored and who can read it.
|
|
162
|
-
- **niri**: When the host does not pass `NIRI_SOCKET`, the server reads it from its
|
|
163
|
-
parent processes or the systemd user environment; without it, discovery needs
|
|
164
|
-
an unambiguous socket matching `WAYLAND_DISPLAY`. Window listing and
|
|
165
|
-
exact focus use `niri msg` with a direct IPC fallback. Missing positions or
|
|
166
|
-
unknown/mixed output scaling leave window-relative coordinates unavailable;
|
|
167
|
-
focus the target and inspect a fresh full-screen screenshot before using
|
|
168
|
-
desktop coordinates.
|
|
169
|
-
- **HTTP MCP**: `computer-use-linux mcp --listen ADDR:PORT` serves streamable
|
|
170
|
-
HTTP at `/mcp`. It needs a token from the file in `COMPUTER_USE_LINUX_MCP_TOKEN_FILE`
|
|
171
|
-
or from `COMPUTER_USE_LINUX_MCP_TOKEN`; clients send `Authorization: Bearer <token>`.
|
|
172
|
-
Prefer the file: a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in
|
|
173
|
-
`/proc/<pid>/environ` while the server runs.
|
|
174
|
-
A bind that is not loopback needs `--allowed-host NAME` for each hostname clients
|
|
175
|
-
use. Requests with an `Origin` header are refused, and `--listen` refuses to start
|
|
176
|
-
with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, `get_tree_page` and calls that pick
|
|
177
|
-
an element by index or selector must pass `snapshot_id`. The port is plain HTTP; put TLS in front
|
|
178
|
-
(service mesh, ingress or sidecar). Idle sessions expire after an hour. A
|
|
179
|
-
`404 Session not found` means initialize again; the cached tree and its
|
|
180
|
-
`snapshot_id` survive re-initializing.
|
|
181
|
-
|
|
182
|
-
## Input & coordinate contracts
|
|
183
|
-
|
|
184
|
-
- **Snapshot ids**: `get_app_state`, `include_tree` observations and
|
|
185
|
-
`get_tree_page` return the `snapshot_id` of their tree. The server caches
|
|
186
|
-
one tree, and each new tree replaces it. Pass `snapshot_id` with an
|
|
187
|
-
`element_index` (or a selector, except on `scroll`) to `click`, `scroll`,
|
|
188
|
-
`perform_action` or `set_value`, and to `get_tree_page`: a call naming a
|
|
189
|
-
replaced snapshot is refused before any input. Without it, the element
|
|
190
|
-
resolves in the latest tree. A full capture-space `x`/`y` point is not
|
|
191
|
-
checked; a `coordinate_space: "atspi"` click is checked.
|
|
192
|
-
- **Element targets**: `click`, `scroll`, `perform_action`, and `set_value`
|
|
193
|
-
accept an optional app target (`pid`, `window_id`, `app_id`, `wm_class`,
|
|
194
|
-
`window_title`). With one, an `element_index` (or a selector, where the
|
|
195
|
-
tool takes one) from another app's `get_app_state` snapshot is refused. When the window backend reports no pid
|
|
196
|
-
for the target window (i3 without `xprop`, COSMIC, other wlroots compositors),
|
|
197
|
-
the snapshot must have been taken for that window. `perform_action` and
|
|
198
|
-
`set_value` do not focus the window; an `element_identifier` is refused only
|
|
199
|
-
when its bus owner is known and is not the target pid.
|
|
200
|
-
- **Click vs. semantic actions**: Plain left `click` by element index or selector invokes the
|
|
201
|
-
element's first AT-SPI action instead of using toolkit bounds when that action
|
|
202
|
-
is `click`, `press`, or `toggle`, avoiding pointer-coordinate conversion. Explicit
|
|
203
|
-
`activate`/`jump` requests belong in `perform_action`, not `click`, even when
|
|
204
|
-
bounds are missing. Explicit
|
|
205
|
-
`x`/`y`, right clicks, and multi-clicks retain pointer semantics. Explicit
|
|
206
|
-
coordinates default to raw screenshot pixels (`coordinate_space: "capture"`).
|
|
207
|
-
On KDE Wayland, `coordinate_space: "atspi"` with `element_index` or a semantic
|
|
208
|
-
selector converts a global AT-SPI screen point through the element's KWin window,
|
|
209
|
-
decoration, output origin, and display scale, even with a client-local AT-SPI
|
|
210
|
-
window frame. It requires both `x`/`y` and element identity
|
|
211
|
-
and cannot be combined with `relative: true`. For coordinate
|
|
212
|
-
`click` or `scroll` with `relative: true`, use the clipped target-window
|
|
213
|
-
screenshot crop origin and divide preview coordinates by screenshot `scale`.
|
|
214
|
-
Do not use widget-local or raw GDK surface coordinates. A window target is
|
|
215
|
-
required.
|
|
216
|
-
- **Selkies containers**: set `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` so screenshots and input use the container's pixelflux computer-use server. No portal prompt. A closed port falls through; a server error does not.
|
|
217
|
-
- **X11 layout risk**: xdotool/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian `br`/ABNT2) or changing keymap state, even with `ok: true`. Successful xdotool text dispatches include a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. This also detects a `setxkbmap` layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Inspect the actual text before pressing Enter, submitting it, or treating it as delivered. Use `include_tree: true` when AT-SPI exposes the text, or `screenshot_after: true`; neither automatically compares the field with the request. Do not retry blindly.
|
|
218
|
-
- **X11 text input**: X11 text input sends LF and CRLF line endings as Return. Focused terminals
|
|
219
|
-
accept keyboard input even when AT-SPI does not expose EditableText. Focus
|
|
220
|
-
feedback describes the target; re-observe its contents to confirm input.
|
|
221
|
-
- **Hyprland background mode**: Set `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` to type into and
|
|
222
|
-
capture a targeted window without focusing it (`send_key_state` and `grim -T`), so the
|
|
223
|
-
target can stay on a hidden workspace. Text is mapped for a US layout; other keymaps
|
|
224
|
-
and non-ASCII characters are refused before any key is sent.
|
|
225
|
-
- **Hyprland `type_text`**: On Hyprland, `type_text` passes one literal wtype argument after `--` for a
|
|
226
|
-
classified nonsecret field, avoiding stdin keymap-refresh corruption in Electron.
|
|
227
|
-
Argument text is visible in process listings, so confirmed password roles are
|
|
228
|
-
rejected. Terminals and ambiguous roles (`Entry`/`Text`, `text box`, unreadable
|
|
229
|
-
roles), or a missing/incomplete/timed-out probe, use private stdin for at most
|
|
230
|
-
100 Unicode characters per call. This includes terminals with AT-SPI support;
|
|
231
|
-
they can still contain password prompts. Larger ambiguous-field text must be
|
|
232
|
-
split into smaller calls or, for a nonterminal field, accessibility enabled.
|
|
233
|
-
The input probe has its own 1.5-second budget; the 250 ms indicator probe only
|
|
234
|
-
masks text. Both protections apply with the indicator disabled. Other compatible
|
|
235
|
-
compositors retain their existing stdin path. Wtype rejects NUL and text over
|
|
236
|
-
131,071 UTF-8 bytes before focusing a window or displaying text. `ok: true`
|
|
237
|
-
from a keyboard backend confirms dispatch only and leaves field contents
|
|
238
|
-
unverified. Only the GNOME AT-SPI fallback reads the field back: its success
|
|
239
|
-
is `ok: true` with a message that starts with "Typed through AT-SPI
|
|
240
|
-
EditableText". Its failures have `ok: false` and start with "AT-SPI
|
|
241
|
-
EditableText insert was not verified". Re-observe before retrying. Exact window focus alone does not suppress missing-focus
|
|
242
|
-
warnings: only a complete scoped tree containing window/menu chrome without
|
|
243
|
-
content makes the feedback informational. Such trees get a conditional
|
|
244
|
-
Electron/Chromium hint computed before compaction; this is not positive Electron
|
|
245
|
-
detection.
|
|
246
|
-
|
|
247
|
-
Chromium can mark a `document web` focused alongside its contenteditable
|
|
248
|
-
editor. The focus probe keeps searching that document and reports the editor;
|
|
249
|
-
a document that holds focus itself still gets a warning, which no longer
|
|
250
|
-
claims the text was lost. An editor can carry the `editable` state without the
|
|
251
|
-
EditableText interface (`supports_editable_text: false`): `type_text` can
|
|
252
|
-
reach it, `set_value` cannot. Compact nodes leave a false
|
|
253
|
-
`supports_editable_text` out. Verify its text or a screenshot before
|
|
254
|
-
submitting or retrying. Embedded browser documents can use up an app-scoped
|
|
255
|
-
snapshot's node budget before the app's own controls; when
|
|
256
|
-
`accessibility_tree_truncated` is true, a missing editor proves nothing. Raise
|
|
257
|
-
`max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard
|
|
258
|
-
cap 64); a busy renderer may give only a few thousand nodes before the
|
|
259
|
-
deadline. A snapshot stops reading at 10 seconds
|
|
260
|
-
and returns what it read, marked truncated; if the editor is still missing,
|
|
261
|
-
hide the embedded browser pane for the snapshot.
|
|
262
|
-
|
|
263
|
-
When a result would pass 32,000 bytes, the tree comes back as its leading
|
|
264
|
-
nodes plus `accessibility_tree_spill` (`path`, `node_count`, `bytes`,
|
|
265
|
-
`inline_node_count`). `path` is a private JSON file holding the whole tree
|
|
266
|
-
with the same `element_index` values (under `$XDG_RUNTIME_DIR` when it is
|
|
267
|
-
private, otherwise `~/.cache`). Search it instead of requesting the tree
|
|
268
|
-
again; hosts that cannot read files page through it with `get_tree_page`. The
|
|
269
|
-
file lasts until the next snapshot or until the server exits.
|
|
270
|
-
|
|
271
|
-
Tree nodes use a compact encoding: null, empty, zero and false fields are
|
|
272
|
-
left out, containers do not carry the U+FFFC placeholder text Chromium gives
|
|
273
|
-
them, and `actions` lists action names. `verbose: true` returns the full
|
|
274
|
-
encoding with action descriptions and keybindings. `interactive_only: true`
|
|
275
|
-
returns only nodes with an action, editable text or a value, plus headings,
|
|
276
|
-
status and live regions, windows and dialogs, with containers collapsed and
|
|
277
|
-
the same `element_index` values as the full tree. When `title` or
|
|
278
|
-
`window_id` names one of several windows under one AT-SPI application root,
|
|
279
|
-
as in Chromium, the tree covers that window plus the app's popups and
|
|
280
|
-
dialogs, and drops its other named windows.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.57",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
11
|
-
export const GENERATED_TOOL_CATALOG_HASH = "
|
|
12
|
-
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.57";
|
|
11
|
+
export const GENERATED_TOOL_CATALOG_HASH = "2bb4a324370516666dd67bc2c676007de98ebfb3c8179796578e51eaa4275728";
|
|
12
|
+
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "dfc420c6297a2f61f92e6379083c558cab420d1fdbafccc00e7f8390adef847b";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|
|
14
14
|
[
|
|
15
15
|
{
|
|
@@ -214,6 +214,21 @@ export const GENERATED_MCP_TOOLS =
|
|
|
214
214
|
],
|
|
215
215
|
"type": "string"
|
|
216
216
|
},
|
|
217
|
+
"KeyboardDelivery": {
|
|
218
|
+
"description": "How type_text and press_key deliver input.",
|
|
219
|
+
"oneOf": [
|
|
220
|
+
{
|
|
221
|
+
"const": "focus",
|
|
222
|
+
"description": "Focus the target window, then type into it.",
|
|
223
|
+
"type": "string"
|
|
224
|
+
},
|
|
225
|
+
{
|
|
226
|
+
"const": "background",
|
|
227
|
+
"description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
|
|
228
|
+
"type": "string"
|
|
229
|
+
}
|
|
230
|
+
]
|
|
231
|
+
},
|
|
217
232
|
"ScreenshotOutputFormat": {
|
|
218
233
|
"enum": [
|
|
219
234
|
"png",
|
|
@@ -254,6 +269,18 @@ export const GENERATED_MCP_TOOLS =
|
|
|
254
269
|
"default": "capture",
|
|
255
270
|
"description": "Pixel space for explicit x/y: raw screenshot capture pixels (default),\nor global AT-SPI screen coordinates mapped through the element's KWin window."
|
|
256
271
|
},
|
|
272
|
+
"delivery": {
|
|
273
|
+
"anyOf": [
|
|
274
|
+
{
|
|
275
|
+
"$ref": "#/$defs/KeyboardDelivery"
|
|
276
|
+
},
|
|
277
|
+
{
|
|
278
|
+
"type": "null"
|
|
279
|
+
}
|
|
280
|
+
],
|
|
281
|
+
"default": null,
|
|
282
|
+
"description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). Needs x and y in screenshot capture pixels over the\nwindow the call targets, without relative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
|
|
283
|
+
},
|
|
257
284
|
"element_index": {
|
|
258
285
|
"default": null,
|
|
259
286
|
"description": "Element index from the latest get_app_state or include_tree snapshot.",
|
|
@@ -1317,9 +1344,24 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1317
1344
|
"openWorldHint": true,
|
|
1318
1345
|
"readOnlyHint": false
|
|
1319
1346
|
},
|
|
1320
|
-
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). With PIXELFLUX_CU and PIXELFLUX_CU_TOKEN set, chords go to pixelflux. An unreachable pixelflux falls through to the routes below; a pixelflux error does not. On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1347
|
+
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, kp_enter (keypad Enter), escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). With PIXELFLUX_CU and PIXELFLUX_CU_TOKEN set, chords go to pixelflux. An unreachable pixelflux falls through to the routes below; a pixelflux error does not. On X11, chords go through xdotool (XTEST), or ydotool when xdotool is missing or cannot start. On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. On KDE Plasma every key in a portal chord is a physical keycode. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass delivery=\"background\" (the default when an element or browser_tab is named) to leave the user's keyboard focus alone: input goes to a named element through AT-SPI, to a browser tab over a DevTools port the user opened (COMPUTER_USE_LINUX_DEVTOOLS), or to a terminal's program through tmux; when no route serves the target (the reason says which setting or desktop would open one), the first such call in a server session sends nothing and returns a notice (background.retry_to_continue=true); repeating it, and every later such call, goes through normal focused input and reports background.used=false. The result's background field names the route used. On an element, only keys that mean an action map (Enter or space on a button or menu item, space on a check box, radio button or toggle, Enter on a link or a single-line field with an activate action); others have no route there. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1321
1348
|
"inputSchema": {
|
|
1322
1349
|
"$defs": {
|
|
1350
|
+
"KeyboardDelivery": {
|
|
1351
|
+
"description": "How type_text and press_key deliver input.",
|
|
1352
|
+
"oneOf": [
|
|
1353
|
+
{
|
|
1354
|
+
"const": "focus",
|
|
1355
|
+
"description": "Focus the target window, then type into it.",
|
|
1356
|
+
"type": "string"
|
|
1357
|
+
},
|
|
1358
|
+
{
|
|
1359
|
+
"const": "background",
|
|
1360
|
+
"description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
|
|
1361
|
+
"type": "string"
|
|
1362
|
+
}
|
|
1363
|
+
]
|
|
1364
|
+
},
|
|
1323
1365
|
"ScreenshotOutputFormat": {
|
|
1324
1366
|
"enum": [
|
|
1325
1367
|
"png",
|
|
@@ -1338,6 +1380,43 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1338
1380
|
"null"
|
|
1339
1381
|
]
|
|
1340
1382
|
},
|
|
1383
|
+
"browser_tab": {
|
|
1384
|
+
"default": null,
|
|
1385
|
+
"description": "Background delivery to a browser tab over DevTools: a target id, or a\ncase-insensitive substring of one page's URL or title. Needs\nCOMPUTER_USE_LINUX_DEVTOOLS (the loopback port of a browser the user\nstarted with --remote-debugging-port, or its DevToolsActivePort\nfile). Input goes to the tab's focused element.",
|
|
1386
|
+
"type": [
|
|
1387
|
+
"string",
|
|
1388
|
+
"null"
|
|
1389
|
+
]
|
|
1390
|
+
},
|
|
1391
|
+
"delivery": {
|
|
1392
|
+
"anyOf": [
|
|
1393
|
+
{
|
|
1394
|
+
"$ref": "#/$defs/KeyboardDelivery"
|
|
1395
|
+
},
|
|
1396
|
+
{
|
|
1397
|
+
"type": "null"
|
|
1398
|
+
}
|
|
1399
|
+
],
|
|
1400
|
+
"default": null,
|
|
1401
|
+
"description": "\"focus\" focuses the target window and types into it. \"background\"\nleaves the user's keyboard focus where it is and delivers through the\nroute the call names: an element (element_index, element_identifier\nor role/name/states), a browser_tab, or a terminal (tty, terminal_pid,\nterminal_command, terminal_cwd, or a terminal window) whose program\nruns in tmux. When no route serves the target, the first such call in\na server session sends nothing and returns a notice; repeated, and\nfor every later such call, the input goes through normal focused\ninput (an element gets keyboard focus first, a browser_tab targets\nthe one window whose title contains it). A wrong or ambiguous target is still an\nerror. Default: \"background\" when an element or browser_tab is named,\nelse \"focus\"."
|
|
1402
|
+
},
|
|
1403
|
+
"element_identifier": {
|
|
1404
|
+
"default": null,
|
|
1405
|
+
"description": "A node's object_ref from get_app_state or get_tree_page, for\nbackground delivery to that element. Wins over element_index and the\nselectors.",
|
|
1406
|
+
"type": [
|
|
1407
|
+
"string",
|
|
1408
|
+
"null"
|
|
1409
|
+
]
|
|
1410
|
+
},
|
|
1411
|
+
"element_index": {
|
|
1412
|
+
"default": null,
|
|
1413
|
+
"description": "Background delivery to an element: its element_index from the latest\nget_app_state or include_tree snapshot. type_text inserts at its caret\n(or end) through AT-SPI EditableText and reads it back; press_key runs\nthe AT-SPI action the key means (Enter or space on a button or menu\nitem, space on a check box, radio button or toggle, Enter on a link\nor on a single-line field with an activate action).",
|
|
1414
|
+
"minimum": 0,
|
|
1415
|
+
"type": [
|
|
1416
|
+
"integer",
|
|
1417
|
+
"null"
|
|
1418
|
+
]
|
|
1419
|
+
},
|
|
1341
1420
|
"format": {
|
|
1342
1421
|
"anyOf": [
|
|
1343
1422
|
{
|
|
@@ -1384,6 +1463,14 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1384
1463
|
"null"
|
|
1385
1464
|
]
|
|
1386
1465
|
},
|
|
1466
|
+
"name": {
|
|
1467
|
+
"default": null,
|
|
1468
|
+
"description": "Background delivery to the element whose accessible name contains\nthis, ignoring case.",
|
|
1469
|
+
"type": [
|
|
1470
|
+
"string",
|
|
1471
|
+
"null"
|
|
1472
|
+
]
|
|
1473
|
+
},
|
|
1387
1474
|
"pid": {
|
|
1388
1475
|
"default": null,
|
|
1389
1476
|
"description": "Process id. It must match exactly one window.",
|
|
@@ -1402,6 +1489,14 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1402
1489
|
"null"
|
|
1403
1490
|
]
|
|
1404
1491
|
},
|
|
1492
|
+
"role": {
|
|
1493
|
+
"default": null,
|
|
1494
|
+
"description": "Background delivery to the element whose AT-SPI role contains this,\nignoring case (with name and states, it must match one element).",
|
|
1495
|
+
"type": [
|
|
1496
|
+
"string",
|
|
1497
|
+
"null"
|
|
1498
|
+
]
|
|
1499
|
+
},
|
|
1405
1500
|
"scale": {
|
|
1406
1501
|
"description": "Additional downscale factor from 0.0 to 1.0, applied before max dimensions.",
|
|
1407
1502
|
"format": "float",
|
|
@@ -1435,6 +1530,22 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1435
1530
|
"null"
|
|
1436
1531
|
]
|
|
1437
1532
|
},
|
|
1533
|
+
"snapshot_id": {
|
|
1534
|
+
"default": null,
|
|
1535
|
+
"description": "snapshot_id of the result element_index or the selector comes from.\nIf a later snapshot replaced it, the call is refused before input.",
|
|
1536
|
+
"type": [
|
|
1537
|
+
"string",
|
|
1538
|
+
"null"
|
|
1539
|
+
]
|
|
1540
|
+
},
|
|
1541
|
+
"states": {
|
|
1542
|
+
"default": [],
|
|
1543
|
+
"description": "AT-SPI states the background target element must all have.",
|
|
1544
|
+
"items": {
|
|
1545
|
+
"type": "string"
|
|
1546
|
+
},
|
|
1547
|
+
"type": "array"
|
|
1548
|
+
},
|
|
1438
1549
|
"terminal_command": {
|
|
1439
1550
|
"default": null,
|
|
1440
1551
|
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
|
@@ -1870,6 +1981,21 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1870
1981
|
"description": "Scroll an element, window, or point in a direction by a number of pages (5 wheel notches per page). With a window target and no x/y/element_index, scrolls at the centre of the targeted window; with no point or target, at the current pointer. A window target is raised and focused first; nothing is sent when focus cannot be verified. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1871
1982
|
"inputSchema": {
|
|
1872
1983
|
"$defs": {
|
|
1984
|
+
"KeyboardDelivery": {
|
|
1985
|
+
"description": "How type_text and press_key deliver input.",
|
|
1986
|
+
"oneOf": [
|
|
1987
|
+
{
|
|
1988
|
+
"const": "focus",
|
|
1989
|
+
"description": "Focus the target window, then type into it.",
|
|
1990
|
+
"type": "string"
|
|
1991
|
+
},
|
|
1992
|
+
{
|
|
1993
|
+
"const": "background",
|
|
1994
|
+
"description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
|
|
1995
|
+
"type": "string"
|
|
1996
|
+
}
|
|
1997
|
+
]
|
|
1998
|
+
},
|
|
1873
1999
|
"ScreenshotOutputFormat": {
|
|
1874
2000
|
"enum": [
|
|
1875
2001
|
"png",
|
|
@@ -1888,6 +2014,18 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1888
2014
|
"null"
|
|
1889
2015
|
]
|
|
1890
2016
|
},
|
|
2017
|
+
"delivery": {
|
|
2018
|
+
"anyOf": [
|
|
2019
|
+
{
|
|
2020
|
+
"$ref": "#/$defs/KeyboardDelivery"
|
|
2021
|
+
},
|
|
2022
|
+
{
|
|
2023
|
+
"type": "null"
|
|
2024
|
+
}
|
|
2025
|
+
],
|
|
2026
|
+
"default": null,
|
|
2027
|
+
"description": "\"background\" sends the input through the agent's own pointer, so the\nuser's pointer and keyboard focus stay where they are: on X11 without\na window manager (an XInput 2 master pointer) and on sway 1.10 or later\n(a transient seat). Needs x and y in screenshot capture pixels over the\nwindow the call targets, without relative or an element. Elsewhere the\nfirst such call in a server session sends nothing and returns a\nnotice; repeated, and for every later such call, it goes through the\nuser's pointer as with \"focus\". Default: \"focus\"."
|
|
2028
|
+
},
|
|
1891
2029
|
"direction": {
|
|
1892
2030
|
"description": "up, down, left or right (case-insensitive).",
|
|
1893
2031
|
"type": "string"
|
|
@@ -2324,9 +2462,24 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2324
2462
|
"openWorldHint": true,
|
|
2325
2463
|
"readOnlyHint": false
|
|
2326
2464
|
},
|
|
2327
|
-
"description": "Type literal text using keyboard input, optionally after focusing a target window or terminal selector. Keyboard success confirms dispatch, not field-content verification; re-observe the target before retrying. On GNOME, when a configured keyboard layout group cannot type a character of the text, the text goes into the focused editable field of the verified target window through AT-SPI EditableText instead: inserted at the caret, replacing a selection, and reported as ok only when the field reads it back with the rest unchanged. Password fields, text with a newline, tab or other control character, and focus outside an editable field of that active window keep the refusal. X11 xdotool can silently substitute symbols under non-US or changing XKB layouts; inspect the text before pressing Enter or submitting it. The wtype backend rejects NUL and text over 131071 UTF-8 bytes before focus or indicators. On Hyprland, confirmed password fields are rejected, and ambiguous fields and terminals use private stdin for at most 100 Unicode characters per call. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
2465
|
+
"description": "Type literal text using keyboard input, optionally after focusing a target window or terminal selector. Keyboard success confirms dispatch, not field-content verification; re-observe the target before retrying. On GNOME, when a configured keyboard layout group cannot type a character of the text, the text goes into the focused editable field of the verified target window through AT-SPI EditableText instead: inserted at the caret, replacing a selection, and reported as ok only when the field reads it back with the rest unchanged. Password fields, text with a newline, tab or other control character, and focus outside an editable field of that active window keep the refusal. Pass delivery=\"background\" (the default when an element or browser_tab is named) to leave the user's keyboard focus alone: input goes to a named element through AT-SPI, to a browser tab over a DevTools port the user opened (COMPUTER_USE_LINUX_DEVTOOLS), or to a terminal's program through tmux; when no route serves the target (the reason says which setting or desktop would open one), the first such call in a server session sends nothing and returns a notice (background.retry_to_continue=true); repeating it, and every later such call, goes through normal focused input and reports background.used=false. The result's background field names the route used. Into a named element the text goes in at its caret (or end) and must read back; a password field is filled unverified. X11 xdotool can silently substitute symbols under non-US or changing XKB layouts; inspect the text before pressing Enter or submitting it. The wtype backend rejects NUL and text over 131071 UTF-8 bytes before focus or indicators. On Hyprland, confirmed password fields are rejected, and ambiguous fields and terminals use private stdin for at most 100 Unicode characters per call. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
2328
2466
|
"inputSchema": {
|
|
2329
2467
|
"$defs": {
|
|
2468
|
+
"KeyboardDelivery": {
|
|
2469
|
+
"description": "How type_text and press_key deliver input.",
|
|
2470
|
+
"oneOf": [
|
|
2471
|
+
{
|
|
2472
|
+
"const": "focus",
|
|
2473
|
+
"description": "Focus the target window, then type into it.",
|
|
2474
|
+
"type": "string"
|
|
2475
|
+
},
|
|
2476
|
+
{
|
|
2477
|
+
"const": "background",
|
|
2478
|
+
"description": "Leave the user's keyboard focus alone and deliver through a route the\ncall names.",
|
|
2479
|
+
"type": "string"
|
|
2480
|
+
}
|
|
2481
|
+
]
|
|
2482
|
+
},
|
|
2330
2483
|
"ScreenshotOutputFormat": {
|
|
2331
2484
|
"enum": [
|
|
2332
2485
|
"png",
|
|
@@ -2345,6 +2498,43 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2345
2498
|
"null"
|
|
2346
2499
|
]
|
|
2347
2500
|
},
|
|
2501
|
+
"browser_tab": {
|
|
2502
|
+
"default": null,
|
|
2503
|
+
"description": "Background delivery to a browser tab over DevTools: a target id, or a\ncase-insensitive substring of one page's URL or title. Needs\nCOMPUTER_USE_LINUX_DEVTOOLS (the loopback port of a browser the user\nstarted with --remote-debugging-port, or its DevToolsActivePort\nfile). Input goes to the tab's focused element.",
|
|
2504
|
+
"type": [
|
|
2505
|
+
"string",
|
|
2506
|
+
"null"
|
|
2507
|
+
]
|
|
2508
|
+
},
|
|
2509
|
+
"delivery": {
|
|
2510
|
+
"anyOf": [
|
|
2511
|
+
{
|
|
2512
|
+
"$ref": "#/$defs/KeyboardDelivery"
|
|
2513
|
+
},
|
|
2514
|
+
{
|
|
2515
|
+
"type": "null"
|
|
2516
|
+
}
|
|
2517
|
+
],
|
|
2518
|
+
"default": null,
|
|
2519
|
+
"description": "\"focus\" focuses the target window and types into it. \"background\"\nleaves the user's keyboard focus where it is and delivers through the\nroute the call names: an element (element_index, element_identifier\nor role/name/states), a browser_tab, or a terminal (tty, terminal_pid,\nterminal_command, terminal_cwd, or a terminal window) whose program\nruns in tmux. When no route serves the target, the first such call in\na server session sends nothing and returns a notice; repeated, and\nfor every later such call, the input goes through normal focused\ninput (an element gets keyboard focus first, a browser_tab targets\nthe one window whose title contains it). A wrong or ambiguous target is still an\nerror. Default: \"background\" when an element or browser_tab is named,\nelse \"focus\"."
|
|
2520
|
+
},
|
|
2521
|
+
"element_identifier": {
|
|
2522
|
+
"default": null,
|
|
2523
|
+
"description": "A node's object_ref from get_app_state or get_tree_page, for\nbackground delivery to that element. Wins over element_index and the\nselectors.",
|
|
2524
|
+
"type": [
|
|
2525
|
+
"string",
|
|
2526
|
+
"null"
|
|
2527
|
+
]
|
|
2528
|
+
},
|
|
2529
|
+
"element_index": {
|
|
2530
|
+
"default": null,
|
|
2531
|
+
"description": "Background delivery to an element: its element_index from the latest\nget_app_state or include_tree snapshot. type_text inserts at its caret\n(or end) through AT-SPI EditableText and reads it back; press_key runs\nthe AT-SPI action the key means (Enter or space on a button or menu\nitem, space on a check box, radio button or toggle, Enter on a link\nor on a single-line field with an activate action).",
|
|
2532
|
+
"minimum": 0,
|
|
2533
|
+
"type": [
|
|
2534
|
+
"integer",
|
|
2535
|
+
"null"
|
|
2536
|
+
]
|
|
2537
|
+
},
|
|
2348
2538
|
"format": {
|
|
2349
2539
|
"anyOf": [
|
|
2350
2540
|
{
|
|
@@ -2387,6 +2577,14 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2387
2577
|
"null"
|
|
2388
2578
|
]
|
|
2389
2579
|
},
|
|
2580
|
+
"name": {
|
|
2581
|
+
"default": null,
|
|
2582
|
+
"description": "Background delivery to the element whose accessible name contains\nthis, ignoring case.",
|
|
2583
|
+
"type": [
|
|
2584
|
+
"string",
|
|
2585
|
+
"null"
|
|
2586
|
+
]
|
|
2587
|
+
},
|
|
2390
2588
|
"pid": {
|
|
2391
2589
|
"default": null,
|
|
2392
2590
|
"description": "Process id. It must match exactly one window.",
|
|
@@ -2405,6 +2603,14 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2405
2603
|
"null"
|
|
2406
2604
|
]
|
|
2407
2605
|
},
|
|
2606
|
+
"role": {
|
|
2607
|
+
"default": null,
|
|
2608
|
+
"description": "Background delivery to the element whose AT-SPI role contains this,\nignoring case (with name and states, it must match one element).",
|
|
2609
|
+
"type": [
|
|
2610
|
+
"string",
|
|
2611
|
+
"null"
|
|
2612
|
+
]
|
|
2613
|
+
},
|
|
2408
2614
|
"scale": {
|
|
2409
2615
|
"description": "Additional downscale factor from 0.0 to 1.0, applied before max dimensions.",
|
|
2410
2616
|
"format": "float",
|
|
@@ -2438,6 +2644,22 @@ export const GENERATED_MCP_TOOLS =
|
|
|
2438
2644
|
"null"
|
|
2439
2645
|
]
|
|
2440
2646
|
},
|
|
2647
|
+
"snapshot_id": {
|
|
2648
|
+
"default": null,
|
|
2649
|
+
"description": "snapshot_id of the result element_index or the selector comes from.\nIf a later snapshot replaced it, the call is refused before input.",
|
|
2650
|
+
"type": [
|
|
2651
|
+
"string",
|
|
2652
|
+
"null"
|
|
2653
|
+
]
|
|
2654
|
+
},
|
|
2655
|
+
"states": {
|
|
2656
|
+
"default": [],
|
|
2657
|
+
"description": "AT-SPI states the background target element must all have.",
|
|
2658
|
+
"items": {
|
|
2659
|
+
"type": "string"
|
|
2660
|
+
},
|
|
2661
|
+
"type": "array"
|
|
2662
|
+
},
|
|
2441
2663
|
"terminal_command": {
|
|
2442
2664
|
"default": null,
|
|
2443
2665
|
"description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
|
package/pi/extension/index.ts
CHANGED
|
@@ -242,6 +242,7 @@ const FORWARDED_ENVIRONMENT = [
|
|
|
242
242
|
"RUST_LOG",
|
|
243
243
|
"COMPUTER_USE_LINUX_AGENT_NAME",
|
|
244
244
|
"COMPUTER_USE_LINUX_COSMIC_HELPER",
|
|
245
|
+
"COMPUTER_USE_LINUX_DEVTOOLS",
|
|
245
246
|
"COMPUTER_USE_LINUX_ENABLE_SHELL",
|
|
246
247
|
"COMPUTER_USE_LINUX_FORCE_PORTAL_KEYBOARD",
|
|
247
248
|
"COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER",
|
|
@@ -134,6 +134,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
|
|
|
134
134
|
|
|
135
135
|
### Host-specific guides
|
|
136
136
|
|
|
137
|
+
- [GitHub Copilot CLI setup](https://github.com/agent-sh/computer-use-linux#github-copilot-cli)
|
|
137
138
|
- [Hermes setup](references/hermes-setup.md)
|
|
138
139
|
- [Pi coding agent setup](references/pi-setup.md)
|
|
139
140
|
- [Selkies container setup](references/selkies-setup.md)
|
|
@@ -235,6 +236,7 @@ directly to a window-relative click.
|
|
|
235
236
|
- `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive.
|
|
236
237
|
- On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
|
|
237
238
|
- On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
|
|
239
|
+
- To act without taking the user's keyboard, pass `delivery: "background"` to `type_text` or `press_key` (naming an element or `browser_tab` implies it). Name an element from `get_app_state` (text is inserted and read back; Enter or space runs a button's, link's, check box's or single-line field's action), a `browser_tab` when `COMPUTER_USE_LINUX_DEVTOOLS` names the port of a browser the user started with `--remote-debugging-port`, or a terminal by `tty` or terminal selectors (kitty remote control, wezterm, tmux or GNU screen). On X11 and sway 1.10+ a plain window target also works: the agent gets its own keyboard focus (an XInput 2 master or a transient seat), and `click`/`scroll` with `delivery: "background"`, `x`/`y` and a window target use the agent's own pointer (X11 only with no window manager). On sway, Xwayland, GTK and Chromium windows and Alt/Super chords have no background route; on X11, window manager shortcuts have none. `doctor` shows the seat in `capabilities.background_seat`. When no route serves a call, the first one in the session sends nothing and returns a notice (`background.retry_to_continue: true`) with the reason; repeating the call sends it through normal input, which moves the user's keyboard focus and pointer, and later calls with no route do that without asking again (`background.used: false`). Repeat only if taking the user's focus is acceptable; otherwise use the setting the reason names, or agent-workspace-linux (https://github.com/agent-sh/agent-workspace-linux), which gives the agent its own session with its own keyboard and pointer. A wrong or ambiguous target is still an error.
|
|
238
240
|
- Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
|
|
239
241
|
- Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.
|
|
240
242
|
- `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.
|