@agent-sh/computer-use-linux 0.7.49 → 0.7.50
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -169,6 +169,7 @@ The binary also exposes the same capabilities from the CLI for scripting and deb
|
|
|
169
169
|
|
|
170
170
|
```
|
|
171
171
|
computer-use-linux mcp # stdio MCP server
|
|
172
|
+
computer-use-linux mcp --listen 127.0.0.1:5200 # streamable HTTP MCP; needs COMPUTER_USE_LINUX_MCP_TOKEN
|
|
172
173
|
computer-use-linux doctor # JSON readiness report
|
|
173
174
|
computer-use-linux setup # enable AT-SPI
|
|
174
175
|
computer-use-linux guard-accessibility # explicit foreground GNOME accessibility guard
|
|
@@ -274,7 +275,7 @@ You will still need AT-SPI enabled (`computer-use-linux setup`) and one input ba
|
|
|
274
275
|
|
|
275
276
|
## Wire it into your MCP host
|
|
276
277
|
|
|
277
|
-
The binary speaks MCP over stdio through `rmcp` 3.5. It answers `initialize` with the client's protocol version when that is 2024-11-05, 2025-03-26, 2025-06-18 or 2025-11-25, and with 2025-11-25 for any other.
|
|
278
|
+
The binary speaks MCP over stdio through `rmcp` 3.5, or over streamable HTTP with `mcp --listen` (see [Generic MCP client](#generic-mcp-client)). It answers `initialize` with the client's protocol version when that is 2024-11-05, 2025-03-26, 2025-06-18 or 2025-11-25, and with 2025-11-25 for any other. 2026-07-28 clients skip `initialize`: they send `server/discover` and carry the version in each request's `_meta`, on stdio or HTTP. For stdio, pass `mcp` as the only argument; everything else is configured through MCP tool calls.
|
|
278
279
|
|
|
279
280
|
### Claude Code and Codex plugin
|
|
280
281
|
|
|
@@ -439,7 +440,7 @@ For one-off installs without adding the tap first, Hermes also accepts `hermes s
|
|
|
439
440
|
|
|
440
441
|
### Generic MCP client
|
|
441
442
|
|
|
442
|
-
Spawn the binary with `["mcp"]` as the argv tail. It speaks MCP (JSON-RPC over stdio) through rmcp, with the protocol versions [listed above](#wire-it-into-your-mcp-host); capability discovery happens through `tools/list` and the `doctor` tool. The server normally needs no MCP-specific configuration, but desktop runtime environment still matters (`DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR`, portals, AT-SPI, `ydotoold`, and optionally `COMPUTER_USE_LINUX_COSMIC_HELPER`).
|
|
443
|
+
Spawn the binary with `["mcp"]` as the argv tail. It speaks MCP (JSON-RPC over stdio) through rmcp, with the protocol versions [listed above](#wire-it-into-your-mcp-host); capability discovery happens through `tools/list` and the `doctor` tool. `mcp --listen ADDR:PORT` serves the same protocol over streamable HTTP at `/mcp` instead; without `--listen` no socket is opened. It needs a bearer token from the file named by `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` (for example a mounted Kubernetes Secret) or from `COMPUTER_USE_LINUX_MCP_TOKEN`. Prefer the file, because a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in `/proc/<pid>/environ` while the server runs, even though child processes do not inherit it. Clients send `Authorization: Bearer <token>`; a missing or wrong token gets 401. `localhost`, `127.0.0.1` and `::1` are always accepted as the `Host`. A bind that is not loopback, including `0.0.0.0` and `[::]`, needs each name clients use as `--allowed-host NAME` (repeatable, any port, case-insensitive); another `Host` gets 403. Any request with an `Origin` header gets 403, because browser clients are not supported. `--listen` refuses to start while `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, every `get_tree_page` call, and `click`, `scroll`, `set_value` and `perform_action` calls that pick an element by `element_index` or selector, must pass the `snapshot_id` of the caller's latest `get_app_state` or `include_tree` result; calls by `element_identifier` or capture-space `x`/`y` do not. Clients share one cached tree, so two clients that alternate `get_app_state` refuse each other's ids; call `get_app_state` again after a refusal. Responses may be SSE (`text/event-stream`). 2024-11-05 and 2025 clients get sessions; 2026-07-28 requests run stateless. SIGTERM and SIGINT stop the listener cleanly with exit status 128 plus the signal number, and a connection that does not send its request headers within 10 seconds is closed. The port is plain HTTP: put TLS in front (service mesh, ingress or sidecar) when it leaves the host. The server normally needs no other MCP-specific configuration, but desktop runtime environment still matters (`DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR`, portals, AT-SPI, `ydotoold`, and optionally `COMPUTER_USE_LINUX_COSMIC_HELPER`). Idle sessions expire after an hour. A `404 Session not found` means the session expired, so initialize again; the cached tree and its `snapshot_id` survive re-initializing.
|
|
443
444
|
|
|
444
445
|
### Selkies containers (linuxserver.io)
|
|
445
446
|
|
|
@@ -498,6 +499,8 @@ Most setups need none of these: the server and installers detect backends and pa
|
|
|
498
499
|
| `COMPUTER_USE_LINUX_SCREENSHOT_BACKEND` | Force a single screenshot backend, skipping the fallback chain. Accepts `gnome-shell`, `portal`, `x11`, `gnome-screenshot`, or `pixelflux`. `x11` works only on a native X11 session. `pixelflux` needs `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN`. Pin `gnome-screenshot` for background/systemd contexts where the GNOME Shell and portal DBus paths are denied. |
|
|
499
500
|
| `PIXELFLUX_CU` | Loopback port of a Selkies pixelflux computer-use server (`5000` or `127.0.0.1:5000`). With `PIXELFLUX_CU_TOKEN`, screenshots try pixelflux before the other backends, and click, scroll, drag, press_key, and type_text go there without a portal prompt. A closed port falls through. A server error does not. Anything other than loopback is ignored. |
|
|
500
501
|
| `PIXELFLUX_CU_TOKEN` | Bearer token for `PIXELFLUX_CU`. Never logged. Both variables must be set. |
|
|
502
|
+
| `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` | Path of a file holding the `mcp --listen` bearer token, such as a mounted Kubernetes Secret. Surrounding whitespace is trimmed. Setting both this and `COMPUTER_USE_LINUX_MCP_TOKEN` is an error. |
|
|
503
|
+
| `COMPUTER_USE_LINUX_MCP_TOKEN` | The `mcp --listen` bearer token, when no token file is used. Clients send `Authorization: Bearer <token>`. The token is never logged, and both variables are removed from the environment so child processes do not inherit them. Prefer `COMPUTER_USE_LINUX_MCP_TOKEN_FILE`: a token passed here stays readable in `/proc/<pid>/environ` for the server's lifetime. |
|
|
501
504
|
| `COMPUTER_USE_LINUX_ENABLE_SHELL` | Set exactly to `1` before starting the MCP server to register the destructive `run_shell` tool. Unset by default. Do not enable for untrusted or unattended MCP hosts. |
|
|
502
505
|
|
|
503
506
|
**Build-time identity overrides** (set while compiling a downstream embedded
|
|
@@ -520,7 +523,7 @@ files.
|
|
|
520
523
|
|
|
521
524
|
- **Accessibility tree**: [`atspi`](https://crates.io/crates/atspi) crate (tokio backend) talks to the AT-SPI registry on the user session bus. The tree is flattened to `(role, name, text, states, bounds)` tuples and indexed; element indices are stable for the duration of a `get_app_state` snapshot. A new snapshot is published only between input operations. `click`, `scroll`, `perform_action`, and `set_value` hold the input lock while they check and resolve an index, so a checked element cannot turn into another app's node mid-action.
|
|
522
525
|
- **DBus where desktops expose it**: [`zbus`](https://crates.io/crates/zbus) for portal calls (`org.freedesktop.portal.Screenshot`, `…RemoteDesktop`, `…ScreenCast`), GNOME Shell screenshots (`org.gnome.Shell.Screenshot`), the bundled GNOME extension's `dev.avifenesh.ComputerUseLinux.WindowControl` service, and temporary KWin scripting.
|
|
523
|
-
- **MCP transport**: [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature
|
|
526
|
+
- **MCP transport**: [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature for stdio, and its streamable HTTP server on `axum` for `mcp --listen`. No socket is opened without `--listen`.
|
|
524
527
|
- **Input fallback**: on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when it is installed on a compatible compositor (not GNOME, KDE or COSMIC), unless `COMPUTER_USE_LINUX_FORCE_PORTAL_KEYBOARD=1`; `wtype` supports Unicode through the virtual-keyboard protocol. Otherwise KDE uses its clipboard route and other Wayland sessions use portal keysyms, which never fall back to ydotool. ydotool 1.0.3+ with its `ydotoold` socket types only when the selected wtype is missing or cannot start, when KDE cannot read the Klipper clipboard, or when ydotool is forced. A wtype that started and failed is returned without replaying the text. `press_key` chords use the portal when ydotool is absent; on a session whose bus answers that it has no RemoteDesktop portal (wlroots compositors such as Sway, labwc, river or cage), they go through `wtype` instead. The route is chosen before any input is sent, so a failed or denied portal request is never replayed through wtype. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
|
|
525
528
|
- **Hyprland background mode**: with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a targeted `type_text` or `press_key` is sent as `hl.dsp.send_key_state` down/up pairs for the target's address, written straight to Hyprland's control socket in one `[[BATCH]]` request per 64 key presses (never through `hyprctl` arguments, so typed text cannot be read from `/proc/<pid>/cmdline`), and the window is never focused. Hyprland answers each dispatch, so when every dispatch in a failed batch was answered the error states how many key presses were delivered (and whether a key was left down), and the caller can send only the remainder. An empty or incomplete reply, a timeout, or an ok after an error proves nothing about what ran, so the error gives the confirmed count and says the rest may have been delivered. The action indicator shows the input; it is masked unless an AT-SPI probe of the target app finds a non-secret editable field or a terminal. The dispatcher resolves a key name to its unshifted physical key and applies only the modifiers it is given, so text goes through a US-layout table (shifted symbols become SHIFT plus the base key); unmappable characters or a non-US main keymap refuse the whole call. An unfocused target is captured with `grim -T <stableId>` (ext-foreign-toplevel image capture). Hyprland stops sending frame callbacks to windows on hidden workspaces, so their buffer holds the last frame drawn on screen; each capture makes the client repaint, so a hidden-workspace target is captured every 150 ms until two consecutive frames match (at most 8 captures) and the result's `source` says whether it settled.
|
|
526
529
|
- **Hyprland/Electron text**: for classified nonsecret fields on Hyprland, wtype passes one literal argument after `--` to avoid stdin keymap-refresh corruption. Argument text is visible in process listings; confirmed password fields are rejected. Terminals and ambiguous fields (with or without AT-SPI) use private stdin for at most 100 Unicode characters per call; split larger input into smaller calls or expose accessibility. The input probe has its own 1.5-second budget; the 250 ms indicator probe only masks text. Other compatible compositors retain their existing stdin path. NUL and text over 131,071 UTF-8 bytes are rejected before focus or indicators. Success confirms dispatch only; re-observe the field before retrying.
|
|
@@ -550,7 +553,8 @@ Computer-use tooling is, by definition, a privilege-escalation surface. The thre
|
|
|
550
553
|
- **AT-SPI exposes window contents to any client on your session bus.** Enabling the AT-SPI bridge (`setup_accessibility`) is a prerequisite for this binary; it's also what screen readers use, and it shares the same trust boundary.
|
|
551
554
|
- **The GNOME Shell extension** is loaded only into your user's GNOME Shell, runs in the Shell's JS sandbox, and exposes a single DBus interface on the user session bus. It does not request any extra permissions.
|
|
552
555
|
- **The indicator shows what agents type.** `type_text` text (its last 64 characters) and `press_key` keycaps are drawn on screen unless the target is, or might be, a password field: both are replaced by a fixed mask when the focused element's AT-SPI role is a password role or cannot be read within 250 ms. Generic text boxes and terminal focus are also masked because their roles do not establish whether input is obscured. `set_value` values are always masked, because an app can change a field to a password after its accessibility snapshot. The text also travels over a socket in the user's private runtime directory. Set `COMPUTER_USE_LINUX_INDICATOR_HIDE_TEXT=1` to mask all typed text and keycaps, or `COMPUTER_USE_LINUX_INDICATOR=0` to turn the indicator off.
|
|
553
|
-
- **No network.** This binary opens no TCP/UDP listener, makes no outbound Internet connections, and ships no telemetry. It does use local session transports such as DBus and the per-user `ydotoold` Unix socket. With `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` set, it also connects to that loopback TCP port, and a `DISPLAY` that names a host makes X11 queries connect to that X server.
|
|
556
|
+
- **No network by default.** This binary opens no TCP/UDP listener unless `mcp --listen` is passed, makes no outbound Internet connections, and ships no telemetry. It does use local session transports such as DBus and the per-user `ydotoold` Unix socket. With `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` set, it also connects to that loopback TCP port, and a `DISPLAY` that names a host makes X11 queries connect to that X server.
|
|
557
|
+
- **HTTP MCP is opt-in and unencrypted.** `mcp --listen` opens a plain HTTP streamable MCP endpoint. It requires a bearer token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN` and answers 401 to any other `Authorization` value. It answers 403 to a `Host` not on its allow list and to any request with an `Origin` header, and it refuses to start with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Bind it to `127.0.0.1` or a private address, and put TLS in front (service mesh, ingress or sidecar) before it crosses a trust boundary. Stdio remains the default and opens no port.
|
|
554
558
|
- **Mutating tools are explicit.** The MCP tool list annotates read-only versus mutating tools, and CI fails if the published tool annotations drift from the table above. Treat those annotations as hints; the host is still responsible for user approval and policy.
|
|
555
559
|
|
|
556
560
|
If you're running this on a shared workstation, set `ydotoold`'s socket permissions to `0600` (the default) and audit which processes on your user can `connect()` to it.
|
package/npm/README.md
CHANGED
|
@@ -160,6 +160,18 @@ does not yet forward the flag or include the tool in its catalog.
|
|
|
160
160
|
unknown/mixed output scaling leave window-relative coordinates unavailable;
|
|
161
161
|
focus the target and inspect a fresh full-screen screenshot before using
|
|
162
162
|
desktop coordinates.
|
|
163
|
+
- **HTTP MCP**: `computer-use-linux mcp --listen ADDR:PORT` serves streamable
|
|
164
|
+
HTTP at `/mcp`. It needs a token from the file in `COMPUTER_USE_LINUX_MCP_TOKEN_FILE`
|
|
165
|
+
or from `COMPUTER_USE_LINUX_MCP_TOKEN`; clients send `Authorization: Bearer <token>`.
|
|
166
|
+
Prefer the file: a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in
|
|
167
|
+
`/proc/<pid>/environ` while the server runs.
|
|
168
|
+
A bind that is not loopback needs `--allowed-host NAME` for each hostname clients
|
|
169
|
+
use. Requests with an `Origin` header are refused, and `--listen` refuses to start
|
|
170
|
+
with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, `get_tree_page` and calls that pick
|
|
171
|
+
an element by index or selector must pass `snapshot_id`. The port is plain HTTP; put TLS in front
|
|
172
|
+
(service mesh, ingress or sidecar). Idle sessions expire after an hour. A
|
|
173
|
+
`404 Session not found` means initialize again; the cached tree and its
|
|
174
|
+
`snapshot_id` survive re-initializing.
|
|
163
175
|
|
|
164
176
|
## Input & coordinate contracts
|
|
165
177
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.50",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,7 +7,7 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.50";
|
|
11
11
|
export const GENERATED_TOOL_CATALOG_HASH = "cffd6953843133dfee35c6c5bfc4f98e177798a31cc05e711a18d6b96ae7d2e4";
|
|
12
12
|
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "1b9f83b4ce6e5b07b25eae206dae6af7a31dda69fd073ad55d50f03bbd4bdb66";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|
|
@@ -229,6 +229,7 @@ directly to a window-relative click.
|
|
|
229
229
|
- Already-running GTK, Qt, and Electron apps may need a restart after AT-SPI is enabled.
|
|
230
230
|
- GNOME may show a portal prompt on the first screenshot or `get_app_state` call with screenshots enabled.
|
|
231
231
|
- Inside a Selkies container, `PIXELFLUX_CU` plus `PIXELFLUX_CU_TOKEN` sends screenshots and input to pixelflux on loopback, with no portal prompt. A closed port falls through. A server error does not, so do not retry that call through another backend. pixelflux takes left, right and middle clicks with counts 1 to 3, and multi-clicks only with the left button; other clicks go to the next backend. Its results carry no off-screen or monitor-gap warning.
|
|
232
|
+
- `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive.
|
|
232
233
|
- On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
|
|
233
234
|
- On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
|
|
234
235
|
- Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
|
|
@@ -5,7 +5,7 @@ description: "Run computer-use-linux inside a linuxserver.io Selkies desktop con
|
|
|
5
5
|
|
|
6
6
|
# Selkies Container Setup
|
|
7
7
|
|
|
8
|
-
linuxserver.io's Selkies images (`webtop`, `chrome`, and the other desktop apps) run a full desktop in a container. Run `computer-use-linux` inside the container and let the MCP client on the host reach it over stdio through `docker exec`.
|
|
8
|
+
linuxserver.io's Selkies images (`webtop`, `chrome`, and the other desktop apps) run a full desktop in a container. Run `computer-use-linux` inside the container and let the MCP client on the host reach it over stdio through `docker exec`. This stdio setup opens no port. To reach the server over the network instead, see [Serve MCP over HTTP](#serve-mcp-over-http).
|
|
9
9
|
|
|
10
10
|
## Start the container with `PELORUS=true`
|
|
11
11
|
|
|
@@ -74,6 +74,30 @@ On the host, point the client at the launcher through `docker exec -i`:
|
|
|
74
74
|
}
|
|
75
75
|
```
|
|
76
76
|
|
|
77
|
+
## Serve MCP over HTTP
|
|
78
|
+
|
|
79
|
+
When the client cannot `docker exec` into the container, for example in Kubernetes, run the launcher with `--listen`. Mount the token from a Secret and set `COMPUTER_USE_LINUX_MCP_TOKEN_FILE=/run/secrets/mcp/token` in the container environment (`docker run -e …` or the pod spec's `env:`). The launcher runs under s6 `with-contenv`, which drops variables set inline on its command line and loads only the container environment. Then name the Service clients use:
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
/config/.local/bin/computer-use-linux-selkies mcp --listen 0.0.0.0:5200 --allowed-host desktop
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
`--allowed-host` takes the Service name (`desktop`, or `desktop.ns.svc`); repeat it for each name clients use. A loopback bind such as `127.0.0.1:5200` needs no `--allowed-host`. Do not set `COMPUTER_USE_LINUX_ENABLE_SHELL=1`; `--listen` refuses to start with it. The port is plain HTTP, so put TLS in front with a service mesh, ingress or sidecar.
|
|
86
|
+
|
|
87
|
+
For Claude Code, add the server to `.mcp.json` with the URL and the bearer header:
|
|
88
|
+
|
|
89
|
+
```json
|
|
90
|
+
{
|
|
91
|
+
"type": "http",
|
|
92
|
+
"url": "http://desktop:5200/mcp",
|
|
93
|
+
"headers": { "Authorization": "Bearer <token>" }
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Other clients take the same URL and `Authorization: Bearer <token>` header in their own config format.
|
|
98
|
+
|
|
99
|
+
Over HTTP, pass the `snapshot_id` from your latest `get_app_state` to every `get_tree_page` call and every call that picks an element by index or selector.
|
|
100
|
+
|
|
77
101
|
## Check it
|
|
78
102
|
|
|
79
103
|
```bash
|