@agent-sh/computer-use-linux 0.7.48 → 0.7.50
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -9,6 +9,8 @@
|
|
|
9
9
|
</p>
|
|
10
10
|
</div>
|
|
11
11
|
|
|
12
|
+
> Running open models for your company? [**Tiyuvta**](https://tiyuvta.ai/services/) helps with model choice, deployment and optimization, and fine-tuning on your hardware or cloud account.
|
|
13
|
+
|
|
12
14
|
<p align="center">
|
|
13
15
|
<a href="https://github.com/sponsors/avifenesh"><img src="https://img.shields.io/badge/Sponsor-%E2%9D%A4-db61a2?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="Sponsor"></a>
|
|
14
16
|
</p>
|
|
@@ -83,9 +85,9 @@ The crate was extracted from [`codex-desktop-linux`](https://github.com/avifenes
|
|
|
83
85
|
| **Opt-in host tools** | `complete_interaction` | Parameter-free desktop completion notification (`notify-send`); registered only when `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`. |
|
|
84
86
|
| | `run_shell` | Bounded non-login `/bin/sh -c` execution; registered only when `COMPUTER_USE_LINUX_ENABLE_SHELL=1` (absent by default). |
|
|
85
87
|
|
|
86
|
-
Every tree result carries a `snapshot_id`: `get_app_state`, an `include_tree` observation, and `get_tree_page`. The server caches one tree, and each new tree replaces it. Pass the `snapshot_id` with an `element_index` or selector to `click`, `scroll`, `perform_action` or `set_value`, and to `get_tree_page`: if a later tree replaced that snapshot, the call is refused before any window is raised or input is sent, since the index could name another element by then. A full `x`/`y` point does not come from the tree, so it is not checked. Without `snapshot_id`, the element resolves in the latest tree. An `element_identifier` names the AT-SPI object itself and is not checked against a snapshot. Ids differ between server processes, so an id from before a restart is refused.
|
|
88
|
+
Every tree result carries a `snapshot_id`: `get_app_state`, an `include_tree` observation, and `get_tree_page`. The server caches one tree, and each new tree replaces it. Pass the `snapshot_id` with an `element_index` (or a selector, except on `scroll`) to `click`, `scroll`, `perform_action` or `set_value`, and to `get_tree_page`: if a later tree replaced that snapshot, the call is refused before any window is raised or input is sent, since the index could name another element by then. A full capture-space `x`/`y` point does not come from the tree, so it is not checked. A `coordinate_space: "atspi"` click maps through its element, so it is checked. Without `snapshot_id`, the element resolves in the latest tree. An `element_identifier` names the AT-SPI object itself and is not checked against a snapshot. Ids differ between server processes, so an id from before a restart is refused.
|
|
87
89
|
|
|
88
|
-
`click`, `scroll`, `perform_action`, and `set_value` take an optional app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`). With one, an `element_index` or selector from another app's `get_app_state` snapshot is refused; call `get_app_state` for the target app first. When the window backend reports no pid for a target window (i3 without `xprop`, COSMIC, other wlroots compositors), the snapshot must have been taken for that window, or the call is refused. `perform_action` and `set_value` do not focus the target window. An `element_identifier` is refused when its owner on the accessibility bus is a different pid than the target; it is not checked when either pid is unknown.
|
|
90
|
+
`click`, `scroll`, `perform_action`, and `set_value` take an optional app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`). With one, an `element_index` (or a selector, where the tool takes one) from another app's `get_app_state` snapshot is refused; call `get_app_state` for the target app first. When the window backend reports no pid for a target window (i3 without `xprop`, COSMIC, other wlroots compositors), the snapshot must have been taken for that window, or the call is refused. `perform_action` and `set_value` do not focus the target window. An `element_identifier` is refused when its owner on the accessibility bus is a different pid than the target; it is not checked when either pid is unknown.
|
|
89
91
|
|
|
90
92
|
### Post-action observation in one round trip
|
|
91
93
|
|
|
@@ -167,6 +169,7 @@ The binary also exposes the same capabilities from the CLI for scripting and deb
|
|
|
167
169
|
|
|
168
170
|
```
|
|
169
171
|
computer-use-linux mcp # stdio MCP server
|
|
172
|
+
computer-use-linux mcp --listen 127.0.0.1:5200 # streamable HTTP MCP; needs COMPUTER_USE_LINUX_MCP_TOKEN
|
|
170
173
|
computer-use-linux doctor # JSON readiness report
|
|
171
174
|
computer-use-linux setup # enable AT-SPI
|
|
172
175
|
computer-use-linux guard-accessibility # explicit foreground GNOME accessibility guard
|
|
@@ -272,7 +275,7 @@ You will still need AT-SPI enabled (`computer-use-linux setup`) and one input ba
|
|
|
272
275
|
|
|
273
276
|
## Wire it into your MCP host
|
|
274
277
|
|
|
275
|
-
The binary speaks MCP over stdio through `rmcp` 3.5. It answers `initialize` with the client's protocol version when that is 2024-11-05, 2025-03-26, 2025-06-18 or 2025-11-25, and with 2025-11-25 for any other.
|
|
278
|
+
The binary speaks MCP over stdio through `rmcp` 3.5, or over streamable HTTP with `mcp --listen` (see [Generic MCP client](#generic-mcp-client)). It answers `initialize` with the client's protocol version when that is 2024-11-05, 2025-03-26, 2025-06-18 or 2025-11-25, and with 2025-11-25 for any other. 2026-07-28 clients skip `initialize`: they send `server/discover` and carry the version in each request's `_meta`, on stdio or HTTP. For stdio, pass `mcp` as the only argument; everything else is configured through MCP tool calls.
|
|
276
279
|
|
|
277
280
|
### Claude Code and Codex plugin
|
|
278
281
|
|
|
@@ -437,7 +440,7 @@ For one-off installs without adding the tap first, Hermes also accepts `hermes s
|
|
|
437
440
|
|
|
438
441
|
### Generic MCP client
|
|
439
442
|
|
|
440
|
-
Spawn the binary with `["mcp"]` as the argv tail. It speaks MCP (JSON-RPC over stdio) through rmcp, with the protocol versions [listed above](#wire-it-into-your-mcp-host); capability discovery happens through `tools/list` and the `doctor` tool. The server normally needs no MCP-specific configuration, but desktop runtime environment still matters (`DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR`, portals, AT-SPI, `ydotoold`, and optionally `COMPUTER_USE_LINUX_COSMIC_HELPER`).
|
|
443
|
+
Spawn the binary with `["mcp"]` as the argv tail. It speaks MCP (JSON-RPC over stdio) through rmcp, with the protocol versions [listed above](#wire-it-into-your-mcp-host); capability discovery happens through `tools/list` and the `doctor` tool. `mcp --listen ADDR:PORT` serves the same protocol over streamable HTTP at `/mcp` instead; without `--listen` no socket is opened. It needs a bearer token from the file named by `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` (for example a mounted Kubernetes Secret) or from `COMPUTER_USE_LINUX_MCP_TOKEN`. Prefer the file, because a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in `/proc/<pid>/environ` while the server runs, even though child processes do not inherit it. Clients send `Authorization: Bearer <token>`; a missing or wrong token gets 401. `localhost`, `127.0.0.1` and `::1` are always accepted as the `Host`. A bind that is not loopback, including `0.0.0.0` and `[::]`, needs each name clients use as `--allowed-host NAME` (repeatable, any port, case-insensitive); another `Host` gets 403. Any request with an `Origin` header gets 403, because browser clients are not supported. `--listen` refuses to start while `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, every `get_tree_page` call, and `click`, `scroll`, `set_value` and `perform_action` calls that pick an element by `element_index` or selector, must pass the `snapshot_id` of the caller's latest `get_app_state` or `include_tree` result; calls by `element_identifier` or capture-space `x`/`y` do not. Clients share one cached tree, so two clients that alternate `get_app_state` refuse each other's ids; call `get_app_state` again after a refusal. Responses may be SSE (`text/event-stream`). 2024-11-05 and 2025 clients get sessions; 2026-07-28 requests run stateless. SIGTERM and SIGINT stop the listener cleanly with exit status 128 plus the signal number, and a connection that does not send its request headers within 10 seconds is closed. The port is plain HTTP: put TLS in front (service mesh, ingress or sidecar) when it leaves the host. The server normally needs no other MCP-specific configuration, but desktop runtime environment still matters (`DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR`, portals, AT-SPI, `ydotoold`, and optionally `COMPUTER_USE_LINUX_COSMIC_HELPER`). Idle sessions expire after an hour. A `404 Session not found` means the session expired, so initialize again; the cached tree and its `snapshot_id` survive re-initializing.
|
|
441
444
|
|
|
442
445
|
### Selkies containers (linuxserver.io)
|
|
443
446
|
|
|
@@ -496,6 +499,8 @@ Most setups need none of these: the server and installers detect backends and pa
|
|
|
496
499
|
| `COMPUTER_USE_LINUX_SCREENSHOT_BACKEND` | Force a single screenshot backend, skipping the fallback chain. Accepts `gnome-shell`, `portal`, `x11`, `gnome-screenshot`, or `pixelflux`. `x11` works only on a native X11 session. `pixelflux` needs `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN`. Pin `gnome-screenshot` for background/systemd contexts where the GNOME Shell and portal DBus paths are denied. |
|
|
497
500
|
| `PIXELFLUX_CU` | Loopback port of a Selkies pixelflux computer-use server (`5000` or `127.0.0.1:5000`). With `PIXELFLUX_CU_TOKEN`, screenshots try pixelflux before the other backends, and click, scroll, drag, press_key, and type_text go there without a portal prompt. A closed port falls through. A server error does not. Anything other than loopback is ignored. |
|
|
498
501
|
| `PIXELFLUX_CU_TOKEN` | Bearer token for `PIXELFLUX_CU`. Never logged. Both variables must be set. |
|
|
502
|
+
| `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` | Path of a file holding the `mcp --listen` bearer token, such as a mounted Kubernetes Secret. Surrounding whitespace is trimmed. Setting both this and `COMPUTER_USE_LINUX_MCP_TOKEN` is an error. |
|
|
503
|
+
| `COMPUTER_USE_LINUX_MCP_TOKEN` | The `mcp --listen` bearer token, when no token file is used. Clients send `Authorization: Bearer <token>`. The token is never logged, and both variables are removed from the environment so child processes do not inherit them. Prefer `COMPUTER_USE_LINUX_MCP_TOKEN_FILE`: a token passed here stays readable in `/proc/<pid>/environ` for the server's lifetime. |
|
|
499
504
|
| `COMPUTER_USE_LINUX_ENABLE_SHELL` | Set exactly to `1` before starting the MCP server to register the destructive `run_shell` tool. Unset by default. Do not enable for untrusted or unattended MCP hosts. |
|
|
500
505
|
|
|
501
506
|
**Build-time identity overrides** (set while compiling a downstream embedded
|
|
@@ -518,7 +523,7 @@ files.
|
|
|
518
523
|
|
|
519
524
|
- **Accessibility tree**: [`atspi`](https://crates.io/crates/atspi) crate (tokio backend) talks to the AT-SPI registry on the user session bus. The tree is flattened to `(role, name, text, states, bounds)` tuples and indexed; element indices are stable for the duration of a `get_app_state` snapshot. A new snapshot is published only between input operations. `click`, `scroll`, `perform_action`, and `set_value` hold the input lock while they check and resolve an index, so a checked element cannot turn into another app's node mid-action.
|
|
520
525
|
- **DBus where desktops expose it**: [`zbus`](https://crates.io/crates/zbus) for portal calls (`org.freedesktop.portal.Screenshot`, `…RemoteDesktop`, `…ScreenCast`), GNOME Shell screenshots (`org.gnome.Shell.Screenshot`), the bundled GNOME extension's `dev.avifenesh.ComputerUseLinux.WindowControl` service, and temporary KWin scripting.
|
|
521
|
-
- **MCP transport**: [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature
|
|
526
|
+
- **MCP transport**: [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature for stdio, and its streamable HTTP server on `axum` for `mcp --listen`. No socket is opened without `--listen`.
|
|
522
527
|
- **Input fallback**: on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when it is installed on a compatible compositor (not GNOME, KDE or COSMIC), unless `COMPUTER_USE_LINUX_FORCE_PORTAL_KEYBOARD=1`; `wtype` supports Unicode through the virtual-keyboard protocol. Otherwise KDE uses its clipboard route and other Wayland sessions use portal keysyms, which never fall back to ydotool. ydotool 1.0.3+ with its `ydotoold` socket types only when the selected wtype is missing or cannot start, when KDE cannot read the Klipper clipboard, or when ydotool is forced. A wtype that started and failed is returned without replaying the text. `press_key` chords use the portal when ydotool is absent; on a session whose bus answers that it has no RemoteDesktop portal (wlroots compositors such as Sway, labwc, river or cage), they go through `wtype` instead. The route is chosen before any input is sent, so a failed or denied portal request is never replayed through wtype. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
|
|
523
528
|
- **Hyprland background mode**: with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a targeted `type_text` or `press_key` is sent as `hl.dsp.send_key_state` down/up pairs for the target's address, written straight to Hyprland's control socket in one `[[BATCH]]` request per 64 key presses (never through `hyprctl` arguments, so typed text cannot be read from `/proc/<pid>/cmdline`), and the window is never focused. Hyprland answers each dispatch, so when every dispatch in a failed batch was answered the error states how many key presses were delivered (and whether a key was left down), and the caller can send only the remainder. An empty or incomplete reply, a timeout, or an ok after an error proves nothing about what ran, so the error gives the confirmed count and says the rest may have been delivered. The action indicator shows the input; it is masked unless an AT-SPI probe of the target app finds a non-secret editable field or a terminal. The dispatcher resolves a key name to its unshifted physical key and applies only the modifiers it is given, so text goes through a US-layout table (shifted symbols become SHIFT plus the base key); unmappable characters or a non-US main keymap refuse the whole call. An unfocused target is captured with `grim -T <stableId>` (ext-foreign-toplevel image capture). Hyprland stops sending frame callbacks to windows on hidden workspaces, so their buffer holds the last frame drawn on screen; each capture makes the client repaint, so a hidden-workspace target is captured every 150 ms until two consecutive frames match (at most 8 captures) and the result's `source` says whether it settled.
|
|
524
529
|
- **Hyprland/Electron text**: for classified nonsecret fields on Hyprland, wtype passes one literal argument after `--` to avoid stdin keymap-refresh corruption. Argument text is visible in process listings; confirmed password fields are rejected. Terminals and ambiguous fields (with or without AT-SPI) use private stdin for at most 100 Unicode characters per call; split larger input into smaller calls or expose accessibility. The input probe has its own 1.5-second budget; the 250 ms indicator probe only masks text. Other compatible compositors retain their existing stdin path. NUL and text over 131,071 UTF-8 bytes are rejected before focus or indicators. Success confirms dispatch only; re-observe the field before retrying.
|
|
@@ -548,7 +553,8 @@ Computer-use tooling is, by definition, a privilege-escalation surface. The thre
|
|
|
548
553
|
- **AT-SPI exposes window contents to any client on your session bus.** Enabling the AT-SPI bridge (`setup_accessibility`) is a prerequisite for this binary; it's also what screen readers use, and it shares the same trust boundary.
|
|
549
554
|
- **The GNOME Shell extension** is loaded only into your user's GNOME Shell, runs in the Shell's JS sandbox, and exposes a single DBus interface on the user session bus. It does not request any extra permissions.
|
|
550
555
|
- **The indicator shows what agents type.** `type_text` text (its last 64 characters) and `press_key` keycaps are drawn on screen unless the target is, or might be, a password field: both are replaced by a fixed mask when the focused element's AT-SPI role is a password role or cannot be read within 250 ms. Generic text boxes and terminal focus are also masked because their roles do not establish whether input is obscured. `set_value` values are always masked, because an app can change a field to a password after its accessibility snapshot. The text also travels over a socket in the user's private runtime directory. Set `COMPUTER_USE_LINUX_INDICATOR_HIDE_TEXT=1` to mask all typed text and keycaps, or `COMPUTER_USE_LINUX_INDICATOR=0` to turn the indicator off.
|
|
551
|
-
- **No network.** This binary opens no TCP/UDP listener, makes no outbound Internet connections, and ships no telemetry. It does use local session transports such as DBus and the per-user `ydotoold` Unix socket. With `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` set, it also connects to that loopback TCP port, and a `DISPLAY` that names a host makes X11 queries connect to that X server.
|
|
556
|
+
- **No network by default.** This binary opens no TCP/UDP listener unless `mcp --listen` is passed, makes no outbound Internet connections, and ships no telemetry. It does use local session transports such as DBus and the per-user `ydotoold` Unix socket. With `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` set, it also connects to that loopback TCP port, and a `DISPLAY` that names a host makes X11 queries connect to that X server.
|
|
557
|
+
- **HTTP MCP is opt-in and unencrypted.** `mcp --listen` opens a plain HTTP streamable MCP endpoint. It requires a bearer token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN` and answers 401 to any other `Authorization` value. It answers 403 to a `Host` not on its allow list and to any request with an `Origin` header, and it refuses to start with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Bind it to `127.0.0.1` or a private address, and put TLS in front (service mesh, ingress or sidecar) before it crosses a trust boundary. Stdio remains the default and opens no port.
|
|
552
558
|
- **Mutating tools are explicit.** The MCP tool list annotates read-only versus mutating tools, and CI fails if the published tool annotations drift from the table above. Treat those annotations as hints; the host is still responsible for user approval and policy.
|
|
553
559
|
|
|
554
560
|
If you're running this on a shared workstation, set `ydotoold`'s socket permissions to `0600` (the default) and audit which processes on your user can `connect()` to it.
|
package/npm/README.md
CHANGED
|
@@ -160,20 +160,33 @@ does not yet forward the flag or include the tool in its catalog.
|
|
|
160
160
|
unknown/mixed output scaling leave window-relative coordinates unavailable;
|
|
161
161
|
focus the target and inspect a fresh full-screen screenshot before using
|
|
162
162
|
desktop coordinates.
|
|
163
|
+
- **HTTP MCP**: `computer-use-linux mcp --listen ADDR:PORT` serves streamable
|
|
164
|
+
HTTP at `/mcp`. It needs a token from the file in `COMPUTER_USE_LINUX_MCP_TOKEN_FILE`
|
|
165
|
+
or from `COMPUTER_USE_LINUX_MCP_TOKEN`; clients send `Authorization: Bearer <token>`.
|
|
166
|
+
Prefer the file: a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in
|
|
167
|
+
`/proc/<pid>/environ` while the server runs.
|
|
168
|
+
A bind that is not loopback needs `--allowed-host NAME` for each hostname clients
|
|
169
|
+
use. Requests with an `Origin` header are refused, and `--listen` refuses to start
|
|
170
|
+
with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, `get_tree_page` and calls that pick
|
|
171
|
+
an element by index or selector must pass `snapshot_id`. The port is plain HTTP; put TLS in front
|
|
172
|
+
(service mesh, ingress or sidecar). Idle sessions expire after an hour. A
|
|
173
|
+
`404 Session not found` means initialize again; the cached tree and its
|
|
174
|
+
`snapshot_id` survive re-initializing.
|
|
163
175
|
|
|
164
176
|
## Input & coordinate contracts
|
|
165
177
|
|
|
166
178
|
- **Snapshot ids**: `get_app_state`, `include_tree` observations and
|
|
167
179
|
`get_tree_page` return the `snapshot_id` of their tree. The server caches
|
|
168
180
|
one tree, and each new tree replaces it. Pass `snapshot_id` with an
|
|
169
|
-
`element_index` or selector to `click`, `scroll`,
|
|
170
|
-
`set_value`, and to `get_tree_page`: a call naming a
|
|
171
|
-
refused before any input. Without it, the element
|
|
172
|
-
tree.
|
|
181
|
+
`element_index` (or a selector, except on `scroll`) to `click`, `scroll`,
|
|
182
|
+
`perform_action` or `set_value`, and to `get_tree_page`: a call naming a
|
|
183
|
+
replaced snapshot is refused before any input. Without it, the element
|
|
184
|
+
resolves in the latest tree. A full capture-space `x`/`y` point is not
|
|
185
|
+
checked; a `coordinate_space: "atspi"` click is checked.
|
|
173
186
|
- **Element targets**: `click`, `scroll`, `perform_action`, and `set_value`
|
|
174
187
|
accept an optional app target (`pid`, `window_id`, `app_id`, `wm_class`,
|
|
175
|
-
`window_title`). With one, an `element_index` or selector
|
|
176
|
-
`get_app_state` snapshot is refused. When the window backend reports no pid
|
|
188
|
+
`window_title`). With one, an `element_index` (or a selector, where the
|
|
189
|
+
tool takes one) from another app's `get_app_state` snapshot is refused. When the window backend reports no pid
|
|
177
190
|
for the target window (i3 without `xprop`, COSMIC, other wlroots compositors),
|
|
178
191
|
the snapshot must have been taken for that window. `perform_action` and
|
|
179
192
|
`set_value` do not focus the window; an `element_identifier` is refused only
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.50",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
11
|
-
export const GENERATED_TOOL_CATALOG_HASH = "
|
|
12
|
-
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.50";
|
|
11
|
+
export const GENERATED_TOOL_CATALOG_HASH = "cffd6953843133dfee35c6c5bfc4f98e177798a31cc05e711a18d6b96ae7d2e4";
|
|
12
|
+
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "1b9f83b4ce6e5b07b25eae206dae6af7a31dda69fd073ad55d50f03bbd4bdb66";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|
|
14
14
|
[
|
|
15
15
|
{
|
|
@@ -146,7 +146,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
|
|
|
146
146
|
4. If `can_build_accessibility_tree` is false, run `setup_accessibility` and restart the target app.
|
|
147
147
|
5. If `can_query_windows` or `can_focus_windows` is false on GNOME Wayland, run `setup_window_targeting` and ask the user to log out and back in if setup says the shell extension needs a reload.
|
|
148
148
|
6. Before targeted input, call `list_windows` or `focused_window` and verify the intended window by title, app id, pid, or wm class.
|
|
149
|
-
7. Prefer semantic targeting from `get_app_state`: use element indices or role/name/text/states selectors, and pass the `snapshot_id` of the result they came from. Every `get_app_state` and `include_tree` result replaces the one cached tree; a call that names a replaced snapshot is refused before any input, so take the indices and `snapshot_id` from the newest result. Automatic recovery of a dead semantic target requires verified PID/window ownership and invalidates cached indices; call `get_app_state` again before further index-based actions. When several apps are in play, pass the app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`) to `click`, `scroll`, `perform_action`, and `set_value`: an index or selector from another app's snapshot is then refused instead of acting on that app. On `click` and `scroll` the target also raises and focuses that window, and nothing is sent when focus cannot be verified; on GNOME Wayland, raising a window needs the Shell extension. The title field there is `window_title`. Where the window backend reports no pid (i3 without `xprop`, COSMIC, and wlr-foreign-toplevel on stock labwc and other wlroots compositors), call `get_app_state` for the same window first or pass `pid`, or the call is refused.
|
|
149
|
+
7. Prefer semantic targeting from `get_app_state`: use element indices or role/name/text/states selectors, and pass the `snapshot_id` of the result they came from. Every `get_app_state` and `include_tree` result replaces the one cached tree; a call that names a replaced snapshot is refused before any input, so take the indices and `snapshot_id` from the newest result. Automatic recovery of a dead semantic target requires verified PID/window ownership and invalidates cached indices; call `get_app_state` again before further index-based actions. When several apps are in play, pass the app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`) to `click`, `scroll`, `perform_action`, and `set_value`: an index (or a selector, where the tool takes one) from another app's snapshot is then refused instead of acting on that app. On `click` and `scroll` the target also raises and focuses that window, and nothing is sent when focus cannot be verified; on GNOME Wayland, raising a window needs the Shell extension. The title field there is `window_title`. Where the window backend reports no pid (i3 without `xprop`, COSMIC, and wlr-foreign-toplevel on stock labwc and other wlroots compositors), call `get_app_state` for the same window first or pass `pid`, or the call is refused.
|
|
150
150
|
8. Use coordinates only when the UI surface has no useful accessibility tree.
|
|
151
151
|
9. For text input, prefer `type_text` with a target selector (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `tty`, `terminal_pid`, `terminal_command`, or `terminal_cwd`) rather than relying on current focus. A `window_id` is valid only as returned by a recent `list_windows`; if a call rejects it as outside 0 to 2^53-1 or matches no window, re-run `list_windows`.
|
|
152
152
|
10. After mutating actions (`click`, `drag`, `scroll`, `press_key`, `type_text`, `perform_action`, `set_value`, `activate_window`, `move_window`, `resize_window`), either pass `screenshot_after: true | "screen" | "window"` (with optional `format`, `quality`, `scale`, `max_width`, `max_height`, `max_bytes`) and/or `include_tree: true` on the action call itself to observe the result in the same tool call, or re-check state with `get_app_state`, `focused_window`, or an app-specific readback. A failed action (`ok: false`) returns no post-action screenshot or tree, so call `get_app_state`; `set_value` still observes when `verified` is `false`.
|
|
@@ -229,6 +229,7 @@ directly to a window-relative click.
|
|
|
229
229
|
- Already-running GTK, Qt, and Electron apps may need a restart after AT-SPI is enabled.
|
|
230
230
|
- GNOME may show a portal prompt on the first screenshot or `get_app_state` call with screenshots enabled.
|
|
231
231
|
- Inside a Selkies container, `PIXELFLUX_CU` plus `PIXELFLUX_CU_TOKEN` sends screenshots and input to pixelflux on loopback, with no portal prompt. A closed port falls through. A server error does not, so do not retry that call through another backend. pixelflux takes left, right and middle clicks with counts 1 to 3, and multi-clicks only with the left button; other clicks go to the next backend. Its results carry no off-screen or monitor-gap warning.
|
|
232
|
+
- `mcp --listen ADDR:PORT` serves streamable HTTP at `/mcp` and requires a token from `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` or `COMPUTER_USE_LINUX_MCP_TOKEN`. A non-loopback bind needs `--allowed-host NAME`. The port is plain HTTP; put TLS in front. Do not retry a 401 by putting the token in a URL. Over HTTP, `get_tree_page` and index and selector calls need `snapshot_id`; if refused, call `get_app_state` again. Idle sessions expire after an hour; on `404 Session not found`, initialize again. The cached tree and its `snapshot_id` survive.
|
|
232
233
|
- On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
|
|
233
234
|
- On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. The active Hyprland keymap must be US, or background `type_text` and `press_key` calls are refused; background `get_app_state` and `screenshot` still capture. Text must be printable ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
|
|
234
235
|
- Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
|
|
@@ -5,7 +5,7 @@ description: "Run computer-use-linux inside a linuxserver.io Selkies desktop con
|
|
|
5
5
|
|
|
6
6
|
# Selkies Container Setup
|
|
7
7
|
|
|
8
|
-
linuxserver.io's Selkies images (`webtop`, `chrome`, and the other desktop apps) run a full desktop in a container. Run `computer-use-linux` inside the container and let the MCP client on the host reach it over stdio through `docker exec`.
|
|
8
|
+
linuxserver.io's Selkies images (`webtop`, `chrome`, and the other desktop apps) run a full desktop in a container. Run `computer-use-linux` inside the container and let the MCP client on the host reach it over stdio through `docker exec`. This stdio setup opens no port. To reach the server over the network instead, see [Serve MCP over HTTP](#serve-mcp-over-http).
|
|
9
9
|
|
|
10
10
|
## Start the container with `PELORUS=true`
|
|
11
11
|
|
|
@@ -74,6 +74,30 @@ On the host, point the client at the launcher through `docker exec -i`:
|
|
|
74
74
|
}
|
|
75
75
|
```
|
|
76
76
|
|
|
77
|
+
## Serve MCP over HTTP
|
|
78
|
+
|
|
79
|
+
When the client cannot `docker exec` into the container, for example in Kubernetes, run the launcher with `--listen`. Mount the token from a Secret and set `COMPUTER_USE_LINUX_MCP_TOKEN_FILE=/run/secrets/mcp/token` in the container environment (`docker run -e …` or the pod spec's `env:`). The launcher runs under s6 `with-contenv`, which drops variables set inline on its command line and loads only the container environment. Then name the Service clients use:
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
/config/.local/bin/computer-use-linux-selkies mcp --listen 0.0.0.0:5200 --allowed-host desktop
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
`--allowed-host` takes the Service name (`desktop`, or `desktop.ns.svc`); repeat it for each name clients use. A loopback bind such as `127.0.0.1:5200` needs no `--allowed-host`. Do not set `COMPUTER_USE_LINUX_ENABLE_SHELL=1`; `--listen` refuses to start with it. The port is plain HTTP, so put TLS in front with a service mesh, ingress or sidecar.
|
|
86
|
+
|
|
87
|
+
For Claude Code, add the server to `.mcp.json` with the URL and the bearer header:
|
|
88
|
+
|
|
89
|
+
```json
|
|
90
|
+
{
|
|
91
|
+
"type": "http",
|
|
92
|
+
"url": "http://desktop:5200/mcp",
|
|
93
|
+
"headers": { "Authorization": "Bearer <token>" }
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Other clients take the same URL and `Authorization: Bearer <token>` header in their own config format.
|
|
98
|
+
|
|
99
|
+
Over HTTP, pass the `snapshot_id` from your latest `get_app_state` to every `get_tree_page` call and every call that picks an element by index or selector.
|
|
100
|
+
|
|
77
101
|
## Check it
|
|
78
102
|
|
|
79
103
|
```bash
|