@agent-sh/computer-use-linux 0.7.35 → 0.7.36
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -11,8 +11,10 @@
|
|
|
11
11
|
|
|
12
12
|
> Running open models for your company? [**Tiyuvta**](https://tiyuvta.ai/services/) helps with model choice, deployment and optimization, and fine-tuning on your hardware or cloud account.
|
|
13
13
|
|
|
14
|
-
>
|
|
15
|
-
|
|
14
|
+
<p align="center">
|
|
15
|
+
<a href="https://github.com/sponsors/avifenesh"><img src="https://img.shields.io/badge/Sponsor-%E2%9D%A4-db61a2?style=for-the-badge&logo=githubsponsors&logoColor=white" alt="Sponsor"></a>
|
|
16
|
+
</p>
|
|
17
|
+
<p align="center"><b>Built and maintained with a lot of agent tokens and hours. If it saves you time, <a href="https://github.com/sponsors/avifenesh">sponsoring</a> helps keep it going.</b></p>
|
|
16
18
|
|
|
17
19
|
`computer-use-linux` gives AI agents real, structured control of a local Linux desktop: semantic AT-SPI accessibility trees, bounded screenshots, compositor-aware window targeting, Wayland/X11 input synthesis, single-round-trip post-action verification, and a click-through on-screen activity indicator across **GNOME, KDE Plasma/KWin, Hyprland, niri, COSMIC, i3, labwc, and X11/EWMH**.
|
|
18
20
|
|
|
@@ -51,7 +53,7 @@ The Rust crate is published as [`computer-use-linux`](https://crates.io/crates/c
|
|
|
51
53
|
Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive `xdotool` against an X11 root window or shell out to OCR over screenshots. Five things set this one apart:
|
|
52
54
|
|
|
53
55
|
- **Wayland actually works.** Pointer actions use `/dev/uinput` or `org.freedesktop.portal.RemoteDesktop` on Wayland, with `ydotool` / `ydotoold` as the fallback. Literal text prefers KDE clipboard integration on Plasma, `wtype` on compatible Wayland compositors, and portal keysyms on other Wayland sessions, including GNOME with ydotool installed. Portal text startup or conversion errors fail the call without replaying text through ydotool. Raw ydotool text accepts printable ASCII, tab and newline only, and its physical US key positions can produce different text under the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, then on a native X11 session a root-window `GetImage` (device pixels, no toolkit scaling), and finally spawn `gnome-screenshot` for background/systemd contexts where the DBus paths are denied.
|
|
54
|
-
- **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed. Terminal windows are enriched with their controlling TTY and foreground process so agents can target "the terminal running `pytest`" directly. Every `window_id` fits JavaScript's exact integer range (at most 2^53-1), so JSON clients can pass it back unchanged; a larger value is rejected with a prompt to re-run `list_windows`.
|
|
56
|
+
- **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed. Terminal windows are enriched with their controlling TTY and foreground process so agents can target "the terminal running `pytest`" directly. Every `window_id` fits JavaScript's exact integer range (at most 2^53-1), so JSON clients can pass it back unchanged; a larger value is rejected with a prompt to re-run `list_windows`.
|
|
55
57
|
- **Semantic selectors first, pixel coordinates when needed.** Tools like `click`, `perform_action`, and `set_value` accept `role` / `name` / `text` / `states` selectors or `element_index` values backed by AT-SPI, and every mutating tool can return a post-action screenshot and updated accessibility tree in the same round trip. Pixel coordinates remain available for rendering-only surfaces (canvas, games, X clients without ATK).
|
|
56
58
|
- **Visible to humans, invisible to screenshots.** While an agent acts, `computer-use-linux-indicator` renders a click-through overlay with a per-agent cursor glide, click ripple, keycaps/typing banner, screen-edge glow, and status pill — and hides automatically under a shared capture lock before any screenshot is read.
|
|
57
59
|
- **One JSON readiness report.** `computer-use-linux doctor` returns a structured document covering platform, portals, AT-SPI, windowing, input, and a `readiness` summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.
|
|
@@ -188,7 +190,7 @@ Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). niri window list
|
|
|
188
190
|
| i3 | `i3-msg`; optional `xprop` for PID hydration | Lists and focuses i3 windows over the active i3 IPC socket. |
|
|
189
191
|
| labwc with Selkies IPC | `$XDG_RUNTIME_DIR/labwc.sock` | Requires the Selkies IPC patch; stock labwc has no such socket. Lists windows and focuses unique app-id/title matches through `wlr-foreign-toplevel-management`. IDs include pid, app id and title, fit JavaScript integers, and change when the title changes. Ambiguous windows are rejected. Bounds are omitted because the IPC does not expose output scaling. |
|
|
190
192
|
| COSMIC Wayland | `computer-use-linux-cosmic` helper | Installed automatically by `./install.sh`, `cargo install`, and npm. For custom/manual layouts, put the helper next to the main binary, on `PATH`, or point `COMPUTER_USE_LINUX_COSMIC_HELPER` at it. The on-screen indicator was validated here (three outputs, mixed fractional scaling). |
|
|
191
|
-
|
|
|
193
|
+
| Other wlroots compositors (stock labwc, phoc, river-classic) | `zwlr_foreign_toplevel_manager_v1` on `WAYLAND_DISPLAY` | Lists toplevels with title, app id, and activated/minimized state, and focuses unique app-id/title matches. The protocol reports no pids or bounds. IDs derive from the app id and title, fit JavaScript integers, and change when the title changes; windows sharing both are rejected for exact focus. On Sway it answers only when the i3 backend does not: when `i3-msg` is missing, cannot reach the IPC socket, or lists no windows. Compositors that do not advertise the protocol, such as Cage, are not supported. |
|
|
192
194
|
| Generic X11 / XFCE / other EWMH WMs | `wmctrl` plus `xprop` | Lists, focuses, moves, and resizes windows; keyboard input prefers `xdotool`/XTEST, and scroll uses xdotool wheel buttons. Window origins are read from the X server, since `wmctrl -lG` counts the frame offset twice. |
|
|
193
195
|
|
|
194
196
|
If you run on a desktop not covered above, or a covered backend does not come up cleanly, please open an issue with the output of `computer-use-linux doctor` so we can extend the matrix honestly.
|
|
@@ -435,6 +437,10 @@ For one-off installs without adding the tap first, Hermes also accepts `hermes s
|
|
|
435
437
|
|
|
436
438
|
Spawn the binary with `["mcp"]` as the argv tail. It speaks JSON-RPC over stdio per the rmcp 2024-11-05 protocol; capability discovery happens through `tools/list` and the `doctor` tool. The server normally needs no MCP-specific configuration, but desktop runtime environment still matters (`DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR`, portals, AT-SPI, `ydotoold`, and optionally `COMPUTER_USE_LINUX_COSMIC_HELPER`).
|
|
437
439
|
|
|
440
|
+
### Selkies containers (linuxserver.io)
|
|
441
|
+
|
|
442
|
+
linuxserver.io's Selkies desktop images (`webtop`, `chrome`, and others) can run the server inside the container, with the host's MCP client connecting over stdio through `docker exec`. Start the container with `PELORUS=true` so pixelflux serves screenshots and input. A small launcher joins the desktop session, because `docker exec` starts outside it. See the [Selkies container guide](skills/computer-use-linux/references/selkies-setup.md).
|
|
443
|
+
|
|
438
444
|
## First-run checklist
|
|
439
445
|
|
|
440
446
|
1. **Run `doctor`.**
|
|
@@ -510,7 +516,7 @@ files.
|
|
|
510
516
|
- **Accessibility tree** — [`atspi`](https://crates.io/crates/atspi) crate (tokio backend) talks to the AT-SPI registry on the user session bus. The tree is flattened to `(role, name, text, states, bounds)` tuples and indexed; element indices are stable for the duration of a `get_app_state` snapshot. A new snapshot is published only between input operations. `click`, `scroll`, `perform_action`, and `set_value` hold the input lock while they check and resolve an index, so a checked element cannot turn into another app's node mid-action.
|
|
511
517
|
- **DBus where desktops expose it** — [`zbus`](https://crates.io/crates/zbus) for portal calls (`org.freedesktop.portal.Screenshot`, `…RemoteDesktop`, `…ScreenCast`), GNOME Shell screenshots (`org.gnome.Shell.Screenshot`), the bundled GNOME extension's `dev.avifenesh.ComputerUseLinux.WindowControl` service, and temporary KWin scripting.
|
|
512
518
|
- **MCP transport** — [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature; stdio framing, no network.
|
|
513
|
-
- **Input fallback** — on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when installed and the remote-desktop portal is unavailable; `wtype` supports Unicode through the virtual-keyboard protocol on compatible compositors such as Hyprland/wlroots. If wtype is unavailable, the binary falls back to a compatible ydotool 1.0.3+ CLI and `ydotoold` socket. A launched wtype failure is returned without replaying the text. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
|
|
519
|
+
- **Input fallback** — on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when installed and the remote-desktop portal is unavailable; `wtype` supports Unicode through the virtual-keyboard protocol on compatible compositors such as Hyprland/wlroots. If wtype is unavailable, the binary falls back to a compatible ydotool 1.0.3+ CLI and `ydotoold` socket. A launched wtype failure is returned without replaying the text. `press_key` chords use the portal when ydotool is absent; on a session whose bus answers that it has no RemoteDesktop portal (wlroots compositors such as Sway, labwc, river or cage), they go through `wtype` instead. The route is chosen before any input is sent, so a failed or denied portal request is never replayed through wtype. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
|
|
514
520
|
- **Hyprland background mode** — with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a targeted `type_text` or `press_key` is sent as `hl.dsp.send_key_state` down/up pairs for the target's address, written straight to Hyprland's control socket in one `[[BATCH]]` request per 64 key presses (never through `hyprctl` arguments, so typed text cannot be read from `/proc/<pid>/cmdline`), and the window is never focused. Hyprland answers each dispatch, so when every dispatch in a failed batch was answered the error states how many key presses were delivered (and whether a key was left down), and the caller can send only the remainder. An empty or incomplete reply, a timeout, or an ok after an error proves nothing about what ran, so the error gives the confirmed count and says the rest may have been delivered. The action indicator shows the input; it is masked unless an AT-SPI probe of the target app finds a non-secret editable field or a terminal. The dispatcher resolves a key name to its unshifted physical key and applies only the modifiers it is given, so text goes through a US-layout table (shifted symbols become SHIFT plus the base key); unmappable characters or a non-US main keymap refuse the whole call. An unfocused target is captured with `grim -T <stableId>` (ext-foreign-toplevel image capture). Hyprland stops sending frame callbacks to windows on hidden workspaces, so their buffer holds the last frame drawn on screen; each capture makes the client repaint, so a hidden-workspace target is captured every 150 ms until two consecutive frames match (at most 8 captures) and the result's `source` says whether it settled.
|
|
515
521
|
- **Hyprland/Electron text** — for classified nonsecret fields on Hyprland, wtype passes one literal argument after `--` to avoid stdin keymap-refresh corruption. Argument text is visible in process listings; confirmed password fields are rejected. Terminals and ambiguous fields (with or without AT-SPI) use private stdin for at most 100 Unicode characters per call; split larger input into smaller calls or expose accessibility. The input probe has its own 1.5-second budget; the 250 ms indicator probe only masks text. Other compatible compositors retain their existing stdin path. NUL and text over 131,071 UTF-8 bytes are rejected before focus or indicators. Success confirms dispatch only; re-observe the field before retrying.
|
|
516
522
|
- **Limited accessibility feedback** — when exact window focus was verified before input but AT-SPI reports no focused element, the result describes the missing feedback without claiming the input was lost. It still does not verify the field contents. A complete scoped tree containing only application/window containers adds an Electron/Chromium `--force-renderer-accessibility` hint; a truncated tree instead retains its traversal-limit guidance.
|
|
@@ -522,7 +528,7 @@ files.
|
|
|
522
528
|
- **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. Missing or failed xdotool commands return an error without a system-wide fallback. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.
|
|
523
529
|
- **Drag motion.** XTEST, uinput and portal drags use intermediate points paced 30 ms apart. Native X11 uses XTEST on `DISPLAY`; uinput and portal paths use the same trajectory. A failed press or motion still attempts button release, and the gesture is not replayed through another backend. Portal paths require shared endpoints and clamp intermediate points to the nearest shared monitor when the trajectory crosses a monitor gap. Verify the destination after dispatch.
|
|
524
530
|
- **Native X11 transport.** Native X11 queries (window origins, root screenshots, `doctor`) connect and read replies under one shared 2-second deadline enforced in the transport itself, including the handshake and every reply. A stalled X server closes the query connection instead of leaving a blocked worker behind. Unix sockets, literal IP addresses, and `localhost` need no resolver. A remote hostname in `DISPLAY` needs `getent ahosts` for bounded name resolution; if it is missing or fails, the X11 route is reported as unavailable.
|
|
525
|
-
- **Window registry** — `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
|
|
531
|
+
- **Window registry** — `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
|
|
526
532
|
- **niri backend** — prefers `niri msg`, falling back to direct JSON IPC over the same session socket. `is_minimized` maps to `hidden`. Bounds use `layout.window_size` (or `tile_size`) and, when available, the tile position plus the window's offset and the output's logical origin. Uniform output scaling converts these to device pixels; unknown or mixed scaling omits bounds. Missing window positions stay `null`, so listing and focusing still work without enabling coordinate targeting. An accepted focus action is verified by querying the focused window again.
|
|
527
533
|
- **GNOME extension fallback** — recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`.
|
|
528
534
|
- **COSMIC helper** — `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it.
|
package/npm/README.md
CHANGED
|
@@ -14,7 +14,7 @@ Provides:
|
|
|
14
14
|
> `ToolAnnotations` so hosts can distinguish read-only observation from mutating
|
|
15
15
|
> desktop actions.
|
|
16
16
|
|
|
17
|
-
On labwc, window targeting
|
|
17
|
+
On labwc, window targeting uses the Selkies IPC patch's `labwc.sock` when it exists. Stock labwc and other wlroots compositors that advertise wlr-foreign-toplevel-management use that protocol instead, which reports no pids or bounds. Activation requires a unique app-id/title match. Window IDs include the title and must be refreshed after it changes. Ambiguous windows are rejected. Window bounds and relative coordinate targeting are unavailable until output scaling can be verified.
|
|
18
18
|
|
|
19
19
|
Native X11 clicks, scrolls and drags use xdotool/XTEST on `DISPLAY`, ahead of uinput and portal input. Install xdotool for this route. Missing or failed commands return an error without a system-wide fallback. Drags send paced intermediate motion and always attempt release after a failed press or motion; verify the drop target afterward.
|
|
20
20
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.36",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
11
|
-
export const GENERATED_TOOL_CATALOG_HASH = "
|
|
12
|
-
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.36";
|
|
11
|
+
export const GENERATED_TOOL_CATALOG_HASH = "626768053c7e97574226d8d8b382ced55272e50273dc6fe72a81d130b20fdb7a";
|
|
12
|
+
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "628cbd7dc623cfcf7ff2dc18c69960f4c964cf1566045d796e89a76ba7bba9e2";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|
|
14
14
|
[
|
|
15
15
|
{
|
|
@@ -1202,7 +1202,7 @@ export const GENERATED_MCP_TOOLS =
|
|
|
1202
1202
|
"openWorldHint": true,
|
|
1203
1203
|
"readOnlyHint": false
|
|
1204
1204
|
},
|
|
1205
|
-
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1205
|
+
"description": "Press a key or key-combination on the keyboard, optionally after focusing a target window or terminal selector. Key grammar (case-insensitive; hyphens/spaces ignored): combos join with '+', e.g. Ctrl+L or Ctrl+Shift+T. Modifiers: ctrl/control, alt/option, shift, meta/super/cmd/command. Named keys: enter/return, escape/esc, tab, backspace, delete/del, space, home, end, pageup, pagedown, arrowleft/left, arrowright/right, arrowup/up, arrowdown/down, f1-f12. Plus single US letters a-z and digits 0-9. Anything else returns an error (never silently dropped). On Wayland, chords are sent through an active remote desktop portal keyboard session when one is available (or when ydotool is absent), falling back to ydotool otherwise. When ydotool is absent and the session has no RemoteDesktop portal (for example wlroots compositors), chords go through wtype instead. Portal chords send modifiers and named keys as keysyms so remapped keys (e.g. Caps Lock swapped with Control) follow the active keymap; letters and digits are physical US positions. Note: compositor-level shortcuts (e.g. Super+Up) may be consumed by GNOME before reaching the app. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
|
|
1206
1206
|
"inputSchema": {
|
|
1207
1207
|
"$defs": {
|
|
1208
1208
|
"ScreenshotOutputFormat": {
|
|
@@ -11,7 +11,7 @@ compatibility: "Native Pi tools require Pi 0.84.4+ and Node.js 22.19+; the stand
|
|
|
11
11
|
|
|
12
12
|
Use `computer-use-linux` when an agent needs to observe or operate a local Linux desktop: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions.
|
|
13
13
|
|
|
14
|
-
On labwc, window targeting
|
|
14
|
+
On labwc, window targeting uses the Selkies IPC patch's `labwc.sock` when it exists. Stock labwc and other wlroots compositors that advertise wlr-foreign-toplevel-management use that protocol instead, which reports no pids or bounds. Activation requires a unique app-id/title match. Window IDs include the title and must be refreshed after it changes. Ambiguous windows are rejected. Window bounds and relative coordinate targeting are unavailable until output scaling can be verified.
|
|
15
15
|
|
|
16
16
|
Native X11 clicks, scrolls and drags use xdotool/XTEST on `DISPLAY`, ahead of uinput and portal input. Install xdotool for this route. Missing or failed commands return an error without a system-wide fallback. Drags send paced intermediate motion and always attempt release after a failed press or motion; verify the drop target afterward.
|
|
17
17
|
|
|
@@ -134,6 +134,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
|
|
|
134
134
|
|
|
135
135
|
- [Hermes setup](references/hermes-setup.md)
|
|
136
136
|
- [Pi coding agent setup](references/pi-setup.md)
|
|
137
|
+
- [Selkies container setup](references/selkies-setup.md)
|
|
137
138
|
|
|
138
139
|
## Procedure
|
|
139
140
|
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: selkies-setup
|
|
3
|
+
description: "Run computer-use-linux inside a linuxserver.io Selkies desktop container."
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Selkies Container Setup
|
|
7
|
+
|
|
8
|
+
linuxserver.io's Selkies images (`webtop`, `chrome`, and the other desktop apps) run a full desktop in a container. Run `computer-use-linux` inside the container and let the MCP client on the host reach it over stdio through `docker exec`. No port is opened.
|
|
9
|
+
|
|
10
|
+
## Start the container with `PELORUS=true`
|
|
11
|
+
|
|
12
|
+
```bash
|
|
13
|
+
docker run -d --name desktop --shm-size=1gb -e PELORUS=true -p 3001:3001 lscr.io/linuxserver/chrome:latest
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
`PELORUS=true` turns on pixelflux's computer-use server, so screenshots and input work without a consent prompt. On labwc images such as `chrome` it also starts the session bus and the AT-SPI registry. As a side effect it runs linuxserver.io's own Pelorus agent on port 5100 inside the container, which stays private unless you publish that port.
|
|
17
|
+
|
|
18
|
+
## Install the binary
|
|
19
|
+
|
|
20
|
+
Install as `abc`, the desktop user, into `/config`, which survives container restarts. Re-run the same block to update to a newer release:
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
docker exec -u abc desktop sh -c '
|
|
24
|
+
set -e
|
|
25
|
+
mkdir -p /config/.local/bin && cd /config/.local/bin
|
|
26
|
+
asset=computer-use-linux-$(uname -m)-unknown-linux-gnu
|
|
27
|
+
base=https://github.com/agent-sh/computer-use-linux/releases/latest/download
|
|
28
|
+
curl -fsSLO "$base/$asset" -O "$base/$asset.sha256"
|
|
29
|
+
sha256sum -c "$asset.sha256"
|
|
30
|
+
mv "$asset" computer-use-linux && chmod +x computer-use-linux && rm "$asset.sha256"'
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
Release binaries need glibc. Alpine-based tags such as `webtop:alpine-kde` use musl, so pick a Debian, Ubuntu, Fedora, or Arch tag, for example `webtop:debian-kde`.
|
|
34
|
+
|
|
35
|
+
## Add the session launcher
|
|
36
|
+
|
|
37
|
+
`docker exec` starts outside the desktop session, with no session bus and no `WAYLAND_DISPLAY`. Selkies saves the variables its own services use (`XDG_RUNTIME_DIR`, `PIXELFLUX_CU`, `PIXELFLUX_CU_TOKEN`) in `/run/s6/container_environment`, where s6's `with-contenv` loads them. The launcher does that, then takes the session bus from a desktop process: `plasmashell` on KDE webtops, which also gives it the display, or `labwc` on app images. labwc's own `WAYLAND_DISPLAY` is `wayland-1`, Selkies' display, so the launcher points at the desktop's `wayland-0` instead. The remaining desktop variables already come from `with-contenv`, so on labwc the bus is the only thing taken from the desktop process.
|
|
38
|
+
|
|
39
|
+
```bash
|
|
40
|
+
docker exec -i -u abc desktop sh -c 'cat > /config/.local/bin/computer-use-linux-selkies && chmod +x /config/.local/bin/computer-use-linux-selkies' <<'EOF'
|
|
41
|
+
#!/command/with-contenv bash
|
|
42
|
+
# Run computer-use-linux inside the Selkies desktop session.
|
|
43
|
+
# with-contenv supplies XDG_RUNTIME_DIR and the pixelflux variables.
|
|
44
|
+
if session=$(pgrep -xo plasmashell); then
|
|
45
|
+
# KDE webtops: plasmashell carries the whole session environment.
|
|
46
|
+
vars='DBUS_SESSION_BUS_ADDRESS|DISPLAY|WAYLAND_DISPLAY|XDG_CURRENT_DESKTOP|XDG_SESSION_TYPE'
|
|
47
|
+
elif session=$(pgrep -xo labwc); then
|
|
48
|
+
# labwc images: labwc's own WAYLAND_DISPLAY is Selkies' display, so take only its bus.
|
|
49
|
+
export WAYLAND_DISPLAY=wayland-0
|
|
50
|
+
vars='DBUS_SESSION_BUS_ADDRESS'
|
|
51
|
+
else
|
|
52
|
+
echo "computer-use-linux-selkies: no running plasmashell or labwc session" >&2
|
|
53
|
+
exit 1
|
|
54
|
+
fi
|
|
55
|
+
while IFS= read -r var; do
|
|
56
|
+
export "${var?}"
|
|
57
|
+
done < <(tr '\0' '\n' < "/proc/${session}/environ" | grep -E "^(${vars})=")
|
|
58
|
+
if [ -z "${DBUS_SESSION_BUS_ADDRESS:-}" ]; then
|
|
59
|
+
echo "computer-use-linux-selkies: could not read the session bus from /proc/${session}/environ" >&2
|
|
60
|
+
exit 1
|
|
61
|
+
fi
|
|
62
|
+
exec /config/.local/bin/computer-use-linux "${@:-mcp}"
|
|
63
|
+
EOF
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
## Configure the MCP client
|
|
67
|
+
|
|
68
|
+
On the host, point the client at the launcher through `docker exec -i`:
|
|
69
|
+
|
|
70
|
+
```json
|
|
71
|
+
{
|
|
72
|
+
"command": "docker",
|
|
73
|
+
"args": ["exec", "-i", "-u", "abc", "desktop", "/config/.local/bin/computer-use-linux-selkies"]
|
|
74
|
+
}
|
|
75
|
+
```
|
|
76
|
+
|
|
77
|
+
## Check it
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
docker exec -u abc desktop /config/.local/bin/computer-use-linux-selkies doctor
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
`capabilities.screenshot` and `capabilities.input` should start with `pixelflux`, and `capabilities.window_control` should list `labwc` or `kwin`.
|
|
84
|
+
|
|
85
|
+
On KDE webtops, AT-SPI starts disabled. Call `setup_accessibility` once, then restart any app that was already open so it publishes its accessibility tree.
|