@agent-sh/computer-use-linux 0.4.9 → 0.5.0

package/README.md

@@ -9,6 +9,8 @@
   </p>
 </div>
 
+> ⚡ Running this agent 24/7? [**tiyuvta inference**](https://inference.tiyuvta.ai) — hosted LLM inference built for always-on agents, OpenAI/Anthropic-compatible APIs.
+
 `computer-use-linux` reads accessibility trees, takes screenshots, and drives clicks, scrolls, and keystrokes across GNOME, KDE/KWin, Hyprland, i3, and COSMIC — Wayland-first, X11 best-effort.
 
 ```bash
@@ -24,7 +26,7 @@ The Rust crate is published as [`computer-use-linux`](https://crates.io/crates/c
 
 Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive `xdotool` against an X11 root window or shell out to OCR over screenshots. Four things set this one apart:
 
-- **Wayland actually works.** Pointer actions can use the `org.freedesktop.portal.RemoteDesktop` interface on Wayland, with `ydotool` / `ydotoold` (uinput) as the deterministic fallback and keyboard/text path. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, and fall back to spawning `gnome-screenshot` for background/systemd contexts where both DBus paths are denied.
+- **Wayland actually works.** Pointer actions can use the `org.freedesktop.portal.RemoteDesktop` interface on Wayland, with `ydotool` / `ydotoold` (uinput) as the deterministic fallback. Literal text prefers `wtype` on compatible Wayland compositors when portal keyboard input is unavailable, preserving Unicode and the active layout before falling back to ydotool. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, and fall back to spawning `gnome-screenshot` for background/systemd contexts where both DBus paths are denied.
 - **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, i3 IPC, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed.
 - **Semantic selectors, not pixel coordinates.** Tools like `click`, `perform_action`, and `set_value` accept `role` / `name` / `text` / `states` selectors backed by AT-SPI. Pixel coordinates remain available as a fallback for rendering-only surfaces (canvas, games, X clients without ATK).
 - **One JSON readiness report.** `computer-use-linux doctor` returns a structured document covering platform, portals, AT-SPI, windowing, input, and a `readiness` summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.
@@ -71,6 +73,10 @@ Targeted `press_key`/`type_text` results append focused-element feedback from AT
 - `activate_window` — focus a window by `window_id`, `pid`, `app_id`, `wm_class`, `title`, or terminal selectors
 - `move_window` / `resize_window` — reposition or resize a window in desktop coordinates (GNOME Shell extension backend); useful to recover windows that are partially off-screen
 
+**Conditional host execution**
+
+- `run_shell` — same-user `/bin/sh -c` execution without login-profile loading, registered only when the server operator starts the MCP process with `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. It is deliberately absent by default and is not a sandbox.
+
 ### MCP safety contract
 
 `computer-use-linux` is not a read-only data source. It can observe the local desktop and, when a mutating tool is called, can change real application state. The `tools/list` response includes MCP `ToolAnnotations` so hosts can surface this distinction before invocation:
@@ -81,9 +87,12 @@ Targeted `press_key`/`type_text` results append focused-element feedback from AT
 | Local setup mutators | `setup_accessibility`, `setup_window_targeting` | `readOnlyHint=false`, `destructiveHint=false`, `idempotentHint=true`; modifies user desktop configuration by enabling accessibility or installing/enabling the GNOME window-targeting extension. |
 | UI state mutators | `activate_window`, `move_window`, `resize_window`, `scroll`, `screenshot` | `readOnlyHint=false`, `destructiveHint=false`; changes focus, geometry, or scroll position in the live desktop, or raises a window to capture it. |
 | Desktop action mutators | `click`, `drag`, `press_key`, `type_text`, `perform_action`, `set_value` | `readOnlyHint=false`, `destructiveHint=true`, `openWorldHint=true`; can trigger arbitrary actions in whatever local application is targeted. |
+| Conditional host-code execution | `run_shell` | Absent unless `COMPUTER_USE_LINUX_ENABLE_SHELL=1`; when enabled, `readOnlyHint=false`, `destructiveHint=true`, `idempotentHint=false`, `openWorldHint=true`. Runs with the MCP server user's host permissions. |
 
 Annotations are safety hints, not an authorization system. MCP hosts should still ask the user before calls that could submit, delete, send, purchase, overwrite, or otherwise commit state.
 
+`run_shell` is an explicit trust-boundary opt-in, not a restricted command runner. Enabling it grants an approved MCP call the same file and network authority as the user running the server. The tool clears the ambient environment and inherits only a small desktop/runtime allowlist (`PATH`, home/user/locale fields, display/session-bus fields); additional variables must be supplied in the visible call payload. Commands use a fixed non-login `/bin/sh`, an existing canonical working directory, a 30-second default / 120-second hard timeout, process-group cleanup, and stderr audit records keyed by the command SHA-256 rather than command text. Collected streams up to 8 MiB are returned with a 512 KiB per-stream response cap and truncation flag; exceeding 8 MiB on either stream fails the call without partial output. These controls bound accidental leakage and runaway work; they do not make arbitrary shell code safe.
+
 The binary also exposes the same capabilities from the CLI for scripting and debugging:
 
 ```
@@ -151,6 +160,7 @@ Then, as needed:
 
 ```bash
 sudo apt install ydotool at-spi2-core         # ydotool 1.0.3+ when using this fallback
+sudo apt install wtype                        # optional Unicode typing on wlroots/Hyprland Wayland
 systemctl --user enable --now ydotoold         # only when doctor selects ydotool
 computer-use-linux setup                      # gsettings AT-SPI bridge
 computer-use-linux setup-window-targeting     # GNOME Shell extension
@@ -233,20 +243,34 @@ Restart Claude Desktop. The tools should appear in the tools list.
 ### Pi Coding Agent
 
 ```bash
-pi install npm:pi-mcp-adapter
 pi install npm:@agent-sh/computer-use-linux
 ```
 
-Restart pi or run `/reload`. The MCP proxy tool `mcp()` will have the desktop tools available:
+Restart Pi or run `/reload`. The package exposes one small loader initially;
+the real tools keep their upstream schemas and are enabled only when Computer
+Use is needed:
+
+Native tools require Pi 0.84.4 or newer (Node.js 22.19 or newer). The
+standalone npm CLI wrapper continues to support Node.js 18 or newer.
+
+```
+computer_use_linux_tools({ tools: ["doctor", "list_windows"] })
+computer_use_linux_doctor({})
+computer_use_linux_list_windows({})
+```
+
+You can also search by capability:
 
 ```
-mcp({ server: "computer-use-linux" })             # list all tools
-mcp({ search: "windows" })                         # search for window tools
-mcp({ tool: "computer_use_linux_doctor" })         # run readiness check
-mcp({ tool: "computer_use_linux_list_windows" })   # list desktop windows
+computer_use_linux_tools({ query: "observe a window and click a control" })
 ```
 
-The extension auto-registers the computer-use-linux MCP server into pi-mcp-adapter's config. If the binary is not found, check the [Pi setup guide](skills/computer-use-linux/references/pi-setup.md).
+No separate MCP adapter or manual MCP configuration is required. Pi starts one
+computer-use-linux process lazily on the first real tool call, reuses it for the
+session so accessibility snapshots remain valid, serializes desktop actions,
+and closes it on reload, session switch, or exit. See the
+[Pi setup guide](skills/computer-use-linux/references/pi-setup.md) for migration
+from older adapter-based installs.
 
 ### Hermes Agent
 
@@ -340,6 +364,7 @@ Most setups need none of these — `doctor` and the installers pick sensible def
 | `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through `ydotool`, skipping the portal and KDE clipboard paths; pointer forcing also skips native-X11 `xdotool` coordinate clicks. |
 | `COMPUTER_USE_LINUX_FORCE_XDOTOOL_KEYBOARD` | Prefer `xdotool`/XTEST keyboard input when `DISPLAY` is available. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_KEYBOARD=1` takes precedence. |
 | `COMPUTER_USE_LINUX_SCREENSHOT_BACKEND` | Force a single screenshot backend, skipping the fallback chain. Accepts `gnome-shell`, `portal`, or `gnome-screenshot`. Pin `gnome-screenshot` for background/systemd contexts where the GNOME Shell and portal DBus paths are denied. |
+| `COMPUTER_USE_LINUX_ENABLE_SHELL` | Set exactly to `1` before starting the MCP server to register the destructive `run_shell` tool. Unset by default. Do not enable for untrusted or unattended MCP hosts. |
 
 **Build-time identity overrides** (set while compiling a downstream embedded
 bundle): `CUL_GNOME_EXTENSION_UUID`, `CUL_DBUS_SERVICE`, and
@@ -361,7 +386,7 @@ files.
 - **Accessibility tree** — [`atspi`](https://crates.io/crates/atspi) crate (tokio backend) talks to the AT-SPI registry on the user session bus. The tree is flattened to `(role, name, text, states, bounds)` tuples and indexed; element indices are stable for the duration of a `get_app_state` snapshot.
 - **DBus where desktops expose it** — [`zbus`](https://crates.io/crates/zbus) for portal calls (`org.freedesktop.portal.Screenshot`, `…RemoteDesktop`, `…ScreenCast`), GNOME Shell screenshots (`org.gnome.Shell.Screenshot`), the bundled GNOME extension's `dev.avifenesh.ComputerUseLinux.WindowControl` service, and temporary KWin scripting.
 - **MCP transport** — [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature; stdio framing, no network.
-- **Input fallback** — on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, when the remote-desktop portal isn't available or the host wants deterministic injection, the binary uses a compatible ydotool 1.0.3+ CLI and `ydotoold` socket, which writes to `/dev/uinput`. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
+- **Input fallback** — on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when installed and the remote-desktop portal is unavailable; `wtype` supports Unicode through the virtual-keyboard protocol on compatible compositors such as Hyprland/wlroots. If wtype is unavailable, the binary falls back to a compatible ydotool 1.0.3+ CLI and `ydotoold` socket. A launched wtype failure is returned without replaying the text. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
 - **Native X11 coordinate clicks** — eligible native X11 sessions use one supervised `xdotool mousemove -- X Y click --repeat N BUTTON` command for left, middle, and right clicks; ydotool is used only when xdotool cannot launch, while a launched nonzero xdotool command is reported as an error without replay. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this xdotool path.
 - **Window registry** — `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, i3 IPC, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
 - **GNOME extension fallback** — recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`.

package/npm/README.md

@@ -12,6 +12,7 @@ desktop actions.
 ```bash
 npm install -g @agent-sh/computer-use-linux
 computer-use-linux doctor
+pi install npm:@agent-sh/computer-use-linux
 hermes skills tap add agent-sh/computer-use-linux
 hermes skills install agent-sh/computer-use-linux/computer-use-linux
 hermes mcp add computer-use-linux --command computer-use-linux --args mcp
@@ -35,6 +36,11 @@ GitHub release for this package version and verifies the `.sha256` asset before
 installing it. It also installs the matching `computer-use-linux-cosmic` helper
 used for COSMIC desktop window targeting.
 
+When installed through Pi, the package supplies native, dynamically loaded
+`computer_use_linux_*` tools. No separate MCP adapter or manual MCP
+configuration is required. Native tools require Pi 0.84.4 or newer; the
+standalone CLI wrapper retains Node.js 18 support.
+
 If you already built or installed the binary yourself, set
 `COMPUTER_USE_LINUX_BIN=/path/to/computer-use-linux` to make the wrapper use
 that executable instead.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.4.9",
+  "version": "0.5.0",
   "description": "Linux desktop-control MCP server: AT-SPI accessibility trees, Wayland/X11 input, screenshots, and compositor window targeting.",
   "license": "MIT",
   "type": "commonjs",
@@ -40,18 +40,41 @@
     "npm/README.md",
     "npm/bin/computer-use-linux.js",
     "npm/install.js",
-    "pi/"
+    "pi/extension/index.ts",
+    "pi/extension/generated-tools.ts",
+    "pi/extension/mcp-client.bundle.cjs",
+    "pi/extension/THIRD_PARTY_NOTICES.txt"
   ],
   "pi": {
     "extensions": ["./pi/extension/index.ts"],
     "skills": ["./skills/computer-use-linux/SKILL.md"]
   },
   "scripts": {
+    "build:pi": "npm run build --prefix pi",
+    "check:pi-bundle": "npm run build:check --prefix pi",
     "postinstall": "node npm/install.js",
-    "pack:check": "npm pack --dry-run",
+    "pack:check": "node scripts/check_npm_package.mjs",
+    "test:pi": "npm test --prefix pi",
+    "typecheck:pi": "npm run typecheck --prefix pi",
     "test:wrapper": "node npm/bin/computer-use-linux.js --help"
   },
   "engines": {
     "node": ">=18"
+  },
+  "peerDependencies": {
+    "@earendil-works/pi-ai": "*",
+    "@earendil-works/pi-coding-agent": "*",
+    "typebox": "*"
+  },
+  "peerDependenciesMeta": {
+    "@earendil-works/pi-ai": {
+      "optional": true
+    },
+    "@earendil-works/pi-coding-agent": {
+      "optional": true
+    },
+    "typebox": {
+      "optional": true
+    }
   }
 }