@agent-sh/computer-use-linux 0.7.34 → 0.7.35

package/README.md

@@ -11,6 +11,9 @@
 
 > Running open models for your company? [**Tiyuvta**](https://tiyuvta.ai/services/) helps with model choice, deployment and optimization, and fine-tuning on your hardware or cloud account.
 
+> Built and maintained with a lot of agent tokens and hours.
+> If it saves you time, [sponsoring](https://github.com/sponsors/avifenesh) helps keep it going.
+
 `computer-use-linux` gives AI agents real, structured control of a local Linux desktop: semantic AT-SPI accessibility trees, bounded screenshots, compositor-aware window targeting, Wayland/X11 input synthesis, single-round-trip post-action verification, and a click-through on-screen activity indicator across **GNOME, KDE Plasma/KWin, Hyprland, niri, COSMIC, i3, labwc, and X11/EWMH**.
 
 ## Quick start
@@ -180,7 +183,7 @@ Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). niri window list
 | GNOME Wayland | GNOME Shell extension first, `org.gnome.Shell.Introspect` fallback | Full target. The extension provides exact window activation when GNOME blocks native introspection; Introspect can list windows and verify the current app; activating a window requires the extension. |
 | GNOME X11 | `org.gnome.Shell.Introspect`, then generic X11/EWMH | AT-SPI works; keyboard input prefers `xdotool`/XTEST so the live XKB layout resolves keys correctly. Scroll uses xdotool wheel buttons so GTK 3 does not drop the event after a pointer warp. |
 | KDE Plasma / KWin | temporary KWin DBus scripting | Lists and focuses windows through Plasma 5 or 6 `org.kde.KWin` scripting APIs when the session bus exposes them. IDs are derived from the KWin window UUID, stay the same for a window across calls, and fit JavaScript integers. |
-| Hyprland | `hyprctl clients -j` and `hyprctl dispatch focuswindow` | Requires `hyprctl` in the desktop session. |
+| Hyprland | `hyprctl clients -j` and `hyprctl dispatch focuswindow` | Requires `hyprctl` in the desktop session. With `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, targeted keyboard input uses `send_key_state` and targeted capture uses `grim -T`, so neither focuses the window. |
 | niri | `niri msg` with a direct JSON IPC fallback | Lists toplevels and focuses exact window IDs. Export the session's `NIRI_SOCKET`; otherwise discovery requires an unambiguous socket matching `WAYLAND_DISPLAY`. The fallback works without the `niri` binary. Missing positions remain `null`; bounds are omitted if output scaling is unknown or mixed. |
 | i3 | `i3-msg`; optional `xprop` for PID hydration | Lists and focuses i3 windows over the active i3 IPC socket. |
 | labwc with Selkies IPC | `$XDG_RUNTIME_DIR/labwc.sock` | Requires the Selkies IPC patch; stock labwc has no such socket. Lists windows and focuses unique app-id/title matches through `wlr-foreign-toplevel-management`. IDs include pid, app id and title, fit JavaScript integers, and change when the title changes. Ambiguous windows are rejected. Bounds are omitted because the IPC does not expose output scaling. |
@@ -476,6 +479,7 @@ Most setups need none of these — `doctor` and the installers pick sensible def
 | `CU_DISABLE_ABS_POINTER` | Disable the uinput absolute pointer. Wayland coordinate actions then need the RemoteDesktop portal; they refuse input rather than use ydotool's relative-motion approximation. Native X11 fallback is unchanged. |
 | `COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER` / `…_KEYBOARD` | Prefer the RemoteDesktop portal on Wayland, skipping auto-detection. Pointer forcing skips the absolute uinput backend. Forced ydotool or XTEST keyboard overrides take precedence; KDE literal text keeps its clipboard route. Portal startup and text conversion failures return errors without replaying through ydotool. |
 | `COMPUTER_USE_LINUX_PORTAL_SCROLL_INVERT` | Set to `1` to reverse vertical scrolling through the RemoteDesktop portal. Defaults to normal direction on both GNOME and KDE Plasma; horizontal direction is unchanged. |
+| `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND` | Set exactly to `1` on Hyprland to work on a window without focusing it. `type_text` and `press_key` calls that name a target window send keys to it through Hyprland's `send_key_state` dispatcher instead of focusing it and typing, and `get_app_state` and `screenshot` capture an unfocused target alone with `grim -T` instead of raising it or using the portal. The user's focus, cursor and workspace do not change, so the target can sit on a hidden workspace. Text is mapped for a US layout: other keymaps and characters outside printable ASCII, tab and newline are refused before any key is sent. Browser accelerators such as Ctrl+R are ignored by an unfocused Chromium. Unset, input focuses the target as before. |
 | `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP` | Set exactly to `1` to ask the RemoteDesktop portal to remember pointer and keyboard grants across processes. The first dialog still appears (on GNOME the remember box starts checked). Later processes reuse the single-use restore token from `Start`, stored mode `0600` under `$XDG_STATE_HOME/computer-use-linux/` (or `~/.local/state/computer-use-linux/`). Pointer and keyboard tokens are separate. Needs interface version 2. Unset, every new process is prompted. No effect when input is not using the portal. |
 | `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through `ydotool`, skipping the portal and KDE clipboard paths; pointer forcing also skips the absolute uinput backend and native-X11 `xdotool` clicks, scrolls and drags. Coordinate pointer actions on Wayland then refuse input because ydotool cannot position them accurately; scrolling at the current pointer remains available. |
 | `YDOTOOL_SOCKET` | Explicit ydotool socket path. When unset, only an owner-only `.ydotool_socket` in the user's private runtime directory is accepted; the legacy shared `/tmp/.ydotool_socket` is never selected automatically. Setting this variable is an operator trust override for custom layouts. |
@@ -507,6 +511,7 @@ files.
 - **DBus where desktops expose it** — [`zbus`](https://crates.io/crates/zbus) for portal calls (`org.freedesktop.portal.Screenshot`, `…RemoteDesktop`, `…ScreenCast`), GNOME Shell screenshots (`org.gnome.Shell.Screenshot`), the bundled GNOME extension's `dev.avifenesh.ComputerUseLinux.WindowControl` service, and temporary KWin scripting.
 - **MCP transport** — [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature; stdio framing, no network.
 - **Input fallback** — on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when installed and the remote-desktop portal is unavailable; `wtype` supports Unicode through the virtual-keyboard protocol on compatible compositors such as Hyprland/wlroots. If wtype is unavailable, the binary falls back to a compatible ydotool 1.0.3+ CLI and `ydotoold` socket. A launched wtype failure is returned without replaying the text. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
+- **Hyprland background mode** — with `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1`, a targeted `type_text` or `press_key` is sent as `hl.dsp.send_key_state` down/up pairs for the target's address, written straight to Hyprland's control socket in one `[[BATCH]]` request per 64 key presses (never through `hyprctl` arguments, so typed text cannot be read from `/proc/<pid>/cmdline`), and the window is never focused. Hyprland answers each dispatch, so when every dispatch in a failed batch was answered the error states how many key presses were delivered (and whether a key was left down), and the caller can send only the remainder. An empty or incomplete reply, a timeout, or an ok after an error proves nothing about what ran, so the error gives the confirmed count and says the rest may have been delivered. The action indicator shows the input; it is masked unless an AT-SPI probe of the target app finds a non-secret editable field or a terminal. The dispatcher resolves a key name to its unshifted physical key and applies only the modifiers it is given, so text goes through a US-layout table (shifted symbols become SHIFT plus the base key); unmappable characters or a non-US main keymap refuse the whole call. An unfocused target is captured with `grim -T <stableId>` (ext-foreign-toplevel image capture). Hyprland stops sending frame callbacks to windows on hidden workspaces, so their buffer holds the last frame drawn on screen; each capture makes the client repaint, so a hidden-workspace target is captured every 150 ms until two consecutive frames match (at most 8 captures) and the result's `source` says whether it settled.
 - **Hyprland/Electron text** — for classified nonsecret fields on Hyprland, wtype passes one literal argument after `--` to avoid stdin keymap-refresh corruption. Argument text is visible in process listings; confirmed password fields are rejected. Terminals and ambiguous fields (with or without AT-SPI) use private stdin for at most 100 Unicode characters per call; split larger input into smaller calls or expose accessibility. The input probe has its own 1.5-second budget; the 250 ms indicator probe only masks text. Other compatible compositors retain their existing stdin path. NUL and text over 131,071 UTF-8 bytes are rejected before focus or indicators. Success confirms dispatch only; re-observe the field before retrying.
 - **Limited accessibility feedback** — when exact window focus was verified before input but AT-SPI reports no focused element, the result describes the missing feedback without claiming the input was lost. It still does not verify the field contents. A complete scoped tree containing only application/window containers adds an Electron/Chromium `--force-renderer-accessibility` hint; a truncated tree instead retains its traversal-limit guidance.
 - **Focused containers**: Chromium can mark a `document web` focused alongside its contenteditable editor. The focus probe keeps searching that document and reports the editor; a document that holds focus itself still gets a warning, which no longer claims the text was lost. An editor can carry the `editable` state without the EditableText interface (`supports_editable_text: false`): `type_text` can reach it, `set_value` cannot. Verify its text or a screenshot before submitting or retrying.

package/npm/README.md

@@ -182,6 +182,10 @@ does not yet forward the flag or include the tool in its catalog.
 - **X11 text input**: X11 text input sends LF and CRLF line endings as Return. Focused terminals
   accept keyboard input even when AT-SPI does not expose EditableText. Focus
   feedback describes the target; re-observe its contents to confirm input.
+- **Hyprland background mode**: Set `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` to type into and
+  capture a targeted window without focusing it (`send_key_state` and `grim -T`), so the
+  target can stay on a hidden workspace. Text is mapped for a US layout; other keymaps
+  and non-ASCII characters are refused before any key is sent.
 - **Hyprland `type_text`**: On Hyprland, `type_text` passes one literal wtype argument after `--` for a
   classified nonsecret field, avoiding stdin keymap-refresh corruption in Electron.
   Argument text is visible in process listings, so confirmed password roles are

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.7.34",
+  "version": "0.7.35",
   "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
   "license": "MIT",
   "type": "commonjs",

package/pi/extension/generated-tools.ts

@@ -7,7 +7,7 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.7.34";
+export const GENERATED_SERVER_VERSION = "0.7.35";
 export const GENERATED_TOOL_CATALOG_HASH = "88f353a580174a633cfc2a501dc19aca3913d1ec317d33fb38e8ac1d2bcd40d4";
 export const GENERATED_SHELL_TOOL_CATALOG_HASH = "52d70dfdec115799ca570e3419ccddecdd3d170d36492903fc8796fdb14037a9";
 export const GENERATED_MCP_TOOLS =

package/skills/computer-use-linux/SKILL.md

@@ -214,6 +214,7 @@ directly to a window-relative click.
 - GNOME may show a portal prompt on the first screenshot or `get_app_state` call with screenshots enabled.
 - Inside a Selkies container, `PIXELFLUX_CU` plus `PIXELFLUX_CU_TOKEN` sends screenshots and input to pixelflux on loopback, with no portal prompt. A closed port falls through. A server error does not, so do not retry that call through another backend.
 - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
+- On Hyprland, `COMPUTER_USE_LINUX_HYPRLAND_BACKGROUND=1` lets targeted `type_text`, `press_key`, `get_app_state` and `screenshot` work on a window without focusing it, including one on a hidden workspace. Pass `window_id` (or another target) on every call. An AT-SPI action can move focus inside the app, so re-focus a field with its `activate` action before typing into it. Text must be printable US-layout ASCII, tab or newline. An unfocused Chromium ignores browser accelerators such as Ctrl+R, and Chromium joins AT-SPI only when started with `ACCESSIBILITY_ENABLED=1` if the accessibility bus reports itself disabled.
 - Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
 - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.
 - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.