@agent-sh/computer-use-linux 0.7.29 → 0.7.31

package/README.md

@@ -175,7 +175,7 @@ Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). niri window list
 
 | Desktop/session | Window backend | Notes |
 | --- | --- | --- |
-| GNOME Wayland | GNOME Shell extension first, `org.gnome.Shell.Introspect` fallback | Full target. The extension provides exact window activation when GNOME blocks native introspection; Introspect can list windows and focus apps by `app_id` when allowed. |
+| GNOME Wayland | GNOME Shell extension first, `org.gnome.Shell.Introspect` fallback | Full target. The extension provides exact window activation when GNOME blocks native introspection; Introspect can list windows and verify the current app; activating a window requires the extension. |
 | GNOME X11 | `org.gnome.Shell.Introspect`, then generic X11/EWMH | AT-SPI works; keyboard input prefers `xdotool`/XTEST so the live XKB layout resolves keys correctly. Scroll uses xdotool wheel buttons so GTK 3 does not drop the event after a pointer warp. |
 | KDE Plasma / KWin | temporary KWin DBus scripting | Lists and focuses windows through Plasma 5 or 6 `org.kde.KWin` scripting APIs when the session bus exposes them. |
 | Hyprland | `hyprctl clients -j` and `hyprctl dispatch focuswindow` | Requires `hyprctl` in the desktop session. |
@@ -506,14 +506,14 @@ files.
 - **Limited accessibility feedback** — when exact window focus was verified before input but AT-SPI reports no focused element, the result describes the missing feedback without claiming the input was lost. It still does not verify the field contents. A complete scoped tree containing only application/window containers adds an Electron/Chromium `--force-renderer-accessibility` hint; a truncated tree instead retains its traversal-limit guidance.
 - **Focused containers**: Chromium can mark a `document web` focused alongside its contenteditable editor. The focus probe keeps searching that document and reports the editor; a document that holds focus itself still gets a warning, which no longer claims the text was lost. An editor can carry the `editable` state without the EditableText interface (`supports_editable_text: false`): `type_text` can reach it, `set_value` cannot. Verify its text or a screenshot before submitting or retrying.
 - **Large embedded browser trees**: an app-scoped snapshot includes embedded browser documents, which can use up the node budget before the app's own controls. When `accessibility_tree_truncated` is true, a missing editor proves nothing: raise `max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard cap 64). A busy renderer may give only a few thousand nodes before the deadline. A snapshot stops reading at 10 seconds and returns what it read, marked truncated; if the editor is still missing, hide the embedded browser pane for the snapshot.
-- **Wayland coordinate input** — coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll positions through uinput before sending only wheel events through ydotool. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. Scrolling without a point or window/element target keeps using the current pointer position.
+- **Wayland coordinate input** — coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll positions through uinput before sending only wheel events through ydotool. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. Scrolling without a point or window/element target keeps using the current pointer position. Portal keyboard input preserves the receiving window or app target before consent, then restores and verifies its focus after consent. If focus cannot be verified, no text or key chord is sent. Untargeted portal keyboard calls therefore require working window introspection. GNOME Introspect only verifies an already focused app; if consent changes focus, refocus the app manually or install the Shell extension.
 - **Native X11 coordinate clicks** — eligible native X11 sessions use one supervised `xdotool mousemove -- X Y click --repeat N BUTTON` command for left, middle, and right clicks; ydotool is used only when xdotool cannot launch, while a launched nonzero xdotool command is reported as an error without replay. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this xdotool path.
 - **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. A missing xdotool falls back to ydotool; a launched nonzero xdotool command does not. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.
 - **Native X11 transport.** Native X11 queries (window origins, root screenshots, `doctor`) connect and read replies under one shared 2-second deadline enforced in the transport itself, including the handshake and every reply. A stalled X server closes the query connection instead of leaving a blocked worker behind. Unix sockets, literal IP addresses, and `localhost` need no resolver. A remote hostname in `DISPLAY` needs `getent ahosts` for bounded name resolution; if it is missing or fails, the X11 route is reported as unavailable.
 - **Window registry** — `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
 - **niri backend** — prefers `niri msg`, falling back to direct JSON IPC over the same session socket. `is_minimized` maps to `hidden`. Bounds use `layout.window_size` (or `tile_size`) and, when available, the tile position plus the window's offset and the output's logical origin. Uniform output scaling converts these to device pixels; unknown or mixed scaling omits bounds. Missing window positions stay `null`, so listing and focusing still work without enabling coordinate targeting. An accepted focus action is verified by querying the focused window again.
 - **GNOME extension fallback** — recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`.
-- **COSMIC helper** — `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`.
+- **COSMIC helper** — `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it.
 - **Indicator overlay** — the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening.
 - **Terminal enrichment** — `list_windows` cross-references each terminal window with its controlling TTY and the foreground process on that TTY, so `type_text` / `press_key` can target "the terminal where `pytest` is running" without the host ever knowing the window id.
 

package/npm/README.md

@@ -133,6 +133,8 @@ does not yet forward the flag or include the tool in its catalog.
   disabling accessibility; stopping removes its listener and ends writes without
   disabling other clients or restoring an old value. An app launched during a
   reset may still need restarting, so this is not a complete GNOME toggle fix.
+- **COSMIC Wayland**: Pointer coordinates use per-monitor RemoteDesktop portal streams. Approve the consent dialog before continuing. Available EIS connections wait for device readiness before the first input event. Window bounds support targeted screenshots and relative input, and window IDs fit JavaScript's exact integer range. The toplevel protocol does not expose PID.
+- **Portal keyboard focus**: The receiving window or app target is preserved before consent and its focus is restored and verified afterward. If verification fails, no text or key chord is sent. Untargeted portal keyboard calls require working window introspection. GNOME Introspect can only verify an already focused app; refocus it manually after consent if needed.
 - **GNOME Wayland portal persistence**: On GNOME Wayland the remote-control portal asks again for every new server
   process. Set `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in that process's
   environment to reuse a restore token instead. The first dialog still appears.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.7.29",
+  "version": "0.7.31",
   "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
   "license": "MIT",
   "type": "commonjs",

package/pi/extension/generated-tools.ts

@@ -7,7 +7,7 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.7.29";
+export const GENERATED_SERVER_VERSION = "0.7.31";
 export const GENERATED_TOOL_CATALOG_HASH = "711deb97eb5c8a6cbe0a2fadbb64fdffc7592e304479dbbf277d3fcc4d58210f";
 export const GENERATED_SHELL_TOOL_CATALOG_HASH = "1a927deb2579d314a975afc586bd31010862141779cbe7af50cd76f0387a607c";
 export const GENERATED_MCP_TOOLS =

package/skills/computer-use-linux/SKILL.md

@@ -214,7 +214,8 @@ directly to a window-relative click.
 - `click`, `drag`, `press_key`, `type_text`, `perform_action`, and `set_value` can change real application state.
 - When ydotool is selected, `ydotoold` should run as a per-user service with its socket under `/run/user/$UID`, not as a system-wide service.
 - The optional ydotool backend requires version 1.0.3 or newer; `doctor` rejects older or semantically incompatible CLIs even when `ydotoold` is running.
-- On COSMIC, the standard npm, Cargo, and install-script paths install the `computer-use-linux-cosmic` helper automatically. Manual binary installs must copy it too.
+- On COSMIC, the standard npm, Cargo, and install-script paths install the `computer-use-linux-cosmic` helper automatically. Manual binary installs must copy it too. COSMIC coordinate input uses per-monitor RemoteDesktop portal streams; approve its consent dialog. Available EIS connections wait for resumed devices before sending input. Window bounds support targeted screenshots and relative input. PID is optional because the protocol does not expose it.
+- Portal keyboard input preserves the receiving window or app target before consent and verifies its focus again afterward. Untargeted portal typing and key chords require working window introspection; a failed focus check sends no input. GNOME Introspect can only verify current app focus; use the Shell extension to activate windows.
 - An on-screen indicator shows the user each action: a cursor glides to pointer targets and keycaps or typed text appear for keyboard input. Pointer actions wait about 350 ms so the cursor lands first. The indicator is hidden before screenshots and is never part of captured images.
 
 ## Verification