@agent-sh/computer-use-linux 0.7.32 → 0.7.33

package/README.md

@@ -479,7 +479,9 @@ Most setups need none of these — `doctor` and the installers pick sensible def
 | `YDOTOOL_SOCKET` | Explicit ydotool socket path. When unset, only an owner-only `.ydotool_socket` in the user's private runtime directory is accepted; the legacy shared `/tmp/.ydotool_socket` is never selected automatically. Setting this variable is an operator trust override for custom layouts. |
 | `COMPUTER_USE_LINUX_FORCE_XDOTOOL_KEYBOARD` | Prefer `xdotool`/XTEST keyboard input when `DISPLAY` is available. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_KEYBOARD=1` takes precedence. |
 | `COMPUTER_USE_LINUX_XDOTOOL_TYPE_DELAY_MS` | Per-character delay for `xdotool type` in milliseconds (default `12`). `0` is faster but can deliver characters out of order on some X servers. |
-| `COMPUTER_USE_LINUX_SCREENSHOT_BACKEND` | Force a single screenshot backend, skipping the fallback chain. Accepts `gnome-shell`, `portal`, `x11`, or `gnome-screenshot`. `x11` works only on a native X11 session. Pin `gnome-screenshot` for background/systemd contexts where the GNOME Shell and portal DBus paths are denied. |
+| `COMPUTER_USE_LINUX_SCREENSHOT_BACKEND` | Force a single screenshot backend, skipping the fallback chain. Accepts `gnome-shell`, `portal`, `x11`, `gnome-screenshot`, or `pixelflux`. `x11` works only on a native X11 session. `pixelflux` needs `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN`. Pin `gnome-screenshot` for background/systemd contexts where the GNOME Shell and portal DBus paths are denied. |
+| `PIXELFLUX_CU` | Loopback port of a Selkies pixelflux computer-use server (`5000` or `127.0.0.1:5000`). With `PIXELFLUX_CU_TOKEN`, screenshots try pixelflux before the other backends, and click, scroll, drag, press_key, and type_text go there without a portal prompt. A closed port falls through. A server error does not. Anything other than loopback is ignored. |
+| `PIXELFLUX_CU_TOKEN` | Bearer token for `PIXELFLUX_CU`. Never logged. Both variables must be set. |
 | `COMPUTER_USE_LINUX_ENABLE_SHELL` | Set exactly to `1` before starting the MCP server to register the destructive `run_shell` tool. Unset by default. Do not enable for untrusted or unattended MCP hosts. |
 
 **Build-time identity overrides** (set while compiling a downstream embedded
@@ -507,6 +509,7 @@ files.
 - **Limited accessibility feedback** — when exact window focus was verified before input but AT-SPI reports no focused element, the result describes the missing feedback without claiming the input was lost. It still does not verify the field contents. A complete scoped tree containing only application/window containers adds an Electron/Chromium `--force-renderer-accessibility` hint; a truncated tree instead retains its traversal-limit guidance.
 - **Focused containers**: Chromium can mark a `document web` focused alongside its contenteditable editor. The focus probe keeps searching that document and reports the editor; a document that holds focus itself still gets a warning, which no longer claims the text was lost. An editor can carry the `editable` state without the EditableText interface (`supports_editable_text: false`): `type_text` can reach it, `set_value` cannot. Verify its text or a screenshot before submitting or retrying.
 - **Large embedded browser trees**: an app-scoped snapshot includes embedded browser documents, which can use up the node budget before the app's own controls. When `accessibility_tree_truncated` is true, a missing editor proves nothing: raise `max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard cap 64). A busy renderer may give only a few thousand nodes before the deadline. A snapshot stops reading at 10 seconds and returns what it read, marked truncated; if the editor is still missing, hide the embedded browser pane for the snapshot.
+- **Selkies pixelflux**: when `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` are set, screenshots and pointer/keyboard actions use the container's loopback computer-use server first. Element clicks scale AT-SPI bounds by the latest screenshot size over the logical layout (`wlr-output-management` on labwc, the existing desktop layout elsewhere). Coordinate clicks use screenshot pixels. A connection failure falls through to the backends below; an HTTP error does not.
 - **Wayland coordinate input** — coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll positions through uinput before sending only wheel events through ydotool. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. Scrolling without a point or window/element target keeps using the current pointer position. Portal keyboard input preserves the receiving window or app target before consent, then restores and verifies its focus after consent. If focus cannot be verified, no text or key chord is sent. Untargeted portal keyboard calls therefore require working window introspection. GNOME Introspect only verifies an already focused app; if consent changes focus, refocus the app manually or install the Shell extension.
 - **Native X11 coordinate clicks.** Eligible native X11 sessions use one supervised `xdotool mousemove -- X Y click --repeat N BUTTON` command for left, middle, and right clicks; native X11 does not initialize the system-wide uinput device. Missing or failed xdotool commands return an error without routing input to another display. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this xdotool path.
 - **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. Missing or failed xdotool commands return an error without a system-wide fallback. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.

package/npm/README.md

@@ -168,6 +168,7 @@ does not yet forward the flag or include the tool in its catalog.
   screenshot crop origin and divide preview coordinates by screenshot `scale`.
   Do not use widget-local or raw GDK surface coordinates. A window target is
   required.
+- **Selkies containers**: set `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` so screenshots and input use the container's pixelflux computer-use server. No portal prompt. A closed port falls through; a server error does not.
 - **X11 layout risk**: xdotool/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian `br`/ABNT2) or changing keymap state, even with `ok: true`. Successful xdotool text dispatches include a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. This also detects a `setxkbmap` layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Inspect the actual text before pressing Enter, submitting it, or treating it as delivered. Use `include_tree: true` when AT-SPI exposes the text, or `screenshot_after: true`; neither automatically compares the field with the request. Do not retry blindly.
 - **X11 text input**: X11 text input sends LF and CRLF line endings as Return. Focused terminals
   accept keyboard input even when AT-SPI does not expose EditableText. Focus

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.7.32",
+  "version": "0.7.33",
   "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
   "license": "MIT",
   "type": "commonjs",

package/pi/extension/generated-tools.ts

@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.7.32";
-export const GENERATED_TOOL_CATALOG_HASH = "379903a7c526e5517b75cdc50ac9319aaadf0df01441bd96a87ab46787d26ae7";
-export const GENERATED_SHELL_TOOL_CATALOG_HASH = "b08763dff65cdc5388c5202df72e0399f6809a8f8144fa7d37281df5feea2308";
+export const GENERATED_SERVER_VERSION = "0.7.33";
+export const GENERATED_TOOL_CATALOG_HASH = "94d518f3939f7c36842230f687f3246d39f391b542cf5254908a2ea6ecfde5bc";
+export const GENERATED_SHELL_TOOL_CATALOG_HASH = "3343d325e57fd67036793a7c862e855b52e81655628d86f82258803892a04c16";
 export const GENERATED_MCP_TOOLS =
 [
   {

package/skills/computer-use-linux/SKILL.md

@@ -212,6 +212,7 @@ directly to a window-relative click.
 
 - Already-running GTK, Qt, and Electron apps may need a restart after AT-SPI is enabled.
 - GNOME may show a portal prompt on the first screenshot or `get_app_state` call with screenshots enabled.
+- Inside a Selkies container, `PIXELFLUX_CU` plus `PIXELFLUX_CU_TOKEN` sends screenshots and input to pixelflux on loopback, with no portal prompt. A closed port falls through. A server error does not, so do not retry that call through another backend.
 - On GNOME Wayland, the remote-control portal asks on each new `computer-use-linux` process. `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` in the server environment asks the portal to remember the grant (GNOME starts with the remember box checked). Later processes reuse a single-use restore token stored mode 0600 in the user state directory. Unset is the default and prompts every time. The flag does nothing when ydotool, not the portal, is the input backend.
 - Desktop input is stateful. Avoid concurrent tool calls against this MCP server.
 - Pi serializes the native Computer Use tools and keeps one process for the session. If that process exits, do not replay an ambiguous mutating call; obtain a fresh `get_app_state` before another element-based action.