@agent-sh/computer-use-linux 0.7.24 → 0.7.26

package/README.md

@@ -106,7 +106,9 @@ The preference does not substitute an arbitrary action name for a coordinate
 click. Explicit `x`/`y`, right clicks, and double/multiple clicks retain pointer
 semantics. Re-check application state after either kind of activation.
 
-On KDE Wayland, element clicks that need pointer dispatch match the cached AT-SPI frame to a coherent KWin window/output snapshot, then map the target separately to physical capture pixels (uinput) and logical desktop coordinates (portal). Client-local and decorated frames, actual decoration offsets, and nonzero or negative output origins are supported when the geometry and uniform output scale can be verified. Ambiguous windows, mixed or unknown scales, overlapping outputs, or stale pointer geometry fail before input; explicit coordinates keep their existing behavior.
+On KDE Wayland, element clicks that need pointer dispatch match the cached AT-SPI frame to a coherent KWin window/output snapshot, then map the target separately to physical capture pixels (uinput) and logical desktop coordinates (portal). Client-local and decorated frames, actual decoration offsets, and nonzero or negative output origins are supported when the geometry and uniform output scale can be verified. Ambiguous windows, mixed or unknown scales, overlapping outputs, or stale pointer geometry fail before input.
+
+Explicit `x`/`y` defaults to `coordinate_space: "capture"` (raw screenshot pixels). To click a global AT-SPI screen point, pass `coordinate_space: "atspi"` together with its `element_index` or semantic selector; KDE Wayland then applies the same native KWin decoration and scale conversion as an element-bounds click, including when the AT-SPI window frame is client-local. This mode requires both coordinates and element identity and cannot be combined with `relative: true`. Other calls keep their existing coordinate behavior.
 
 For coordinate `click` or `scroll` with `relative: true`, select a target window
 and measure from its clipped screenshot crop origin. Divide preview `x` and
@@ -116,7 +118,7 @@ the origin. A missing window target is rejected.
 
 **X11 text fidelity:** `xdotool`/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian `br`/ABNT2) or changing keymap state, even when `type_text` returns `ok: true`. Every successful xdotool text dispatch includes a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. PointerRoot focus is unsupported because pointer motion can redirect input without a detectable focus event. This also detects a `setxkbmap` layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Re-observe the actual text before pressing Enter, submitting it, or treating it as delivered. Use `include_tree: true` for AT-SPI readback when the application exposes the text, or `screenshot_after: true` to inspect it. These observations do not automatically compare the field to the requested text. Do not retry blindly.
 
-Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when a concrete non-terminal focus is not editable. Terminals accept keyboard input without an EditableText interface; re-observe the contents to confirm input. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report. It also reports `tree_scoped` (false when no app target narrowed the AT-SPI tree, with a warning in `message`) and `accessibility_tree_truncated` (true when the node, depth, or read budget stopped traversal with unread elements left).
+Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when a concrete non-terminal focus is not editable. Terminals accept keyboard input without an EditableText interface; re-observe the contents to confirm input. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report. It also reports `tree_scoped` (false when no app target narrowed the AT-SPI tree, with a warning in `message`) and `accessibility_tree_truncated` (true when the node, depth, read, or time budget stopped traversal with unread elements left).
 
 An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop.
 
@@ -501,6 +503,8 @@ files.
 - **Input fallback** — on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when installed and the remote-desktop portal is unavailable; `wtype` supports Unicode through the virtual-keyboard protocol on compatible compositors such as Hyprland/wlroots. If wtype is unavailable, the binary falls back to a compatible ydotool 1.0.3+ CLI and `ydotoold` socket. A launched wtype failure is returned without replaying the text. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
 - **Hyprland/Electron text** — for classified nonsecret fields on Hyprland, wtype passes one literal argument after `--` to avoid stdin keymap-refresh corruption. Argument text is visible in process listings; confirmed password fields are rejected. Terminals and ambiguous fields (with or without AT-SPI) use private stdin for at most 100 Unicode characters per call; split larger input into smaller calls or expose accessibility. The input probe has its own 1.5-second budget; the 250 ms indicator probe only masks text. Other compatible compositors retain their existing stdin path. NUL and text over 131,071 UTF-8 bytes are rejected before focus or indicators. Success confirms dispatch only; re-observe the field before retrying.
 - **Limited accessibility feedback** — when exact window focus was verified before input but AT-SPI reports no focused element, the result describes the missing feedback without claiming the input was lost. It still does not verify the field contents. A complete scoped tree containing only application/window containers adds an Electron/Chromium `--force-renderer-accessibility` hint; a truncated tree instead retains its traversal-limit guidance.
+- **Focused containers**: Chromium can mark a `document web` focused alongside its contenteditable editor. The focus probe keeps searching that document and reports the editor; a document that holds focus itself still gets a warning, which no longer claims the text was lost. An editor can carry the `editable` state without the EditableText interface (`supports_editable_text: false`): `type_text` can reach it, `set_value` cannot. Verify its text or a screenshot before submitting or retrying.
+- **Large embedded browser trees**: an app-scoped snapshot includes embedded browser documents, which can use up the node budget before the app's own controls. When `accessibility_tree_truncated` is true, a missing editor proves nothing: raise `max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard cap 64). Larger trees cost more context, and a busy renderer may give only a few thousand nodes before the deadline. A snapshot stops reading at 10 seconds and returns what it read, marked truncated; if the editor is still missing, hide the embedded browser pane for the snapshot.
 - **Wayland coordinate input** — coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll positions through uinput before sending only wheel events through ydotool. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. Scrolling without a point or window/element target keeps using the current pointer position.
 - **Native X11 coordinate clicks** — eligible native X11 sessions use one supervised `xdotool mousemove -- X Y click --repeat N BUTTON` command for left, middle, and right clicks; ydotool is used only when xdotool cannot launch, while a launched nonzero xdotool command is reported as an error without replay. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this xdotool path.
 - **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. A missing xdotool falls back to ydotool; a launched nonzero xdotool command does not. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.

package/npm/README.md

@@ -151,7 +151,13 @@ does not yet forward the flag or include the tool in its catalog.
   avoiding pointer-coordinate conversion when that action is available. Explicit
   `activate`/`jump` requests belong in `perform_action`, not `click`, even when
   bounds are missing. Explicit
-  `x`/`y`, right clicks, and multi-clicks retain pointer semantics. For coordinate
+  `x`/`y`, right clicks, and multi-clicks retain pointer semantics. Explicit
+  coordinates default to raw screenshot pixels (`coordinate_space: "capture"`).
+  On KDE Wayland, `coordinate_space: "atspi"` with `element_index` or a semantic
+  selector converts a global AT-SPI screen point through the element's KWin window,
+  decoration, output origin, and display scale, even with a client-local AT-SPI
+  window frame. It requires both `x`/`y` and element identity
+  and cannot be combined with `relative: true`. For coordinate
   `click` or `scroll` with `relative: true`, use the clipped target-window
   screenshot crop origin and divide preview coordinates by screenshot `scale`.
   Do not use widget-local or raw GDK surface coordinates. A window target is
@@ -178,3 +184,18 @@ does not yet forward the flag or include the tool in its catalog.
   content makes the feedback informational. Such trees get a conditional
   Electron/Chromium hint computed before compaction; this is not positive Electron
   detection.
+
+  Chromium can mark a `document web` focused alongside its contenteditable
+  editor. The focus probe keeps searching that document and reports the editor;
+  a document that holds focus itself still gets a warning, which no longer
+  claims the text was lost. An editor can carry the `editable` state without the
+  EditableText interface (`supports_editable_text: false`): `type_text` can
+  reach it, `set_value` cannot. Verify its text or a screenshot before
+  submitting or retrying. Embedded browser documents can use up an app-scoped
+  snapshot's node budget before the app's own controls; when
+  `accessibility_tree_truncated` is true, a missing editor proves nothing. Raise
+  `max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard
+  cap 64); larger trees cost more context, and a busy renderer may give only a
+  few thousand nodes before the deadline. A snapshot stops reading at 10 seconds
+  and returns what it read, marked truncated; if the editor is still missing,
+  hide the embedded browser pane for the snapshot.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.7.24",
+  "version": "0.7.26",
   "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
   "license": "MIT",
   "type": "commonjs",

package/pi/extension/generated-tools.ts

@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.7.24";
-export const GENERATED_TOOL_CATALOG_HASH = "dee39a43a4d642b61b861f4aa90cdabc9a416befbc10d463a005bcbb892eaf14";
-export const GENERATED_SHELL_TOOL_CATALOG_HASH = "b78f54e50d3a2d615af92ee2a2373599684f02a802b06f1226981f985e34f456";
+export const GENERATED_SERVER_VERSION = "0.7.26";
+export const GENERATED_TOOL_CATALOG_HASH = "ec0af1cf8525312554ae5b698f6cb6629190b189afe77ee1a8c3356d9dded33c";
+export const GENERATED_SHELL_TOOL_CATALOG_HASH = "ca71b492f3682185e6b1ff1e0b9e9e95ef07ce1087c2bbfdcf05b99284cb3112";
 export const GENERATED_MCP_TOOLS =
 [
   {
@@ -187,9 +187,16 @@ export const GENERATED_MCP_TOOLS =
       "openWorldHint": true,
       "readOnlyHint": false
     },
-    "description": "Click an element by index, semantic selector, or desktop coordinate pixels from screenshot metadata. Plain left activation prefers a native AT-SPI click/press/toggle action, avoiding toolkit coordinate scaling. Entry activate and slider jump actions are not substituted for pointer clicks. Explicit coordinates, right clicks, and multi-clicks retain pointer semantics. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
+    "description": "Click an element by index, semantic selector, or desktop coordinate pixels from screenshot metadata. Plain left activation prefers a native AT-SPI click/press/toggle action, avoiding toolkit coordinate scaling. Entry activate and slider jump actions are not substituted for pointer clicks. Explicit coordinates, right clicks, and multi-clicks retain pointer semantics. Explicit x/y default to raw screenshot capture pixels; coordinate_space=\"atspi\" with element_index or a semantic selector maps a global AT-SPI screen point through the element's KWin window and output geometry on KDE Wayland. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
     "inputSchema": {
       "$defs": {
+        "ClickCoordinateSpace": {
+          "enum": [
+            "capture",
+            "atspi"
+          ],
+          "type": "string"
+        },
         "ScreenshotOutputFormat": {
           "enum": [
             "png",
@@ -222,6 +229,11 @@ export const GENERATED_MCP_TOOLS =
             "null"
           ]
         },
+        "coordinate_space": {
+          "$ref": "#/$defs/ClickCoordinateSpace",
+          "default": "capture",
+          "description": "Pixel space for explicit x/y: raw screenshot capture pixels (default),\nor global AT-SPI screen coordinates mapped through the element's KWin window."
+        },
         "element_index": {
           "default": null,
           "minimum": 0,
@@ -631,7 +643,7 @@ export const GENERATED_MCP_TOOLS =
         },
         "max_nodes": {
           "default": null,
-          "description": "Maximum raw AT-SPI nodes to inspect before compaction (default 1000, hard max 2000).",
+          "description": "Maximum raw AT-SPI nodes to inspect before compaction (default 1000, hard max 10000).",
           "minimum": 0,
           "type": [
             "integer",

package/skills/computer-use-linux/SKILL.md

@@ -134,7 +134,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
 ## Procedure
 
 1. In Pi, call `computer_use_linux_tools` with the exact tools or capability you need. Enabled tools use the `computer_use_linux_<name>` prefix, appear starting on the next model turn, and remain active for the session.
-2. Begin every desktop-control turn with `get_app_state`, scoped to the app you are working in: pass `app_name_or_bundle_identifier` or a window target (`window_id`, `pid`, `app_id`, `wm_class`, `title`). An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop. Without a target the result is the whole desktop AT-SPI tree, `tree_scoped` is `false`, and `message` warns; that can flood context. Use `include_screenshot: false` when the accessibility tree is sufficient. If `accessibility_tree_truncated` is `true`, the tree is incomplete: scope to a narrower target and raise `max_nodes` or `max_depth` (hard caps 2000 and 64) rather than lowering them. The compact readiness block identifies missing setup.
+2. Begin every desktop-control turn with `get_app_state`, scoped to the app you are working in: pass `app_name_or_bundle_identifier` or a window target (`window_id`, `pid`, `app_id`, `wm_class`, `title`). An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop. Without a target the result is the whole desktop AT-SPI tree, `tree_scoped` is `false`, and `message` warns; that can flood context. Use `include_screenshot: false` when the accessibility tree is sufficient. If `accessibility_tree_truncated` is `true`, the tree is incomplete: scope to a narrower target and raise `max_nodes` or `max_depth` (hard caps 10000 and 64) rather than lowering them. The compact readiness block identifies missing setup.
 3. Use `doctor` only when you need the full diagnostic report.
 4. If `can_build_accessibility_tree` is false, run `setup_accessibility` and restart the target app.
 5. If `can_query_windows` is false on GNOME Wayland, run `setup_window_targeting` and ask the user to log out and back in if setup says the shell extension needs a reload.
@@ -164,11 +164,32 @@ content makes the feedback informational. Such trees get a conditional
 Electron/Chromium hint computed before compaction; this is not positive Electron
 detection.
 
+Chromium can mark a `document web` focused alongside its contenteditable editor.
+The focus probe keeps searching that document and reports the editor; a document
+that holds focus itself still gets a warning, which no longer claims the text
+was lost. An editor can carry the `editable` state without the EditableText
+interface (`supports_editable_text: false`): `type_text` can reach it,
+`set_value` cannot. Verify its text or a screenshot before submitting or
+retrying. Embedded browser documents can use up an app-scoped snapshot's node
+budget before the app's own controls; when `accessibility_tree_truncated` is
+true, a missing editor proves nothing. Raise `max_nodes` (default 1000, hard cap
+10000) and, if needed, `max_depth` (hard cap 64); larger trees cost more
+context, and a busy renderer may give only a few thousand nodes before the
+deadline. A snapshot stops reading at 10 seconds and returns what it read,
+marked truncated; if the editor is still missing, hide the embedded browser pane
+for the snapshot.
+
 Plain left element/index/selector `click` prefers native AT-SPI `click`,
 `press`, or `toggle` over toolkit bounds, avoiding coordinate
 conversion when available. This preference does not replace a coordinate click
 with an arbitrary action name. Explicit `x`/`y`, right clicks, and double/multiple
-clicks retain pointer semantics.
+clicks retain pointer semantics. Explicit coordinates default to raw screenshot
+pixels (`coordinate_space: "capture"`). On KDE Wayland, pass
+`coordinate_space: "atspi"` with `element_index` or a semantic selector to map
+a global AT-SPI screen point through the element's KWin window, decoration,
+output origin, and display scale, even with a client-local AT-SPI window frame.
+This requires both `x`/`y` and element identity; do not
+combine it with `relative: true` or use it for ordinary screenshot pixels.
 Use `perform_action` explicitly for entry `activate` or slider `jump`; `click`
 never substitutes those actions, including when bounds are unavailable.