@agent-sh/computer-use-linux 0.7.40 → 0.7.42
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md
CHANGED
|
@@ -124,7 +124,7 @@ the origin. A missing window target is rejected.
|
|
|
124
124
|
|
|
125
125
|
**X11 text fidelity:** `xdotool`/XTEST can silently substitute symbols under non-US XKB layouts (reported with Brazilian `br`/ABNT2) or changing keymap state, even when `type_text` returns `ok: true`. Every successful xdotool text dispatch includes a layout warning. XKB layout replacement events, persistent mapping/group changes, and keyboard focus/grab transitions detected across a call return an error after dispatch, without replaying the text. Keyboard grabs can divert characters without changing the polled focused-window ID. If monitoring cannot start, xdotool still dispatches and the response explicitly warns that layout-change and keyboard interception detection were unavailable. PointerRoot focus is unsupported because pointer motion can redirect input without a detectable focus event. This also detects a `setxkbmap` layout changed and restored during one call. Temporary raw remaps or group switches restored within a call and dropped text with a stable map remain unverified. Re-observe the actual text before pressing Enter, submitting it, or treating it as delivered. Use `include_tree: true` for AT-SPI readback when the application exposes the text, or `screenshot_after: true` to inspect it. These observations do not automatically compare the field to the requested text. Do not retry blindly.
|
|
126
126
|
|
|
127
|
-
Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when a concrete non-terminal focus is not editable. Terminals accept keyboard input without an EditableText interface; re-observe the contents to confirm input. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report. Tree nodes use a compact encoding: null, empty, zero and false fields are left out, containers do not carry the U+FFFC placeholder text Chromium gives them, and `actions` lists action names (pass one as `perform_action`'s `action`). `verbose: true` returns the full encoding, with every field and action descriptions and keybindings. `interactive_only: true` returns only nodes with an action, editable text or a value, plus headings, status and live regions, windows and dialogs; containers are collapsed, so `parent_index` names the nearest kept ancestor. Its `element_index` values are the same as in the full tree. When `title` or `window_id` names one of several windows under one AT-SPI application root (Chromium puts every browser window under one), the tree covers that window plus the app's popups and dialogs, and drops its other named windows; `message` says so. It also reports `tree_scoped` (false when no app target narrowed the AT-SPI tree, with a warning in `message`) and `accessibility_tree_truncated` (true when the node, depth, read, or time budget stopped traversal with unread elements left). When a result would pass 32,000 bytes, the tree comes back as its leading nodes plus `accessibility_tree_spill` (`path`, `node_count`, `bytes`, `inline_node_count`). `path` is a private JSON file holding the whole tree with the same `element_index` values, in `$XDG_RUNTIME_DIR/computer-use-linux/trees` when that runtime directory is private, otherwise under `~/.cache` (directories 0700, file 0600). Search the file instead of requesting the tree again; hosts that cannot read files page through the same tree with `get_tree_page`. The file lasts until the next snapshot or until the server exits; after a crash it is swept at the next start or after an hour.
|
|
127
|
+
Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when a concrete non-terminal focus is not editable. Terminals accept keyboard input without an EditableText interface; re-observe the contents to confirm input. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. A click or scroll coordinate between monitors, where monitors of different sizes or offset rows leave part of the screenshot uncovered, gets a warning too: the server reads the monitor layout once per screenshot (from the portal session, KWin, the GNOME Shell extension, Hyprland, kscreen, Sway, COSMIC, niri or wlr-output-management, and from RandR on X11 only) and checks the point in screenshot pixels. Without a layout that fits the screenshot, only its outer rectangle is checked. The warning says what the route did. The portal sends nothing. X11 keeps the pointer on the screen, so a point off the screenshot lands at an edge. On X11 and the uinput pointer, a point between monitors lands at a monitor edge or on no visible pixel. A failed X11 dispatch gets no warning, since nothing was sent. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report. Tree nodes use a compact encoding: null, empty, zero and false fields are left out, containers do not carry the U+FFFC placeholder text Chromium gives them, and `actions` lists action names (pass one as `perform_action`'s `action`). `verbose: true` returns the full encoding, with every field and action descriptions and keybindings. `interactive_only: true` returns only nodes with an action, editable text or a value, plus headings, status and live regions, windows and dialogs; containers are collapsed, so `parent_index` names the nearest kept ancestor. Its `element_index` values are the same as in the full tree. When `title` or `window_id` names one of several windows under one AT-SPI application root (Chromium puts every browser window under one), the tree covers that window plus the app's popups and dialogs, and drops its other named windows; `message` says so. It also reports `tree_scoped` (false when no app target narrowed the AT-SPI tree, with a warning in `message`) and `accessibility_tree_truncated` (true when the node, depth, read, or time budget stopped traversal with unread elements left). When a result would pass 32,000 bytes, the tree comes back as its leading nodes plus `accessibility_tree_spill` (`path`, `node_count`, `bytes`, `inline_node_count`). `path` is a private JSON file holding the whole tree with the same `element_index` values, in `$XDG_RUNTIME_DIR/computer-use-linux/trees` when that runtime directory is private, otherwise under `~/.cache` (directories 0700, file 0600). Search the file instead of requesting the tree again; hosts that cannot read files page through the same tree with `get_tree_page`. The file lasts until the next snapshot or until the server exits; after a crash it is swept at the next start or after an hour.
|
|
128
128
|
|
|
129
129
|
An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop.
|
|
130
130
|
|
|
@@ -521,7 +521,7 @@ files.
|
|
|
521
521
|
- **Focused containers**: Chromium can mark a `document web` focused alongside its contenteditable editor. The focus probe keeps searching that document and reports the editor; a document that holds focus itself still gets a warning, which no longer claims the text was lost. An editor can carry the `editable` state without the EditableText interface (`supports_editable_text: false`): `type_text` can reach it, `set_value` cannot. Verify its text or a screenshot before submitting or retrying.
|
|
522
522
|
- **Large embedded browser trees**: an app-scoped snapshot includes embedded browser documents, which can use up the node budget before the app's own controls. When `accessibility_tree_truncated` is true, a missing editor proves nothing: raise `max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard cap 64). A busy renderer may give only a few thousand nodes before the deadline. A snapshot stops reading at 10 seconds and returns what it read, marked truncated; if the editor is still missing, hide the embedded browser pane for the snapshot.
|
|
523
523
|
- **Selkies pixelflux**: when `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` are set, screenshots and pointer/keyboard actions use the container's loopback computer-use server first. Element clicks scale AT-SPI bounds by the latest screenshot size over the logical layout (`wlr-output-management` on labwc, the existing desktop layout elsewhere). Coordinate clicks use screenshot pixels. A connection failure falls through to the backends below; an HTTP error does not.
|
|
524
|
-
- **Wayland coordinate input
|
|
524
|
+
- **Wayland coordinate input.** Coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll positions through uinput before sending only wheel events through ydotool. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. A device sized for an earlier screenshot is replaced at the next coordinate action. If that fails, the action is not sent, and the result says where a retry goes: a new device from a fresh screenshot, then the RemoteDesktop portal, ydotool, or a refusal on Wayland. Scrolling without a point or window/element target keeps using the current pointer position. Portal keyboard input preserves the receiving window or app target before consent, then restores and verifies its focus after consent. If focus cannot be verified, no text or key chord is sent. Untargeted portal keyboard calls therefore require working window introspection. GNOME Introspect only verifies an already focused app; if consent changes focus, refocus the app manually or install the Shell extension.
|
|
525
525
|
- **Native X11 coordinate clicks.** Eligible native X11 sessions use one supervised `xdotool mousemove -- X Y click --repeat N BUTTON` command for left, middle, and right clicks; native X11 does not initialize the system-wide uinput device. Missing or failed xdotool commands return an error without routing input to another display. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this xdotool path.
|
|
526
526
|
- **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. Missing or failed xdotool commands return an error without a system-wide fallback. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.
|
|
527
527
|
- **Drag motion.** XTEST, uinput and portal drags use intermediate points paced 30 ms apart. Native X11 uses XTEST on `DISPLAY`; uinput and portal paths use the same trajectory. A failed press or motion still attempts button release, and the gesture is not replayed through another backend. Portal paths require shared endpoints and clamp intermediate points to the nearest shared monitor when the trajectory crosses a monitor gap. Verify the destination after dispatch.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.42",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -7,7 +7,7 @@ export interface GeneratedMcpToolDefinition {
|
|
|
7
7
|
annotations: Record<string, unknown>;
|
|
8
8
|
}
|
|
9
9
|
|
|
10
|
-
export const GENERATED_SERVER_VERSION = "0.7.
|
|
10
|
+
export const GENERATED_SERVER_VERSION = "0.7.42";
|
|
11
11
|
export const GENERATED_TOOL_CATALOG_HASH = "c3da735e9c17d0bd15294e604abd28a6008e67d250f6db226da05e3f9504f68f";
|
|
12
12
|
export const GENERATED_SHELL_TOOL_CATALOG_HASH = "96bd879e7d29ef3f3193d49ef6f86b0d561b2ac9c7eeeece326c1637e1c6428f";
|
|
13
13
|
export const GENERATED_MCP_TOOLS =
|
|
@@ -196,6 +196,11 @@ This requires both `x`/`y` and element identity; do not
|
|
|
196
196
|
combine it with `relative: true` or use it for ordinary screenshot pixels.
|
|
197
197
|
Use `perform_action` explicitly for entry `activate` or slider `jump`; `click`
|
|
198
198
|
never substitutes those actions, including when bounds are unavailable.
|
|
199
|
+
A coordinate `click` or `scroll` result warns when the point is off the
|
|
200
|
+
screenshot or between monitors, where no monitor shows that part of the
|
|
201
|
+
screenshot. The result says whether the input was sent, and the warning says
|
|
202
|
+
where it may have landed; pick a point on a monitor and check the screen
|
|
203
|
+
before retrying.
|
|
199
204
|
|
|
200
205
|
### Screenshot-relative coordinates
|
|
201
206
|
|