@agent-sh/computer-use-linux 0.7.46 → 0.7.48

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -50,7 +50,7 @@ The Rust crate is published as [`computer-use-linux`](https://crates.io/crates/c
50
50
 
51
51
  Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive `xdotool` against an X11 root window or shell out to OCR over screenshots. It differs in five ways:
52
52
 
53
- - **Wayland actually works.** On Wayland, pointer actions use `/dev/uinput` or `org.freedesktop.portal.RemoteDesktop`. For pointer actions ydotool only sends wheel events there, and a coordinate action with neither route is refused. Literal text prefers KDE clipboard integration on Plasma, `wtype` on compatible Wayland compositors, and portal keysyms on other Wayland sessions, including GNOME with ydotool installed. Portal text startup or conversion errors fail the call without replaying text through ydotool. Raw ydotool text accepts printable ASCII, tab and newline only, and its physical US key positions can produce different text under the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, then on a native X11 session a root-window `GetImage` (device pixels, no toolkit scaling), and finally spawn `gnome-screenshot` for background/systemd contexts where the DBus paths are denied.
53
+ - **Wayland actually works.** On Wayland, pointer actions use `/dev/uinput` or `org.freedesktop.portal.RemoteDesktop`. ydotool sends only untargeted wheel events there, and a coordinate action with neither route is refused. Literal text prefers KDE clipboard integration on Plasma, `wtype` on compatible Wayland compositors, and portal keysyms on other Wayland sessions, including GNOME with ydotool installed. Portal text startup or conversion errors fail the call without replaying text through ydotool. Raw ydotool text accepts printable ASCII, tab and newline only, and its physical US key positions can produce different text under the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, then on a native X11 session a root-window `GetImage` (device pixels, no toolkit scaling), and finally spawn `gnome-screenshot` for background/systemd contexts where the DBus paths are denied.
54
54
  - **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed. Terminal windows are enriched with their controlling TTY and foreground process so agents can target "the terminal running `pytest`" directly. Every `window_id` fits JavaScript's exact integer range (at most 2^53-1), so JSON clients can pass it back unchanged; a larger value is rejected with a prompt to re-run `list_windows`.
55
55
  - **Semantic selectors first, pixel coordinates when needed.** Tools like `click`, `perform_action`, and `set_value` accept `role` / `name` / `text` / `states` selectors or `element_index` values backed by AT-SPI, and every mutating tool can return a post-action screenshot and updated accessibility tree in the same round trip. Pixel coordinates remain available for rendering-only surfaces (canvas, games, X clients without ATK).
56
56
  - **Visible to humans, invisible to screenshots.** While an agent acts, `computer-use-linux-indicator` renders a click-through overlay with a per-agent cursor glide, click ripple, keycaps/typing banner, screen-edge glow, and status pill, and hides automatically under a shared capture lock before any screenshot is read.
@@ -66,7 +66,7 @@ The crate was extracted from [`codex-desktop-linux`](https://github.com/avifenes
66
66
  | | `setup_accessibility` | Enables and verifies GNOME's `org.gnome.desktop.interface toolkit-accessibility` setting so toolkit apps expose AT-SPI trees. |
67
67
  | | `setup_window_targeting` | Installs and enables the bundled GNOME Shell extension. GNOME Wayland needs it for exact window activation and move/resize; every GNOME session needs it for the on-screen indicator. |
68
68
  | **Discovery & observation** | `get_app_state` | Combined screenshot + compacted AT-SPI accessibility tree with stable `element_index` handles. Scope with `app_name_or_bundle_identifier` or a window target (`window_id`, `pid`, `app_id`, `wm_class`, `title`). |
69
- | | `get_tree_page` | Page through the latest cached tree by node offset, for trees too large to return inline when the host cannot read the spill file. |
69
+ | | `get_tree_page` | Page through the latest cached tree by node offset, for trees too large to return inline when the host cannot read the spill file. Pass `snapshot_id` to refuse a page from a tree a later snapshot replaced. |
70
70
  | | `screenshot` | Bounded PNG or JPEG capture of the full desktop (`full_screen: true` or no target) or a raised and cropped target window (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `raise_window`). |
71
71
  | | `list_windows` | Compositor windows with `window_id`, title, `app_id`, `wm_class`, PID, focus state, client type (`wayland`/`x11`), bounds, and terminal TTY/process metadata. |
72
72
  | | `focused_window` | The window currently holding keyboard focus. |
@@ -83,6 +83,8 @@ The crate was extracted from [`codex-desktop-linux`](https://github.com/avifenes
83
83
  | **Opt-in host tools** | `complete_interaction` | Parameter-free desktop completion notification (`notify-send`); registered only when `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`. |
84
84
  | | `run_shell` | Bounded non-login `/bin/sh -c` execution; registered only when `COMPUTER_USE_LINUX_ENABLE_SHELL=1` (absent by default). |
85
85
 
86
+ Every tree result carries a `snapshot_id`: `get_app_state`, an `include_tree` observation, and `get_tree_page`. The server caches one tree, and each new tree replaces it. Pass the `snapshot_id` with an `element_index` or selector to `click`, `scroll`, `perform_action` or `set_value`, and to `get_tree_page`: if a later tree replaced that snapshot, the call is refused before any window is raised or input is sent, since the index could name another element by then. A full `x`/`y` point does not come from the tree, so it is not checked. Without `snapshot_id`, the element resolves in the latest tree. An `element_identifier` names the AT-SPI object itself and is not checked against a snapshot. Ids differ between server processes, so an id from before a restart is refused.
87
+
86
88
  `click`, `scroll`, `perform_action`, and `set_value` take an optional app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`). With one, an `element_index` or selector from another app's `get_app_state` snapshot is refused; call `get_app_state` for the target app first. When the window backend reports no pid for a target window (i3 without `xprop`, COSMIC, other wlroots compositors), the snapshot must have been taken for that window, or the call is refused. `perform_action` and `set_value` do not focus the target window. An `element_identifier` is refused when its owner on the accessibility bus is a different pid than the target; it is not checked when either pid is unknown.
87
89
 
88
90
  ### Post-action observation in one round trip
@@ -91,7 +93,7 @@ All mutating tools (`click`, `drag`, `scroll`, `press_key`, `type_text`, `perfor
91
93
 
92
94
  - `screenshot_after: true | "screen" | "window" | false`: capture after the action is dispatched and the indicator overlay is hidden (`true` or `"screen"` for full desktop, `"window"` cropped to the action's target window or the focused window when untargeted)
93
95
  - `format`, `quality`, `scale`, `max_width`, `max_height`, `max_bytes`: same screenshot payload controls as `screenshot` and `get_app_state`
94
- - `include_tree: true`: include a fresh compacted AT-SPI accessibility tree and update cached `element_index` values
96
+ - `include_tree: true`: include a fresh compacted AT-SPI accessibility tree and its `snapshot_id`; it replaces the cached tree, so later `element_index` values come from it
95
97
 
96
98
  When post-action screenshot capture succeeds, the image is returned as the first MCP `image` content block alongside the JSON action summary and `screenshot` metadata. If the action succeeds (`ok: true`) and only the post-action screenshot or tree capture fails, `ok` remains `true` and the error is reported in `screenshot_error` / `accessibility_error` and appended to `message`.
97
99
 
@@ -524,14 +526,14 @@ files.
524
526
  - **Focused containers**: Chromium can mark a `document web` focused alongside its contenteditable editor. The focus probe keeps searching that document and reports the editor; a document that holds focus itself still gets a warning, which no longer claims the text was lost. An editor can carry the `editable` state without the EditableText interface (`supports_editable_text: false`): `type_text` can reach it, `set_value` cannot. Verify its text or a screenshot before submitting or retrying.
525
527
  - **Large embedded browser trees**: an app-scoped snapshot includes embedded browser documents, which can use up the node budget before the app's own controls. When `accessibility_tree_truncated` is true, a missing editor proves nothing: raise `max_nodes` (default 1000, hard cap 10000) and, if needed, `max_depth` (hard cap 64). A busy renderer may give only a few thousand nodes before the deadline. A snapshot stops reading at 10 seconds and returns what it read, marked truncated; if the editor is still missing, hide the embedded browser pane for the snapshot.
526
528
  - **Selkies pixelflux**: when `PIXELFLUX_CU` and `PIXELFLUX_CU_TOKEN` are set, screenshots and pointer/keyboard actions use the container's loopback computer-use server first. Element clicks scale AT-SPI bounds by the latest screenshot size over the logical layout (`wlr-output-management` on labwc, the existing desktop layout elsewhere). Coordinate clicks use screenshot pixels. A connection failure falls through to the backends below; an HTTP error does not.
527
- - **Wayland coordinate input.** Coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll positions through uinput before sending only wheel events through ydotool. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. A device sized for an earlier screenshot is replaced at the next coordinate action. If that fails, the action is not sent, and the result says where a retry goes: a new device from a fresh screenshot, then the RemoteDesktop portal, ydotool, or a refusal on Wayland. Scrolling without a point or window/element target keeps using the current pointer position. Portal keyboard input preserves the receiving window or app target before consent, then restores and verifies its focus after consent. If focus cannot be verified, no text or key chord is sent. Untargeted portal keyboard calls therefore require working window introspection. GNOME Introspect only verifies an already focused app; if consent changes focus, refocus the app manually or install the Shell extension.
529
+ - **Wayland coordinate input.** Coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll sends the move and the wheel through the same uinput device, so the compositor routes the wheel to the point the move set, and it needs no ydotool daemon. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. A device sized for an earlier screenshot is replaced at the next coordinate action. If that fails, the action is not sent, and the result says where a retry goes: a new device from a fresh screenshot, then the RemoteDesktop portal, ydotool, or a refusal on Wayland. Scrolling without a point or window/element target keeps using the current pointer position. Portal keyboard input preserves the receiving window or app target before consent, then restores and verifies its focus after consent. If focus cannot be verified, no text or key chord is sent. Untargeted portal keyboard calls therefore require working window introspection. GNOME Introspect only verifies an already focused app; if consent changes focus, refocus the app manually or install the Shell extension.
528
530
  - **Native X11 coordinate clicks.** Eligible native X11 sessions use one supervised `xdotool mousemove -- X Y click --repeat N BUTTON` command for left, middle, and right clicks; native X11 does not initialize the system-wide uinput device. Missing or failed xdotool commands return an error without routing input to another display. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this xdotool path.
529
531
  - **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. Missing or failed xdotool commands return an error without a system-wide fallback. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.
530
532
  - **Drag motion.** XTEST, uinput and portal drags use intermediate points paced 30 ms apart. Native X11 uses XTEST on `DISPLAY`; uinput and portal paths use the same trajectory. A failed press or motion still attempts button release, and the gesture is not replayed through another backend. Portal paths require shared endpoints and clamp intermediate points to the nearest shared monitor when the trajectory crosses a monitor gap. Verify the destination after dispatch.
531
533
  - **Native X11 transport.** Native X11 queries (window origins, root screenshots, `doctor`) connect and read replies under one shared 2-second deadline enforced in the transport itself, including the handshake and every reply. A stalled X server closes the query connection instead of leaving a blocked worker behind. Unix sockets, literal IP addresses, and `localhost` need no resolver. A remote hostname in `DISPLAY` needs `getent ahosts` for bounded name resolution; if it is missing or fails, the X11 route is reported as unavailable.
532
534
  - **Window registry**: `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, labwc's `labwc.sock`, wlr-foreign-toplevel-management, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
533
535
  - **niri backend**: prefers `niri msg`, falling back to direct JSON IPC over the same session socket. `is_minimized` maps to `hidden`. Bounds use `layout.window_size` (or `tile_size`) and, when available, the tile position plus the window's offset and the output's logical origin. Uniform output scaling converts these to device pixels; unknown or mixed scaling omits bounds. Missing window positions stay `null`, so listing and focusing still work without enabling coordinate targeting. An accepted focus action is verified by querying the focused window again.
534
- - **GNOME extension fallback**: recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`. When the extension is already enabled, `setup-window-targeting` and `install.sh` ask for a new login after an upgrade that GNOME Shell would run: a changed `extension.js`, or a changed `metadata.json` field the Shell acts on, such as `shell-version`. The Shell only checks that the name and description are present, and only displays them and the link, so a change to those alone needs none.
536
+ - **GNOME extension fallback**: recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`. `setup-window-targeting` and `install.sh` install it in `$XDG_DATA_HOME/gnome-shell/extensions` (`~/.local/share/gnome-shell/extensions` when `XDG_DATA_HOME` is unset or empty), where GNOME Shell loads user extensions. When the Shell already loaded a user copy, `setup-window-targeting` updates the copy the Shell reports. When the extension is already enabled, both ask for a new login after an upgrade that GNOME Shell would run: a changed `extension.js`, or a changed `metadata.json` field the Shell acts on, such as `shell-version`. The Shell only checks that `name` and `description` are present, and only displays them, `url`, `version-name` and `donations`, so a change to those alone needs none. `install.sh` checks the four string fields and asks for a login when `donations` changes.
535
537
  - **COSMIC helper**: `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`. COSMIC pointer coordinates use per-monitor RemoteDesktop portal streams because its uinput absolute axes map onto the active monitor. Approve the portal dialog before continuing. When ConnectToEIS is available, COSMIC input waits for resumed EIS devices before sending the first event, so portal startup cannot discard the text prefix. The helper returns logical window bounds from COSMIC geometry and xdg-output events; targeted screenshots and relative clicks map these into capture pixels. Window IDs fit JavaScript's exact integer range. PID remains optional because the toplevel protocol does not expose it. Like `computer-use-linux`, the helper checks its command line first: `-h` or `--help` prints help and exits 0, and a command, flag or argument it does not take exits 2 before it connects to the compositor.
536
538
  - **Indicator overlay**: the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening. The indicator takes no arguments: `-h` or `--help` prints help, and any other argument exits 2 before it takes a lock or connects to the compositor.
537
539
  - **Terminal enrichment**: `list_windows` cross-references each terminal window with its controlling TTY and the foreground process on that TTY, so `type_text` / `press_key` can target "the terminal where `pytest` is running" without the host ever knowing the window id.
package/npm/README.md CHANGED
@@ -163,6 +163,13 @@ does not yet forward the flag or include the tool in its catalog.
163
163
 
164
164
  ## Input & coordinate contracts
165
165
 
166
+ - **Snapshot ids**: `get_app_state`, `include_tree` observations and
167
+ `get_tree_page` return the `snapshot_id` of their tree. The server caches
168
+ one tree, and each new tree replaces it. Pass `snapshot_id` with an
169
+ `element_index` or selector to `click`, `scroll`, `perform_action` or
170
+ `set_value`, and to `get_tree_page`: a call naming a replaced snapshot is
171
+ refused before any input. Without it, the element resolves in the latest
172
+ tree.
166
173
  - **Element targets**: `click`, `scroll`, `perform_action`, and `set_value`
167
174
  accept an optional app target (`pid`, `window_id`, `app_id`, `wm_class`,
168
175
  `window_title`). With one, an `element_index` or selector from another app's
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@agent-sh/computer-use-linux",
3
- "version": "0.7.46",
3
+ "version": "0.7.48",
4
4
  "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
5
5
  "license": "MIT",
6
6
  "type": "commonjs",
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
7
7
  annotations: Record<string, unknown>;
8
8
  }
9
9
 
10
- export const GENERATED_SERVER_VERSION = "0.7.46";
11
- export const GENERATED_TOOL_CATALOG_HASH = "5220d9a11046797491e8899ea59c66d1d5d450c1457086867d8ae9784c297acd";
12
- export const GENERATED_SHELL_TOOL_CATALOG_HASH = "5cb7f910f9317b8b249fad1b994f1d771bd0854ddcad3a9b1f4b040ed2c050b5";
10
+ export const GENERATED_SERVER_VERSION = "0.7.48";
11
+ export const GENERATED_TOOL_CATALOG_HASH = "2d363dbef08a07c77fca38591d9e667d6501543b816fbd910504264d4360b249";
12
+ export const GENERATED_SHELL_TOOL_CATALOG_HASH = "92618cb44d6df6dbb898de69e4f4499781e9dcd86ebe691ac430ffb9e14c0372";
13
13
  export const GENERATED_MCP_TOOLS =
14
14
  [
15
15
  {
@@ -366,6 +366,14 @@ export const GENERATED_MCP_TOOLS =
366
366
  ],
367
367
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
368
368
  },
369
+ "snapshot_id": {
370
+ "default": null,
371
+ "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index or selector comes from. If a later snapshot replaced\nit, the call is refused before any input. Without it, the element\nresolves in the latest snapshot.",
372
+ "type": [
373
+ "string",
374
+ "null"
375
+ ]
376
+ },
369
377
  "states": {
370
378
  "default": [],
371
379
  "description": "AT-SPI states the element must all have, case-insensitive (for\nexample focused, checked).",
@@ -594,7 +602,7 @@ export const GENERATED_MCP_TOOLS =
594
602
  "openWorldHint": true,
595
603
  "readOnlyHint": true
596
604
  },
597
- "description": "Get a size-bounded screenshot and accessibility state for a Linux app. Scope the accessibility tree with app_name_or_bundle_identifier or a window_id/pid/app_id/wm_class/title target; omitting a target returns the whole desktop tree and can flood context. Nodes are compact (empty fields left out, actions by name; verbose=true returns every field). Pass interactive_only=true for only actionable nodes, headings and status regions, with the same element_index values. A title or window_id target narrows an app with several windows under one AT-SPI root (Chromium) to that window, keeping the app's popups and dialogs. A window-targeted screenshot needs that window visible and already focused (outside Hyprland background mode); otherwise screenshot_error is set and the tree still returns, so use the screenshot tool to raise and capture it. A tree too large to return inline is written whole to the private file in accessibility_tree_spill. Screenshot results always include coordinate_width, coordinate_height, scale, and format, plus quality for jpeg; callers can request jpeg/quality for compression before resizing.",
605
+ "description": "Get a size-bounded screenshot and accessibility state for a Linux app. Scope the accessibility tree with app_name_or_bundle_identifier or a window_id/pid/app_id/wm_class/title target; omitting a target returns the whole desktop tree and can flood context. Nodes are compact (empty fields left out, actions by name; verbose=true returns every field). Pass interactive_only=true for only actionable nodes, headings and status regions, with the same element_index values. A title or window_id target narrows an app with several windows under one AT-SPI root (Chromium) to that window, keeping the app's popups and dialogs. A window-targeted screenshot needs that window visible and already focused (outside Hyprland background mode); otherwise screenshot_error is set and the tree still returns, so use the screenshot tool to raise and capture it. A tree too large to return inline is written whole to the private file in accessibility_tree_spill. The result's snapshot_id names the tree: element_index and selector actions that pass it are refused once a later tree replaced it. Screenshot results always include coordinate_width, coordinate_height, scale, and format, plus quality for jpeg; callers can request jpeg/quality for compression before resizing.",
598
606
  "inputSchema": {
599
607
  "$defs": {
600
608
  "ScreenshotOutputFormat": {
@@ -803,7 +811,7 @@ export const GENERATED_MCP_TOOLS =
803
811
  "openWorldHint": true,
804
812
  "readOnlyHint": true
805
813
  },
806
- "description": "Return a page of the accessibility tree from the latest get_app_state or include_tree snapshot, starting at node offset (default 0). Use it when a tree was too large to return inline (accessibility_tree_spill is set) and the host cannot read the spill file; page with next_offset until it is null. Element indices match the snapshot, so element_index actions work on these nodes. It does not read the desktop again.",
814
+ "description": "Return a page of the accessibility tree from the latest get_app_state or include_tree snapshot, starting at node offset (default 0). Use it when a tree was too large to return inline (accessibility_tree_spill is set) and the host cannot read the spill file; page with next_offset until it is null. Element indices match the snapshot, so element_index actions work on these nodes. Pass the snapshot's snapshot_id to refuse a page from a tree a later snapshot replaced. It does not read the desktop again.",
807
815
  "inputSchema": {
808
816
  "$schema": "https://json-schema.org/draft/2020-12/schema",
809
817
  "properties": {
@@ -815,6 +823,14 @@ export const GENERATED_MCP_TOOLS =
815
823
  "integer",
816
824
  "null"
817
825
  ]
826
+ },
827
+ "snapshot_id": {
828
+ "default": null,
829
+ "description": "snapshot_id of the get_app_state or include_tree result being paged.\nIf a later snapshot replaced it, no nodes are returned.",
830
+ "type": [
831
+ "string",
832
+ "null"
833
+ ]
818
834
  }
819
835
  },
820
836
  "type": "object"
@@ -1196,6 +1212,14 @@ export const GENERATED_MCP_TOOLS =
1196
1212
  ],
1197
1213
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
1198
1214
  },
1215
+ "snapshot_id": {
1216
+ "default": null,
1217
+ "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index or selector comes from. If a later snapshot replaced\nit, the call is refused before any input. Without it, the element\nresolves in the latest snapshot.",
1218
+ "type": [
1219
+ "string",
1220
+ "null"
1221
+ ]
1222
+ },
1199
1223
  "states": {
1200
1224
  "default": [],
1201
1225
  "description": "AT-SPI states the element must all have, case-insensitive (for\nexample focused, checked).",
@@ -1908,6 +1932,14 @@ export const GENERATED_MCP_TOOLS =
1908
1932
  ],
1909
1933
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
1910
1934
  },
1935
+ "snapshot_id": {
1936
+ "default": null,
1937
+ "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index comes from. If a later snapshot replaced it, the call\nis refused before any input. Without it, the element resolves in the\nlatest snapshot.",
1938
+ "type": [
1939
+ "string",
1940
+ "null"
1941
+ ]
1942
+ },
1911
1943
  "window_id": {
1912
1944
  "default": null,
1913
1945
  "description": "Window id from a recent list_windows. The first target field set\ndecides, in this order: window_id, pid, app_id, wm_class, window_title.",
@@ -2107,6 +2139,14 @@ export const GENERATED_MCP_TOOLS =
2107
2139
  ],
2108
2140
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
2109
2141
  },
2142
+ "snapshot_id": {
2143
+ "default": null,
2144
+ "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index or selector comes from. If a later snapshot replaced\nit, the call is refused before any input. Without it, the element\nresolves in the latest snapshot.",
2145
+ "type": [
2146
+ "string",
2147
+ "null"
2148
+ ]
2149
+ },
2110
2150
  "states": {
2111
2151
  "default": [],
2112
2152
  "description": "AT-SPI states the element must all have, case-insensitive (for\nexample focused, checked).",
@@ -221,6 +221,7 @@ const FORWARDED_ENVIRONMENT = [
221
221
  "XDG_SESSION_TYPE",
222
222
  "XDG_SESSION_DESKTOP",
223
223
  "XDG_DATA_DIRS",
224
+ "XDG_DATA_HOME",
224
225
  "XDG_CONFIG_HOME",
225
226
  "XDG_CACHE_HOME",
226
227
  "XDG_STATE_HOME",
@@ -81,7 +81,7 @@ computer-use-linux setup-window-targeting
81
81
  computer-use-linux doctor | jq .readiness
82
82
  ```
83
83
 
84
- If `doctor` selects ydotool as the input backend, also enable its per-user daemon. `./install.sh` creates `ydotoold.service`; Ubuntu's ydotool package ships `ydotool.service` instead (`systemctl --user enable --now ydotool`). Direct uinput, X11 xdotool, and RemoteDesktop portal input do not require `ydotoold`, but a Wayland scroll at a point needs ydotool for the wheel, or the portal.
84
+ If `doctor` selects ydotool as the input backend, also enable its per-user daemon. `./install.sh` creates `ydotoold.service`; Ubuntu's ydotool package ships `ydotool.service` instead (`systemctl --user enable --now ydotool`). Direct uinput, X11 xdotool, and RemoteDesktop portal input do not require `ydotoold`. On Wayland, ydotool sends only untargeted scrolls; a scroll at a point goes through uinput or the portal, and without the daemon an untargeted scroll goes through the portal too.
85
85
 
86
86
  On niri, the server reads `NIRI_SOCKET` from its parent processes or the systemd
87
87
  user environment when the host does not pass it; without it, discovery needs an
@@ -146,7 +146,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
146
146
  4. If `can_build_accessibility_tree` is false, run `setup_accessibility` and restart the target app.
147
147
  5. If `can_query_windows` or `can_focus_windows` is false on GNOME Wayland, run `setup_window_targeting` and ask the user to log out and back in if setup says the shell extension needs a reload.
148
148
  6. Before targeted input, call `list_windows` or `focused_window` and verify the intended window by title, app id, pid, or wm class.
149
- 7. Prefer semantic targeting from `get_app_state`: use element indices or role/name/text/states selectors. Automatic recovery of a dead semantic target requires verified PID/window ownership and invalidates cached indices; call `get_app_state` again before further index-based actions. When several apps are in play, pass the app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`) to `click`, `scroll`, `perform_action`, and `set_value`: an index or selector from another app's snapshot is then refused instead of acting on that app. On `click` and `scroll` the target also raises and focuses that window, and nothing is sent when focus cannot be verified; on GNOME Wayland, raising a window needs the Shell extension. The title field there is `window_title`. Where the window backend reports no pid (i3 without `xprop`, COSMIC, and wlr-foreign-toplevel on stock labwc and other wlroots compositors), call `get_app_state` for the same window first or pass `pid`, or the call is refused.
149
+ 7. Prefer semantic targeting from `get_app_state`: use element indices or role/name/text/states selectors, and pass the `snapshot_id` of the result they came from. Every `get_app_state` and `include_tree` result replaces the one cached tree; a call that names a replaced snapshot is refused before any input, so take the indices and `snapshot_id` from the newest result. Automatic recovery of a dead semantic target requires verified PID/window ownership and invalidates cached indices; call `get_app_state` again before further index-based actions. When several apps are in play, pass the app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`) to `click`, `scroll`, `perform_action`, and `set_value`: an index or selector from another app's snapshot is then refused instead of acting on that app. On `click` and `scroll` the target also raises and focuses that window, and nothing is sent when focus cannot be verified; on GNOME Wayland, raising a window needs the Shell extension. The title field there is `window_title`. Where the window backend reports no pid (i3 without `xprop`, COSMIC, and wlr-foreign-toplevel on stock labwc and other wlroots compositors), call `get_app_state` for the same window first or pass `pid`, or the call is refused.
150
150
  8. Use coordinates only when the UI surface has no useful accessibility tree.
151
151
  9. For text input, prefer `type_text` with a target selector (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `tty`, `terminal_pid`, `terminal_command`, or `terminal_cwd`) rather than relying on current focus. A `window_id` is valid only as returned by a recent `list_windows`; if a call rejects it as outside 0 to 2^53-1 or matches no window, re-run `list_windows`.
152
152
  10. After mutating actions (`click`, `drag`, `scroll`, `press_key`, `type_text`, `perform_action`, `set_value`, `activate_window`, `move_window`, `resize_window`), either pass `screenshot_after: true | "screen" | "window"` (with optional `format`, `quality`, `scale`, `max_width`, `max_height`, `max_bytes`) and/or `include_tree: true` on the action call itself to observe the result in the same tool call, or re-check state with `get_app_state`, `focused_window`, or an app-specific readback. A failed action (`ok: false`) returns no post-action screenshot or tree, so call `get_app_state`; `set_value` still observes when `verified` is `false`.