@agent-sh/computer-use-linux 0.7.51 → 0.7.53

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -95,6 +95,7 @@ All mutating tools (`click`, `drag`, `scroll`, `press_key`, `type_text`, `perfor
95
95
 
96
96
  - `screenshot_after: true | "screen" | "window" | false`: capture after the action is dispatched and the indicator overlay is hidden (`true` or `"screen"` for full desktop, `"window"` cropped to the action's target window or the focused window when untargeted)
97
97
  - `format`, `quality`, `scale`, `max_width`, `max_height`, `max_bytes`: same screenshot payload controls as `screenshot` and `get_app_state`
98
+ - `screenshot_file_path`: save the post-action screenshot to this new file instead of returning it as image content (see [Screenshot files](#screenshot-files))
98
99
  - `include_tree: true`: include a fresh compacted AT-SPI accessibility tree and its `snapshot_id`; it replaces the cached tree, so later `element_index` values come from it
99
100
 
100
101
  When post-action screenshot capture succeeds, the image is returned as the first MCP `image` content block alongside the JSON action summary and `screenshot` metadata. If the action succeeds (`ok: true`) and only the post-action screenshot or tree capture fails, `ok` remains `true` and the error is reported in `screenshot_error` / `accessibility_error` and appended to `message`.
@@ -103,6 +104,19 @@ When post-action screenshot capture succeeds, the image is returned as the first
103
104
 
104
105
  Screenshot payloads are size-bounded by default before they are returned to the MCP host: max 1920 px width/height and 2 MiB image bytes, with hard caps even when callers request more. Agents that need more detail can pass `max_width`, `max_height`, `max_bytes`, `scale`, `format: "jpeg"`, or `quality`, preferably with a window target or crop. PNG remains the default; JPEG lets callers trade lossless pixels for a smaller payload before the byte cap forces further resizing. Returned screenshot metadata includes `coordinate_width`, `coordinate_height`, `scale`, `format`, and `quality` so callers can convert from a downscaled preview to desktop coordinate pixels.
105
106
 
107
+ ### Screenshot files
108
+
109
+ `screenshot`, `get_app_state` and the `screenshot_after` capture of mutating tools take `screenshot_file_path`. With it, the image is saved to that file and no MCP image content is returned, so a capture costs no image tokens until the agent reads the file. The metadata stays the same (`width`, `height`, `coordinate_width`, `coordinate_height`, `scale`, `format`, `quality`, `bytes`) plus `file_path`: in the `screenshot` caption, and in `screenshot.file_path` for the other tools.
110
+
111
+ - The path must be absolute, and its parent directory must exist. Directories are not created.
112
+ - The extension must match `format`: `.png` for png (the default), `.jpg` or `.jpeg` for jpeg.
113
+ - The path must end with the file name, so a trailing `/` or `/.` is refused.
114
+ - File output creates a new file and never replaces one. Any existing entry at the path is refused: a file, a directory or a symlink.
115
+ - The image is written to a temp file in the same directory, then moved into place with a step that fails if anything exists at the path (`renameat2` with `RENAME_NOREPLACE`, or `link` where that is unsupported). A reader never sees a partial file, and an entry created meanwhile is neither followed nor replaced. The file is created with mode 0600.
116
+ - A bad path is refused before anything is captured or any input is sent.
117
+ - A failed write is never answered with the image inline. `screenshot` returns an error, `get_app_state` sets `screenshot_error`, and a mutating tool sets `screenshot_error` and keeps the action's `ok`.
118
+ - Over stdio the path is on the machine that runs the server. Over HTTP (`mcp --listen`) the parameter is refused, so a remote client gets no way to write files on the host.
119
+
106
120
  ### Input & semantic targeting contracts
107
121
 
108
122
  For a plain left `click` by element index or selector, the element's first
@@ -136,7 +150,7 @@ An `app_id` may be a full desktop ID or its final dot-separated segment; matchin
136
150
 
137
151
  On X11, text input sends LF and CRLF line endings as Return, with one newline per CRLF pair.
138
152
 
139
- On GNOME, a read-only keymap preflight rejects the whole string before input if a character is absent or requires a modifier level Mutter cannot synthesize. The background reader cannot establish the active layout group, so every configured group must support the text. This can reject text that works in one of several groups; use `set_value` on an editable field instead. The check requires `libxkbcommon.so.0` at runtime and stops after a two-second keymap read deadline. It does not cover held modifiers or a keymap changed concurrently with typing.
153
+ On GNOME, a read-only keymap preflight rejects the whole string before input if a character is absent or requires a modifier level Mutter cannot synthesize. The background reader cannot establish the active layout group, so every configured group must support the text. This can reject text that works in one of several groups. Then the text goes into the focused editable field of the verified target window through AT-SPI EditableText: it is inserted at the caret, replaces a selection, and counts as typed (`ok: true`) only when the field reads it back with the rest of its text unchanged (in a field over 4096 characters, the 32 on each side of the insert). The field must be in the AT-SPI window that matches the target and that window must be active, so a field left focused in another window of the app is not used. A password field, focus outside an editable field, and text with a newline, tab or other control character keep the refusal; send keys with `press_key`, or fill the field with `set_value`. The check parses the keymap in pure Rust, with no system XKB library, and stops after a two-second keymap read deadline. It does not cover held modifiers or a keymap changed concurrently with typing.
140
154
 
141
155
  ### On-screen indicator
142
156
 
@@ -154,9 +168,9 @@ Captures require the shared capture lock and, when an overlay is running, its ac
154
168
 
155
169
  | Class | Tools | Contract |
156
170
  | --- | --- | --- |
157
- | Read-only observation | `doctor`, `list_apps`, `list_windows`, `focused_window`, `get_app_state`, `get_tree_page` | `readOnlyHint=true`; may reveal app, window, accessibility, and screenshot contents. `get_app_state` may trigger the desktop screenshot portal prompt. |
171
+ | Read-only observation | `doctor`, `list_apps`, `list_windows`, `focused_window`, `get_app_state`, `get_tree_page` | `readOnlyHint=true`; may reveal app, window, accessibility, and screenshot contents. `get_app_state` may trigger the desktop screenshot portal prompt. Its `screenshot_file_path` file output creates a new file and never replaces one. |
158
172
  | Local setup mutators | `setup_accessibility`, `setup_window_targeting` | `readOnlyHint=false`, `destructiveHint=false`, `idempotentHint=true`; modifies user desktop configuration by enabling accessibility or installing/enabling the GNOME window-targeting extension. |
159
- | UI state mutators | `activate_window`, `move_window`, `resize_window`, `scroll`, `screenshot` | `readOnlyHint=false`, `destructiveHint=false`; changes focus, geometry, or scroll position in the live desktop, or raises a window to capture it. |
173
+ | UI state mutators | `activate_window`, `move_window`, `resize_window`, `scroll`, `screenshot` | `readOnlyHint=false`, `destructiveHint=false`; changes focus, geometry, or scroll position in the live desktop, or raises a window to capture it. `screenshot_file_path` file output creates a new file and never replaces one. |
160
174
  | Desktop action mutators | `click`, `drag`, `press_key`, `type_text`, `perform_action`, `set_value` | `readOnlyHint=false`, `destructiveHint=true`, `openWorldHint=true`; can trigger arbitrary actions in whatever local application is targeted. |
161
175
  | Conditional notification | `complete_interaction` | Absent unless `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`; when enabled, `readOnlyHint=false`, `destructiveHint=false`, `idempotentHint=false`, `openWorldHint=true`. Sends a desktop notification. |
162
176
  | Conditional host-code execution | `run_shell` | Absent unless `COMPUTER_USE_LINUX_ENABLE_SHELL=1`; when enabled, `readOnlyHint=false`, `destructiveHint=true`, `idempotentHint=false`, `openWorldHint=true`. Runs with the MCP server user's host permissions. |
@@ -256,6 +270,7 @@ You will still need AT-SPI enabled (`computer-use-linux setup`) and one input ba
256
270
  ### Option D: prebuilt binaries
257
271
 
258
272
  Linux x86_64 / aarch64 builds are published with each tag. Each binary ships a `.sha256` next to it.
273
+ The `-gnu` builds need glibc 2.28 or newer; the `-musl` builds (`target=x86_64-unknown-linux-musl`) are statically linked for musl distributions such as Alpine.
259
274
 
260
275
  - Latest release: <https://github.com/agent-sh/computer-use-linux/releases/latest>
261
276
 
@@ -440,7 +455,7 @@ For one-off installs without adding the tap first, Hermes also accepts `hermes s
440
455
 
441
456
  ### Generic MCP client
442
457
 
443
- Spawn the binary with `["mcp"]` as the argv tail. It speaks MCP (JSON-RPC over stdio) through rmcp, with the protocol versions [listed above](#wire-it-into-your-mcp-host); capability discovery happens through `tools/list` and the `doctor` tool. `mcp --listen ADDR:PORT` serves the same protocol over streamable HTTP at `/mcp` instead; without `--listen` no socket is opened. It needs a bearer token from the file named by `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` (for example a mounted Kubernetes Secret) or from `COMPUTER_USE_LINUX_MCP_TOKEN`. Prefer the file, because a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in `/proc/<pid>/environ` while the server runs, even though child processes do not inherit it. Clients send `Authorization: Bearer <token>`; a missing or wrong token gets 401. `localhost`, `127.0.0.1` and `::1` are always accepted as the `Host`. A bind that is not loopback, including `0.0.0.0` and `[::]`, needs each name clients use as `--allowed-host NAME` (repeatable, any port, case-insensitive); another `Host` gets 403. Any request with an `Origin` header gets 403, because browser clients are not supported. `--listen` refuses to start while `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, every `get_tree_page` call, and `click`, `scroll`, `set_value` and `perform_action` calls that pick an element by `element_index` or selector, must pass the `snapshot_id` of the caller's latest `get_app_state` or `include_tree` result; calls by `element_identifier` or capture-space `x`/`y` do not. Clients share one cached tree, so two clients that alternate `get_app_state` refuse each other's ids; call `get_app_state` again after a refusal. Responses may be SSE (`text/event-stream`). 2024-11-05 and 2025 clients get sessions; 2026-07-28 requests run stateless. SIGTERM and SIGINT stop the listener cleanly with exit status 128 plus the signal number, and a connection that does not send its request headers within 10 seconds is closed. The port is plain HTTP: put TLS in front (service mesh, ingress or sidecar) when it leaves the host. The server normally needs no other MCP-specific configuration, but desktop runtime environment still matters (`DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR`, portals, AT-SPI, `ydotoold`, and optionally `COMPUTER_USE_LINUX_COSMIC_HELPER`). Idle sessions expire after an hour. A `404 Session not found` means the session expired, so initialize again; the cached tree and its `snapshot_id` survive re-initializing.
458
+ Spawn the binary with `["mcp"]` as the argv tail. It speaks MCP (JSON-RPC over stdio) through rmcp, with the protocol versions [listed above](#wire-it-into-your-mcp-host); capability discovery happens through `tools/list` and the `doctor` tool. `mcp --listen ADDR:PORT` serves the same protocol over streamable HTTP at `/mcp` instead; without `--listen` no socket is opened. It needs a bearer token from the file named by `COMPUTER_USE_LINUX_MCP_TOKEN_FILE` (for example a mounted Kubernetes Secret) or from `COMPUTER_USE_LINUX_MCP_TOKEN`. Prefer the file, because a token in `COMPUTER_USE_LINUX_MCP_TOKEN` stays readable in `/proc/<pid>/environ` while the server runs, even though child processes do not inherit it. Clients send `Authorization: Bearer <token>`; a missing or wrong token gets 401. `localhost`, `127.0.0.1` and `::1` are always accepted as the `Host`. A bind that is not loopback, including `0.0.0.0` and `[::]`, needs each name clients use as `--allowed-host NAME` (repeatable, any port, case-insensitive); another `Host` gets 403. Any request with an `Origin` header gets 403, because browser clients are not supported. `--listen` refuses to start while `COMPUTER_USE_LINUX_ENABLE_SHELL=1`. Over HTTP, every `get_tree_page` call, and `click`, `scroll`, `set_value` and `perform_action` calls that pick an element by `element_index` or selector, must pass the `snapshot_id` of the caller's latest `get_app_state` or `include_tree` result; calls by `element_identifier` or capture-space `x`/`y` do not. Clients share one cached tree, so two clients that alternate `get_app_state` refuse each other's ids; call `get_app_state` again after a refusal. `screenshot_file_path` is refused over HTTP, so remote clients always get screenshots inline. Responses may be SSE (`text/event-stream`). 2024-11-05 and 2025 clients get sessions; 2026-07-28 requests run stateless. SIGTERM and SIGINT stop the listener cleanly with exit status 128 plus the signal number, and a connection that does not send its request headers within 10 seconds is closed. The port is plain HTTP: put TLS in front (service mesh, ingress or sidecar) when it leaves the host. The server normally needs no other MCP-specific configuration, but desktop runtime environment still matters (`DBUS_SESSION_BUS_ADDRESS`, `XDG_RUNTIME_DIR`, portals, AT-SPI, `ydotoold`, and optionally `COMPUTER_USE_LINUX_COSMIC_HELPER`). Idle sessions expire after an hour. A `404 Session not found` means the session expired, so initialize again; the cached tree and its `snapshot_id` survive re-initializing.
444
459
 
445
460
  ### Selkies containers (linuxserver.io)
446
461
 
package/npm/README.md CHANGED
@@ -123,6 +123,12 @@ Mutating tools (`click`, `drag`, `scroll`, `press_key`, `type_text`,
123
123
  `include_tree: true` to return a post-action screenshot and/or compacted AT-SPI
124
124
  tree in the same tool result.
125
125
 
126
+ `screenshot`, `get_app_state` and `screenshot_after` also take
127
+ `screenshot_file_path`, an absolute path where the image is saved instead of
128
+ returned as image content. The parent directory must exist, the extension must
129
+ match `format`, and the file must not exist: file output creates a new file
130
+ and never replaces one. It is refused over HTTP (`mcp --listen`).
131
+
126
132
  For an optional MCP completion notification, set
127
133
  `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1` in the server environment and have the
128
134
  agent call `complete_interaction` after its desktop work. `notify-send` must be
@@ -228,8 +234,11 @@ does not yet forward the flag or include the tool in its catalog.
228
234
  masks text. Both protections apply with the indicator disabled. Other compatible
229
235
  compositors retain their existing stdin path. Wtype rejects NUL and text over
230
236
  131,071 UTF-8 bytes before focusing a window or displaying text. `ok: true`
231
- confirms dispatch only; every text backend leaves field contents unverified.
232
- Re-observe before retrying. Exact window focus alone does not suppress missing-focus
237
+ from a keyboard backend confirms dispatch only and leaves field contents
238
+ unverified. Only the GNOME AT-SPI fallback reads the field back: its success
239
+ is `ok: true` with a message that starts with "Typed through AT-SPI
240
+ EditableText". Its failures have `ok: false` and start with "AT-SPI
241
+ EditableText insert was not verified". Re-observe before retrying. Exact window focus alone does not suppress missing-focus
233
242
  warnings: only a complete scoped tree containing window/menu chrome without
234
243
  content makes the feedback informational. Such trees get a conditional
235
244
  Electron/Chromium hint computed before compaction; this is not positive Electron
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@agent-sh/computer-use-linux",
3
- "version": "0.7.51",
3
+ "version": "0.7.53",
4
4
  "description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
5
5
  "license": "MIT",
6
6
  "type": "commonjs",
@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
7
7
  annotations: Record<string, unknown>;
8
8
  }
9
9
 
10
- export const GENERATED_SERVER_VERSION = "0.7.51";
11
- export const GENERATED_TOOL_CATALOG_HASH = "cffd6953843133dfee35c6c5bfc4f98e177798a31cc05e711a18d6b96ae7d2e4";
12
- export const GENERATED_SHELL_TOOL_CATALOG_HASH = "1b9f83b4ce6e5b07b25eae206dae6af7a31dda69fd073ad55d50f03bbd4bdb66";
10
+ export const GENERATED_SERVER_VERSION = "0.7.53";
11
+ export const GENERATED_TOOL_CATALOG_HASH = "03d66a3e0e8ee39a68fe551e1bfe6cfe99463ef8e3cdbbf53d8d6b501a8435ff";
12
+ export const GENERATED_SHELL_TOOL_CATALOG_HASH = "12bfcef2bf761a9990ba46734fd6a93a54c04a39209bcf17e62eecc7441da87a";
13
13
  export const GENERATED_MCP_TOOLS =
14
14
  [
15
15
  {
@@ -126,6 +126,13 @@ export const GENERATED_MCP_TOOLS =
126
126
  ],
127
127
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
128
128
  },
129
+ "screenshot_file_path": {
130
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
131
+ "type": [
132
+ "string",
133
+ "null"
134
+ ]
135
+ },
129
136
  "terminal_command": {
130
137
  "default": null,
131
138
  "description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
@@ -366,6 +373,13 @@ export const GENERATED_MCP_TOOLS =
366
373
  ],
367
374
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
368
375
  },
376
+ "screenshot_file_path": {
377
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
378
+ "type": [
379
+ "string",
380
+ "null"
381
+ ]
382
+ },
369
383
  "snapshot_id": {
370
384
  "default": null,
371
385
  "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index or selector comes from. If a later snapshot replaced\nit, the call is refused before any input. Without it, the element\nresolves in the latest snapshot.",
@@ -560,6 +574,13 @@ export const GENERATED_MCP_TOOLS =
560
574
  ],
561
575
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
562
576
  },
577
+ "screenshot_file_path": {
578
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
579
+ "type": [
580
+ "string",
581
+ "null"
582
+ ]
583
+ },
563
584
  "start_x": {
564
585
  "description": "Start x in raw screenshot capture pixels: the coordinate_width space.\nDivide preview pixels by the screenshot scale first.",
565
586
  "format": "int32",
@@ -602,7 +623,7 @@ export const GENERATED_MCP_TOOLS =
602
623
  "openWorldHint": true,
603
624
  "readOnlyHint": true
604
625
  },
605
- "description": "Get a size-bounded screenshot and accessibility state for a Linux app. Scope the accessibility tree with app_name_or_bundle_identifier or a window_id/pid/app_id/wm_class/title target; omitting a target returns the whole desktop tree and can flood context. Nodes are compact (empty fields left out, actions by name; verbose=true returns every field). Pass interactive_only=true for only actionable nodes, headings and status regions, with the same element_index values. A title or window_id target narrows an app with several windows under one AT-SPI root (Chromium) to that window, keeping the app's popups and dialogs. A window-targeted screenshot needs that window visible and already focused (outside Hyprland background mode); otherwise screenshot_error is set and the tree still returns, so use the screenshot tool to raise and capture it. A tree too large to return inline is written whole to the private file in accessibility_tree_spill. The result's snapshot_id names the tree: element_index and selector actions that pass it are refused once a later tree replaced it. Screenshot results always include coordinate_width, coordinate_height, scale, and format, plus quality for jpeg; callers can request jpeg/quality for compression before resizing.",
626
+ "description": "Get a size-bounded screenshot and accessibility state for a Linux app. Scope the accessibility tree with app_name_or_bundle_identifier or a window_id/pid/app_id/wm_class/title target; omitting a target returns the whole desktop tree and can flood context. Nodes are compact (empty fields left out, actions by name; verbose=true returns every field). Pass interactive_only=true for only actionable nodes, headings and status regions, with the same element_index values. A title or window_id target narrows an app with several windows under one AT-SPI root (Chromium) to that window, keeping the app's popups and dialogs. A window-targeted screenshot needs that window visible and already focused (outside Hyprland background mode); otherwise screenshot_error is set and the tree still returns, so use the screenshot tool to raise and capture it. A tree too large to return inline is written whole to the private file in accessibility_tree_spill. The result's snapshot_id names the tree: element_index and selector actions that pass it are refused once a later tree replaced it. Screenshot results always include coordinate_width, coordinate_height, scale, and format, plus quality for jpeg; callers can request jpeg/quality for compression before resizing. Pass screenshot_file_path (an absolute path to a new file, stdio only; an existing path is refused) to save the screenshot there instead of returning image content; screenshot.file_path then names it.",
606
627
  "inputSchema": {
607
628
  "$defs": {
608
629
  "ScreenshotOutputFormat": {
@@ -732,6 +753,14 @@ export const GENERATED_MCP_TOOLS =
732
753
  "null"
733
754
  ]
734
755
  },
756
+ "screenshot_file_path": {
757
+ "default": null,
758
+ "description": "Save the screenshot to this absolute path instead of returning it as\nimage content; screenshot.file_path then names it. The parent directory\nmust exist, and the extension must match format (.png, or .jpg/.jpeg for\njpeg). The file must not exist: file output creates a new file and\nnever replaces one. Refused over HTTP (mcp --listen). A failed write is\nreported in screenshot_error.",
759
+ "type": [
760
+ "string",
761
+ "null"
762
+ ]
763
+ },
735
764
  "terminal_command": {
736
765
  "default": null,
737
766
  "description": "Terminal command substring, ignoring ASCII case. Resolves a window\ntarget and scopes the tree when possible.",
@@ -979,6 +1008,13 @@ export const GENERATED_MCP_TOOLS =
979
1008
  ],
980
1009
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
981
1010
  },
1011
+ "screenshot_file_path": {
1012
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
1013
+ "type": [
1014
+ "string",
1015
+ "null"
1016
+ ]
1017
+ },
982
1018
  "terminal_command": {
983
1019
  "default": null,
984
1020
  "description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
@@ -1212,6 +1248,13 @@ export const GENERATED_MCP_TOOLS =
1212
1248
  ],
1213
1249
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
1214
1250
  },
1251
+ "screenshot_file_path": {
1252
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
1253
+ "type": [
1254
+ "string",
1255
+ "null"
1256
+ ]
1257
+ },
1215
1258
  "snapshot_id": {
1216
1259
  "default": null,
1217
1260
  "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index or selector comes from. If a later snapshot replaced\nit, the call is refused before any input. Without it, the element\nresolves in the latest snapshot.",
@@ -1385,6 +1428,13 @@ export const GENERATED_MCP_TOOLS =
1385
1428
  ],
1386
1429
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
1387
1430
  },
1431
+ "screenshot_file_path": {
1432
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
1433
+ "type": [
1434
+ "string",
1435
+ "null"
1436
+ ]
1437
+ },
1388
1438
  "terminal_command": {
1389
1439
  "default": null,
1390
1440
  "description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
@@ -1571,6 +1621,13 @@ export const GENERATED_MCP_TOOLS =
1571
1621
  ],
1572
1622
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
1573
1623
  },
1624
+ "screenshot_file_path": {
1625
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
1626
+ "type": [
1627
+ "string",
1628
+ "null"
1629
+ ]
1630
+ },
1574
1631
  "terminal_command": {
1575
1632
  "default": null,
1576
1633
  "description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
@@ -1651,7 +1708,7 @@ export const GENERATED_MCP_TOOLS =
1651
1708
  "openWorldHint": true,
1652
1709
  "readOnlyHint": false
1653
1710
  },
1654
- "description": "Capture the screen and return it as a viewable, size-bounded image. Optionally target a window (window_id/pid/wm_class/title/app_id): the window is raised to the front and the image is cropped before any resize; outside Hyprland background mode, a target without known bounds is refused, so pass full_screen=true there. Returns the image plus a JSON caption with returned and coordinate dimensions, scale, format, quality, source, cropped_to_window, and window_title. The crop origin is not reported; use relative=true on click or scroll for window-relative input. Callers can request jpeg/quality for compression before resizing.",
1711
+ "description": "Capture the screen and return it as a viewable, size-bounded image. Optionally target a window (window_id/pid/wm_class/title/app_id): the window is raised to the front and the image is cropped before any resize; outside Hyprland background mode, a target without known bounds is refused, so pass full_screen=true there. Returns the image plus a JSON caption with returned and coordinate dimensions, scale, format, quality, source, cropped_to_window, and window_title. The crop origin is not reported; use relative=true on click or scroll for window-relative input. Callers can request jpeg/quality for compression before resizing. Pass screenshot_file_path (an absolute path to a new file, stdio only; an existing path is refused) to save the image there instead: the result is then only the caption, with file_path.",
1655
1712
  "inputSchema": {
1656
1713
  "$defs": {
1657
1714
  "ScreenshotOutputFormat": {
@@ -1755,6 +1812,14 @@ export const GENERATED_MCP_TOOLS =
1755
1812
  "null"
1756
1813
  ]
1757
1814
  },
1815
+ "screenshot_file_path": {
1816
+ "default": null,
1817
+ "description": "Save the image to this absolute path instead of returning it as image\ncontent; the caption's file_path then names it. The parent directory\nmust exist, and the extension must match format (.png, or .jpg/.jpeg for\njpeg). The file must not exist: file output creates a new file and\nnever replaces one. Refused over HTTP (mcp --listen).",
1818
+ "type": [
1819
+ "string",
1820
+ "null"
1821
+ ]
1822
+ },
1758
1823
  "title": {
1759
1824
  "default": null,
1760
1825
  "description": "Window title substring, ignoring ASCII case. When several windows\nmatch, the first wins.",
@@ -1932,6 +1997,13 @@ export const GENERATED_MCP_TOOLS =
1932
1997
  ],
1933
1998
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
1934
1999
  },
2000
+ "screenshot_file_path": {
2001
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
2002
+ "type": [
2003
+ "string",
2004
+ "null"
2005
+ ]
2006
+ },
1935
2007
  "snapshot_id": {
1936
2008
  "default": null,
1937
2009
  "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index comes from. If a later snapshot replaced it, the call\nis refused before any input. Without it, the element resolves in the\nlatest snapshot.",
@@ -2139,6 +2211,13 @@ export const GENERATED_MCP_TOOLS =
2139
2211
  ],
2140
2212
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
2141
2213
  },
2214
+ "screenshot_file_path": {
2215
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
2216
+ "type": [
2217
+ "string",
2218
+ "null"
2219
+ ]
2220
+ },
2142
2221
  "snapshot_id": {
2143
2222
  "default": null,
2144
2223
  "description": "snapshot_id of the get_app_state or include_tree result the\nelement_index or selector comes from. If a later snapshot replaced\nit, the call is refused before any input. Without it, the element\nresolves in the latest snapshot.",
@@ -2236,7 +2315,7 @@ export const GENERATED_MCP_TOOLS =
2236
2315
  "openWorldHint": true,
2237
2316
  "readOnlyHint": false
2238
2317
  },
2239
- "description": "Type literal text using keyboard input, optionally after focusing a target window or terminal selector. Success confirms dispatch, not field-content verification; re-observe the target before retrying. X11 xdotool can silently substitute symbols under non-US or changing XKB layouts; inspect the text before pressing Enter or submitting it. The wtype backend rejects NUL and text over 131071 UTF-8 bytes before focus or indicators. On Hyprland, confirmed password fields are rejected, and ambiguous fields and terminals use private stdin for at most 100 Unicode characters per call. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
2318
+ "description": "Type literal text using keyboard input, optionally after focusing a target window or terminal selector. Keyboard success confirms dispatch, not field-content verification; re-observe the target before retrying. On GNOME, when a configured keyboard layout group cannot type a character of the text, the text goes into the focused editable field of the verified target window through AT-SPI EditableText instead: inserted at the caret, replacing a selection, and reported as ok only when the field reads it back with the rest unchanged. Password fields, text with a newline, tab or other control character, and focus outside an editable field of that active window keep the refusal. X11 xdotool can silently substitute symbols under non-US or changing XKB layouts; inspect the text before pressing Enter or submitting it. The wtype backend rejects NUL and text over 131071 UTF-8 bytes before focus or indicators. On Hyprland, confirmed password fields are rejected, and ambiguous fields and terminals use private stdin for at most 100 Unicode characters per call. Pass screenshot_after (true, \"screen\", or \"window\") or include_tree=true to return post-action observation in the same result.",
2240
2319
  "inputSchema": {
2241
2320
  "$defs": {
2242
2321
  "ScreenshotOutputFormat": {
@@ -2343,6 +2422,13 @@ export const GENERATED_MCP_TOOLS =
2343
2422
  ],
2344
2423
  "description": "Capture and return a post-action screenshot in the same result: `true` or\n`\"screen\"` captures the full desktop; `\"window\"` crops to the action's\ntarget window (or the focused window when untargeted); `false` disables capture."
2345
2424
  },
2425
+ "screenshot_file_path": {
2426
+ "description": "Save the post-action screenshot to this absolute path instead of\nreturning it as image content; needs screenshot_after. The parent\ndirectory must exist, and the extension must match format (.png, or\n.jpg/.jpeg for jpeg). The file must not exist: file output creates a\nnew file and never replaces one. Refused over HTTP (mcp --listen). A\nfailed write is reported in screenshot_error, and the action's ok stays\ntrue.",
2427
+ "type": [
2428
+ "string",
2429
+ "null"
2430
+ ]
2431
+ },
2346
2432
  "terminal_command": {
2347
2433
  "default": null,
2348
2434
  "description": "Substring of the command name or command line of a terminal's shell\nor foreground process, ignoring ASCII case.",
@@ -141,7 +141,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
141
141
  ## Procedure
142
142
 
143
143
  1. In Pi, call `computer_use_linux_tools` with the exact tools or capability you need. Enabled tools use the `computer_use_linux_<name>` prefix, appear starting on the next model turn, and remain active for the session.
144
- 2. Begin every desktop-control turn with `get_app_state`, scoped to the app you are working in: pass `app_name_or_bundle_identifier` or a window target (`window_id`, `pid`, `app_id`, `wm_class`, `title`). An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop. Without a target the result is the whole desktop AT-SPI tree, `tree_scoped` is `false`, and `message` warns; that can flood context. Use `include_screenshot: false` when the accessibility tree is sufficient. If `accessibility_tree_truncated` is `true`, the tree is incomplete: scope to a narrower target and raise `max_nodes` or `max_depth` (hard caps 10000 and 64) rather than lowering them. A tree too large to return inline comes back as its leading nodes plus `accessibility_tree_spill.path`, a private JSON file with the whole tree and the same `element_index` values: search it (for a role, name or text) instead of requesting the tree again, or page through it with `get_tree_page` when you cannot read files. Nodes are compact: null, empty, zero and false fields are left out and `actions` lists action names; `verbose: true` returns every field. To find controls in a large tree, pass `interactive_only: true`: it keeps nodes with an action, editable text or a value, plus headings, status and live regions, windows and dialogs, collapses containers, and keeps the same `element_index` values. A `title` or `window_id` target narrows a Chromium-style app with several windows under one AT-SPI root to that window, keeping the app's popups and dialogs. A window-targeted screenshot needs that window visible and already focused (Hyprland background mode excepted); otherwise `screenshot_error` is set and only the tree returns, so use `screenshot` to raise and capture it. The compact readiness block identifies missing setup.
144
+ 2. Begin every desktop-control turn with `get_app_state`, scoped to the app you are working in: pass `app_name_or_bundle_identifier` or a window target (`window_id`, `pid`, `app_id`, `wm_class`, `title`). An `app_id` may be a full desktop ID or its final dot-separated segment; matching is case-insensitive, ambiguous aliases are rejected, and an unresolved explicit `app_id` returns an empty scoped tree rather than the desktop. Without a target the result is the whole desktop AT-SPI tree, `tree_scoped` is `false`, and `message` warns; that can flood context. Use `include_screenshot: false` when the accessibility tree is sufficient. To keep a screenshot out of context until you need it, pass `screenshot_file_path` (an absolute path to a new file whose extension matches `format`; an existing path is refused; stdio only, refused over HTTP): the image is saved there, `screenshot.file_path` names it, and no image content is returned. `screenshot` takes it too. If `accessibility_tree_truncated` is `true`, the tree is incomplete: scope to a narrower target and raise `max_nodes` or `max_depth` (hard caps 10000 and 64) rather than lowering them. A tree too large to return inline comes back as its leading nodes plus `accessibility_tree_spill.path`, a private JSON file with the whole tree and the same `element_index` values: search it (for a role, name or text) instead of requesting the tree again, or page through it with `get_tree_page` when you cannot read files. Nodes are compact: null, empty, zero and false fields are left out and `actions` lists action names; `verbose: true` returns every field. To find controls in a large tree, pass `interactive_only: true`: it keeps nodes with an action, editable text or a value, plus headings, status and live regions, windows and dialogs, collapses containers, and keeps the same `element_index` values. A `title` or `window_id` target narrows a Chromium-style app with several windows under one AT-SPI root to that window, keeping the app's popups and dialogs. A window-targeted screenshot needs that window visible and already focused (Hyprland background mode excepted); otherwise `screenshot_error` is set and only the tree returns, so use `screenshot` to raise and capture it. The compact readiness block identifies missing setup.
145
145
  3. Use `doctor` only when you need the full diagnostic report.
146
146
  4. If `can_build_accessibility_tree` is false, run `setup_accessibility` and restart the target app.
147
147
  5. If `can_query_windows` or `can_focus_windows` is false on GNOME Wayland, run `setup_window_targeting` and ask the user to log out and back in if setup says the shell extension needs a reload.
@@ -149,7 +149,7 @@ Pi native tools skip this MCP `command` config; see [Pi setup](references/pi-set
149
149
  7. Prefer semantic targeting from `get_app_state`: use element indices or role/name/text/states selectors, and pass the `snapshot_id` of the result they came from. Every `get_app_state` and `include_tree` result replaces the one cached tree; a call that names a replaced snapshot is refused before any input, so take the indices and `snapshot_id` from the newest result. Automatic recovery of a dead semantic target requires verified PID/window ownership and invalidates cached indices; call `get_app_state` again before further index-based actions. When several apps are in play, pass the app target (`pid`, `window_id`, `app_id`, `wm_class`, `window_title`) to `click`, `scroll`, `perform_action`, and `set_value`: an index (or a selector, where the tool takes one) from another app's snapshot is then refused instead of acting on that app. On `click` and `scroll` the target also raises and focuses that window, and nothing is sent when focus cannot be verified; on GNOME Wayland, raising a window needs the Shell extension. The title field there is `window_title`. Where the window backend reports no pid (i3 without `xprop`, COSMIC, and wlr-foreign-toplevel on stock labwc and other wlroots compositors), call `get_app_state` for the same window first or pass `pid`, or the call is refused.
150
150
  8. Use coordinates only when the UI surface has no useful accessibility tree.
151
151
  9. For text input, prefer `type_text` with a target selector (`window_id`, `pid`, `app_id`, `wm_class`, `title`, `tty`, `terminal_pid`, `terminal_command`, or `terminal_cwd`) rather than relying on current focus. A `window_id` is valid only as returned by a recent `list_windows`; if a call rejects it as outside 0 to 2^53-1 or matches no window, re-run `list_windows`.
152
- 10. After mutating actions (`click`, `drag`, `scroll`, `press_key`, `type_text`, `perform_action`, `set_value`, `activate_window`, `move_window`, `resize_window`), either pass `screenshot_after: true | "screen" | "window"` (with optional `format`, `quality`, `scale`, `max_width`, `max_height`, `max_bytes`) and/or `include_tree: true` on the action call itself to observe the result in the same tool call, or re-check state with `get_app_state`, `focused_window`, or an app-specific readback. A failed action (`ok: false`) returns no post-action screenshot or tree, so call `get_app_state`; `set_value` still observes when `verified` is `false`.
152
+ 10. After mutating actions (`click`, `drag`, `scroll`, `press_key`, `type_text`, `perform_action`, `set_value`, `activate_window`, `move_window`, `resize_window`), either pass `screenshot_after: true | "screen" | "window"` (with optional `format`, `quality`, `scale`, `max_width`, `max_height`, `max_bytes`, and `screenshot_file_path` to save the image to a file instead of returning it) and/or `include_tree: true` on the action call itself to observe the result in the same tool call, or re-check state with `get_app_state`, `focused_window`, or an app-specific readback. A failed action (`ok: false`) returns no post-action screenshot or tree, so call `get_app_state`; `set_value` still observes when `verified` is `false`.
153
153
  11. For `set_value`, `verified: true` is the only exact-content confirmation. `verified: false` requires fresh observation; password fields are deliberately unverifiable. The field is omitted for numeric values and early failures.
154
154
 
155
155
  On Hyprland, `type_text` passes one literal wtype argument after `--` for a
@@ -164,8 +164,11 @@ The input probe has its own 1.5-second budget; the 250 ms indicator probe only
164
164
  masks text. Both protections apply with the indicator disabled. Other compatible
165
165
  compositors retain their existing stdin path. Wtype rejects NUL and text over
166
166
  131,071 UTF-8 bytes before focusing a window or displaying text. `ok: true`
167
- confirms dispatch only; every text backend leaves field contents unverified.
168
- Re-observe before retrying. Exact window focus alone does not suppress missing-focus
167
+ from a keyboard backend confirms dispatch only and leaves field contents
168
+ unverified. Only the GNOME AT-SPI fallback reads the field back: its success
169
+ is `ok: true` with a message that starts with "Typed through AT-SPI
170
+ EditableText". Its failures have `ok: false` and start with "AT-SPI
171
+ EditableText insert was not verified". Re-observe before retrying. Exact window focus alone does not suppress missing-focus
169
172
  warnings: only a complete scoped tree containing window/menu chrome without
170
173
  content makes the feedback informational. Such trees get a conditional
171
174
  Electron/Chromium hint computed before compaction; this is not positive Electron