@agent-sh/computer-use-linux 0.7.10 → 0.7.12

package/README.md

@@ -26,7 +26,7 @@ The Rust crate is published as [`computer-use-linux`](https://crates.io/crates/c
 
 Most computer-use MCP servers are macOS-only (they lean on AppKit, AXUIElement, CGEvent). The few that target Linux either drive `xdotool` against an X11 root window or shell out to OCR over screenshots. Four things set this one apart:
 
-- **Wayland actually works.** Pointer actions can use the `org.freedesktop.portal.RemoteDesktop` interface on Wayland, with `ydotool` / `ydotoold` (uinput) as the deterministic fallback. Literal text prefers `wtype` on compatible Wayland compositors when portal keyboard input is unavailable, preserving Unicode and the active layout before falling back to ydotool. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, then on a native X11 session a root-window `GetImage` (device pixels, no toolkit scaling), and finally spawn `gnome-screenshot` for background/systemd contexts where the DBus paths are denied.
+- **Wayland actually works.** Pointer actions can use the `org.freedesktop.portal.RemoteDesktop` interface on Wayland, with `ydotool` / `ydotoold` (uinput) as the deterministic fallback. Literal text prefers KDE clipboard integration on Plasma, `wtype` on compatible Wayland compositors, and portal keysyms on other Wayland sessions, including GNOME with ydotool installed. Portal text startup or conversion errors fail the call without replaying text through ydotool. Raw ydotool text accepts printable ASCII, tab and newline only, and its physical US key positions can produce different text under the active layout. Screenshots use the GNOME Shell DBus screenshot method when present, `org.freedesktop.portal.Screenshot` otherwise, then on a native X11 session a root-window `GetImage` (device pixels, no toolkit scaling), and finally spawn `gnome-screenshot` for background/systemd contexts where the DBus paths are denied.
 - **Window targeting is compositor-aware.** The window registry tries GNOME Shell extension, GNOME Shell Introspect, COSMIC Wayland helper, KWin DBus scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, and generic X11/EWMH in order, then reports exactly which backend won or why each backend failed.
 - **Semantic selectors, not pixel coordinates.** Tools like `click`, `perform_action`, and `set_value` accept `role` / `name` / `text` / `states` selectors backed by AT-SPI. Pixel coordinates remain available as a fallback for rendering-only surfaces (canvas, games, X clients without ATK).
 - **One JSON readiness report.** `computer-use-linux doctor` returns a structured document covering platform, portals, AT-SPI, windowing, input, and a `readiness` summary with explicit blockers and a recommended next step. MCP hosts can render or surface that to the user without parsing prose.
@@ -80,6 +80,8 @@ the origin. A missing window target is rejected.
 
 Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report. It also reports `tree_scoped` (false when no app target narrowed the AT-SPI tree, with a warning in `message`) and `accessibility_tree_truncated` (true when the node, depth, or read budget stopped traversal with unread elements left).
 
+On GNOME, a read-only keymap preflight rejects the whole string before input if a character is absent or requires a modifier level Mutter cannot synthesize. The background reader cannot establish the active layout group, so every configured group must support the text. This can reject text that works in one of several groups; use `set_value` on an editable field instead. The check requires `libxkbcommon.so.0` at runtime and stops after a two-second keymap read deadline. It does not cover held modifiers or a keymap changed concurrently with typing.
+
 **Semantic actions**
 
 - `perform_action` — invoke any AT-SPI action exposed by an element (`Press`, `Activate`, `Toggle`, …); defaults to the primary action
@@ -90,6 +92,16 @@ Targeted `press_key`/`type_text` results append focused-element feedback from AT
 - `activate_window` — focus a window by `window_id`, `pid`, `app_id`, `wm_class`, `title`, or terminal selectors
 - `move_window` / `resize_window` — reposition or resize a window in desktop coordinates (GNOME Shell extension backend); useful to recover windows that are partially off-screen
 
+**On-screen indicator**
+
+While an agent acts, the server drives `computer-use-linux-indicator`, a click-through overlay that shows what it is doing: a software cursor tinted per agent that glides to each coordinate `click`, `drag`, or `scroll` target and ripples on click (AT-SPI actions such as `perform_action`, `set_value`, and element clicks show in the pill without moving it, since no pointer moves); keycaps for `press_key` and a typing line for `type_text`; a glow along every screen edge; and a status pill such as *Claude is using your computer · click*. Pointer actions wait 350 ms so the cursor lands before the real input. While `screenshot` or `get_app_state` captures the screen, the overlay draws nothing, so agents never see it: not their own, not one shown for another agent, and not one that another agent starts mid-capture. This holds with `COMPUTER_USE_LINUX_INDICATOR=0` too, which only stops a server from showing its own actions. The capture waits for the compositor only when something was on screen. The agent name comes from the MCP client's `clientInfo`.
+
+The server starts the indicator on the first action and it exits after ten idle minutes. It stays hidden between bursts of activity. Compositors with `wlr-layer-shell` use the shared-memory overlay. GNOME uses native Shell actors through the bundled extension. Run `setup_window_targeting` once, then log out and back in after installing or updating the extension. Both renderers use the same capture lock and acknowledgement protocol. Set `COMPUTER_USE_LINUX_INDICATOR=0` to turn it off.
+
+Screenshot processing uses one blocking worker per server. Captures enter one pipeline before reading pixels, so concurrent requests do not queue decoded full-screen images. A source can contain up to 134,217,728 pixels; serial processing bounds decoded-image memory while letting MCP answer other requests during resizing and encoding. Session discovery is reused after a successful probe and refreshed when session variables change. Failed discovery is retried.
+
+Captures require the shared capture lock and, when an overlay is running, its acknowledgement that it has hidden. A missing or invalid acknowledgement, a stalled socket, or a failed lock returns a capture error. Concurrent captures keep separate holds; their acknowledgements are exchanged in order.
+
 **Conditional host execution**
 
 - `complete_interaction` - optional desktop completion notification, registered only with `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE=1`. Repeated calls can create repeated notifications; it does not provide desktop exclusivity.
@@ -138,7 +150,7 @@ Validated manually on Ubuntu 25.10 (GNOME Shell 50.1, Wayland). niri window list
 | Hyprland | `hyprctl clients -j` and `hyprctl dispatch focuswindow` | Requires `hyprctl` in the desktop session. |
 | niri | `niri msg` with a direct JSON IPC fallback | Lists toplevels and focuses exact window IDs. Export the session's `NIRI_SOCKET`; otherwise discovery requires an unambiguous socket matching `WAYLAND_DISPLAY`. The fallback works without the `niri` binary. Missing positions remain `null`; bounds are omitted if output scaling is unknown or mixed. |
 | i3 | `i3-msg`; optional `xprop` for PID hydration | Lists and focuses i3 windows over the active i3 IPC socket. |
-| COSMIC Wayland | `computer-use-linux-cosmic` helper | Installed automatically by `./install.sh`, `cargo install`, and npm. For custom/manual layouts, put the helper next to the main binary, on `PATH`, or point `COMPUTER_USE_LINUX_COSMIC_HELPER` at it. |
+| COSMIC Wayland | `computer-use-linux-cosmic` helper | Installed automatically by `./install.sh`, `cargo install`, and npm. For custom/manual layouts, put the helper next to the main binary, on `PATH`, or point `COMPUTER_USE_LINUX_COSMIC_HELPER` at it. The on-screen indicator was validated here (three outputs, mixed fractional scaling). |
 | Sway / generic wlroots | no dedicated backend yet | AT-SPI, screenshots, and global `ydotool` input can still work; exact window list/focus is currently unavailable unless another backend applies. |
 | Generic X11 / XFCE / other EWMH WMs | `wmctrl` plus `xprop` | Lists, focuses, moves, and resizes windows; keyboard input prefers `xdotool`/XTEST, and scroll uses xdotool wheel buttons. Window origins are read from the X server, since `wmctrl -lG` counts the frame offset twice. |
 
@@ -207,7 +219,7 @@ Linux x86_64 / aarch64 builds are published with each tag. Each binary ships a `
 ```bash
 target=x86_64-unknown-linux-gnu
 base=https://github.com/agent-sh/computer-use-linux/releases/latest/download
-for binary in computer-use-linux computer-use-linux-cosmic; do
+for binary in computer-use-linux computer-use-linux-cosmic computer-use-linux-indicator; do
   asset="$binary-$target"
   curl -L -O "$base/$asset"
   curl -L -O "$base/$asset.sha256"
@@ -384,7 +396,7 @@ Spawn the binary with `["mcp"]` as the argv tail. It speaks JSON-RPC over stdio
    computer-use-linux doctor | jq .readiness
    ```
 
-   Aim for `can_register_mcp_tools`, `can_build_accessibility_tree`, `can_send_development_input`, `can_query_windows`, and `can_capture_screenshots` all `true`. The `blockers` array should be empty. `can_capture_screenshots` means a route was detected, not that a test capture succeeded.
+   Aim for `can_register_mcp_tools`, `can_build_accessibility_tree`, `can_send_development_input`, and `can_query_windows` all `true`, with an empty `blockers` array. Doctor lists detected screenshot routes in `capabilities.screenshot`, but does not take a screenshot or request consent. Detected routes are always `screenshot_capture_status: "unverified"` and `can_capture_screenshots: false` in a doctor report. `get_app_state` records the result of its requested raw capture as `verified` or `failed`; only `verified` sets `can_capture_screenshots: true` in that response. If no route is detected, the status is `unavailable`. An unverified capture is a warning, not an installation blocker.
 
 2. **If `accessibility.at_spi_bus.ok = false`** — run `computer-use-linux setup` (or call the `setup_accessibility` MCP tool). This sets:
    - `org.gnome.desktop.interface toolkit-accessibility true`
@@ -393,7 +405,7 @@ Spawn the binary with `["mcp"]` as the argv tail. It speaks JSON-RPC over stdio
 
 3. **If `windowing.can_list_windows = false`** — inspect `doctor.windowing.backends`. On GNOME Wayland, run `computer-use-linux setup-window-targeting` (or call `setup_window_targeting`) to install the bundled `computer-use-linux@avifenesh.dev` Shell extension, then log out and back in so GNOME Shell loads it. On KDE, Hyprland, niri, i3, COSMIC, or generic X11, install or expose the matching compositor tool/helper shown in the backend details.
 
-4. **Grant the screencast portal on first screenshot.** The first time `get_app_state` or any screenshot subcommand runs, GNOME will pop a portal dialog asking to share the screen. Accept once and tick "remember" to make it sticky for the session.
+4. **Verify screenshot capture.** Call `get_app_state` or run `computer-use-linux screenshot`. The Screenshot portal may request consent for the caller's app ID. On GNOME, that dialog can be rejected when the app ID does not match the focused app. A detected portal interface alone does not prove capture permission. Inspect the full backend error and the desktop portal logs if capture fails; `gnome-screenshot` is an optional fallback when GNOME Shell or portal calls are denied.
 
 5. **Confirm compatible ydotool 1.0.3+ and `ydotoold` are available.**
 
@@ -413,10 +425,15 @@ Most setups need none of these — `doctor` and the installers pick sensible def
 | --- | --- |
 | `COMPUTER_USE_LINUX_NOTIFY_ON_COMPLETE` | Set exactly to `1` to expose the optional `complete_interaction` notification tool. Requires `notify-send` and a desktop notification service; disabled by default. |
 | `COMPUTER_USE_LINUX_COSMIC_HELPER` | Path to the `computer-use-linux-cosmic` helper when it isn't next to the binary or on `PATH`. |
-| `CU_DISABLE_ABS_POINTER` | Disable the uinput absolute pointer and click through `ydotool` instead for setups where the abs-pointer device misbehaves. |
-| `COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through the RemoteDesktop portal on Wayland, skipping auto-detection. |
+| `COMPUTER_USE_LINUX_INDICATOR` | Set to `0` (or `false`/`off`/`no`) to turn off the on-screen indicator. On by default. |
+| `COMPUTER_USE_LINUX_INDICATOR_BIN` | Path to the `computer-use-linux-indicator` overlay when it isn't next to the binary or on `PATH`. |
+| `COMPUTER_USE_LINUX_INDICATOR_HIDE_TEXT` | Set exactly to `1` to mask all typed text and keycaps in the indicator. `set_value` values are always masked. |
+| `COMPUTER_USE_LINUX_AGENT_NAME` | Name shown in the indicator instead of the one derived from the MCP client's `clientInfo`. |
+| `CU_DISABLE_ABS_POINTER` | Disable the uinput absolute pointer. Wayland coordinate actions then need the RemoteDesktop portal; they refuse input rather than use ydotool's relative-motion approximation. Native X11 fallback is unchanged. |
+| `COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER` / `…_KEYBOARD` | Prefer the RemoteDesktop portal on Wayland, skipping auto-detection. Pointer forcing skips the absolute uinput backend. Forced ydotool or XTEST keyboard overrides take precedence; KDE literal text keeps its clipboard route. Portal startup and text conversion failures return errors without replaying through ydotool. |
+| `COMPUTER_USE_LINUX_PORTAL_SCROLL_INVERT` | Set to `1` to reverse vertical scrolling through the RemoteDesktop portal. Defaults to normal direction on both GNOME and KDE Plasma; horizontal direction is unchanged. |
 | `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP` | Set exactly to `1` to ask the RemoteDesktop portal to remember pointer and keyboard grants across processes. The first dialog still appears (on GNOME the remember box starts checked). Later processes reuse the single-use restore token from `Start`, stored mode `0600` under `$XDG_STATE_HOME/computer-use-linux/` (or `~/.local/state/computer-use-linux/`). Pointer and keyboard tokens are separate. Needs interface version 2. Unset, every new process is prompted. No effect when input is not using the portal. |
-| `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through `ydotool`, skipping the portal and KDE clipboard paths; pointer forcing also skips native-X11 `xdotool` coordinate clicks and scrolls. |
+| `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER` / `…_KEYBOARD` | Always route pointer / keyboard through `ydotool`, skipping the portal and KDE clipboard paths; pointer forcing also skips the absolute uinput backend and native-X11 `xdotool` coordinate clicks and scrolls. Coordinate pointer actions on Wayland then refuse input because ydotool cannot position them accurately; scrolling at the current pointer remains available. |
 | `YDOTOOL_SOCKET` | Explicit ydotool socket path. When unset, only an owner-only `.ydotool_socket` in the user's private runtime directory is accepted; the legacy shared `/tmp/.ydotool_socket` is never selected automatically. Setting this variable is an operator trust override for custom layouts. |
 | `COMPUTER_USE_LINUX_FORCE_XDOTOOL_KEYBOARD` | Prefer `xdotool`/XTEST keyboard input when `DISPLAY` is available. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_KEYBOARD=1` takes precedence. |
 | `COMPUTER_USE_LINUX_XDOTOOL_TYPE_DELAY_MS` | Per-character delay for `xdotool type` in milliseconds (default `12`). `0` is faster but can deliver characters out of order on some X servers. |
@@ -436,7 +453,7 @@ files.
 | `COMPUTER_USE_LINUX_BIN` | Run this absolute binary path instead of the one bundled by the npm package. Relative paths are ignored. |
 | `COMPUTER_USE_LINUX_DOWNLOAD_BASE` | Override the GitHub release base URL the installer downloads from (mirrors, air-gapped hosts). |
 | `COMPUTER_USE_LINUX_SKIP_DOWNLOAD=1` | Skip the post-install binary download entirely. |
-| `COMPUTER_USE_LINUX_LOCAL_BINARY` / `…_LOCAL_COSMIC_HELPER` | Install from a local build instead of downloading (used by CI and local testing). |
+| `COMPUTER_USE_LINUX_LOCAL_BINARY` / `…_LOCAL_COSMIC_HELPER` / `…_LOCAL_INDICATOR` | Install from a local build instead of downloading (used by CI and local testing). |
 
 ## Architecture
 
@@ -444,13 +461,15 @@ files.
 - **DBus where desktops expose it** — [`zbus`](https://crates.io/crates/zbus) for portal calls (`org.freedesktop.portal.Screenshot`, `…RemoteDesktop`, `…ScreenCast`), GNOME Shell screenshots (`org.gnome.Shell.Screenshot`), the bundled GNOME extension's `dev.avifenesh.ComputerUseLinux.WindowControl` service, and temporary KWin scripting.
 - **MCP transport** — [`rmcp`](https://crates.io/crates/rmcp) with the `transport-io` feature; stdio framing, no network.
 - **Input fallback** — on X11, keyboard input prefers `xdotool`/XTEST and falls back only when xdotool cannot launch. On Wayland, literal text uses `wtype` when installed and the remote-desktop portal is unavailable; `wtype` supports Unicode through the virtual-keyboard protocol on compatible compositors such as Hyprland/wlroots. If wtype is unavailable, the binary falls back to a compatible ydotool 1.0.3+ CLI and `ydotoold` socket. A launched wtype failure is returned without replaying the text. `install.sh` can configure `ydotoold`; the `setup` command only enables the GNOME AT-SPI bridge.
+- **Wayland coordinate input** — coordinate clicks, drags, and targeted scrolls require the absolute uinput pointer or an approved RemoteDesktop portal session. Targeted scroll positions through uinput before sending only wheel events through ydotool. If accurate positioning is unavailable, the call returns `ok: false` before sending coordinate input. The absolute pointer still requires a successful full-desktop capture when first created; an existing device does not need another capture. Scrolling without a point or window/element target keeps using the current pointer position.
 - **Native X11 coordinate clicks** — eligible native X11 sessions use one supervised `xdotool mousemove -- X Y click --repeat N BUTTON` command for left, middle, and right clicks; ydotool is used only when xdotool cannot launch, while a launched nonzero xdotool command is reported as an error without replay. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this xdotool path.
-- **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. A missing xdotool falls back to ydotool; a launched nonzero xdotool command does not. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path. Wayland portal scroll is unchanged.
+- **Native X11 scroll.** The same sessions send one `xdotool` command, `mousemove -- X Y click --delay 12 --repeat N BUTTON` when a point is set, or `click` alone when it is not. Buttons are 4 up, 5 down, 6 left, and 7 right, and `N` is the same notch count ydotool would pass to `mousemove --wheel` (five per page). ydotool's absolute move warps through (0, 0); GTK 3 then drops the single following wheel event because re-entry resets its XI2 scroll valuators. XTEST has no scroll valuators, so GTK takes the button path. A missing xdotool falls back to ydotool; a launched nonzero xdotool command does not. `COMPUTER_USE_LINUX_FORCE_YDOTOOL_POINTER=1` skips this path.
 - **Native X11 transport.** Native X11 queries (window origins, root screenshots, `doctor`) connect and read replies under one shared 2-second deadline enforced in the transport itself, including the handshake and every reply. A stalled X server closes the query connection instead of leaving a blocked worker behind. Unix sockets, literal IP addresses, and `localhost` need no resolver. A remote hostname in `DISPLAY` needs `getent ahosts` for bounded name resolution; if it is missing or fails, the X11 route is reported as unavailable.
 - **Window registry** — `list_windows`, `focused_window`, `activate_window`, `press_key`, and `type_text` share a backend registry. It tries GNOME extension, GNOME Introspect, COSMIC helper, KWin scripting, Hyprland `hyprctl`, the niri IPC socket, i3 IPC, and generic X11/EWMH in that order, skipping empty or failed backends so another compositor backend can answer.
 - **niri backend** — prefers `niri msg`, falling back to direct JSON IPC over the same session socket. `is_minimized` maps to `hidden`. Bounds use `layout.window_size` (or `tile_size`) and, when available, the tile position plus the window's offset and the output's logical origin. Uniform output scaling converts these to device pixels; unknown or mixed scaling omits bounds. Missing window positions stay `null`, so listing and focusing still work without enabling coordinate targeting. An accepted focus action is verified by querying the focused window again.
 - **GNOME extension fallback** — recent GNOME builds deny `org.gnome.Shell.Introspect.GetWindows` to non-blessed clients. The bundled Shell extension exposes window data and exact activation under `dev.avifenesh.ComputerUseLinux.WindowControl`.
 - **COSMIC helper** — `computer-use-linux-cosmic` talks to COSMIC toplevel protocols and is resolved from `COMPUTER_USE_LINUX_COSMIC_HELPER`, next to the running binary, or from `PATH`.
+- **Indicator overlay** — the server sends one JSON datagram per action (`{"agent","tool","x","y","space","keys","text"}`, where `x`/`y` are screenshot pixels and `space` is the capture size; or `{"capture":"begin"}`/`{"capture":"end"}` around captures; `begin` is answered with `shown` or `empty` when the sender's socket is bound) to `$XDG_RUNTIME_DIR/computer-use-linux-indicator.sock`. `computer-use-linux-indicator` (resolved like the COSMIC helper, via `COMPUTER_USE_LINUX_INDICATOR_BIN`) draws nothing while any server holds a shared `flock` on `computer-use-linux-indicator.capture` next to the socket (held for the length of each capture and released by the kernel if a server dies), and otherwise draws on small `wlr-layer-shell` surfaces with [`tiny-skia`](https://crates.io/crates/tiny-skia) into shared-memory buffers, honouring fractional scaling through `wp_fractional_scale_v1` and `wp_viewporter`. The cursor surface moves by changing its margins, so a glide redraws nothing. On GNOME the helper relays events to the verified Shell extension, which draws non-interactive actors. The helper still owns the shared socket and probes the capture lock. If the helper exits, the extension hides its actors when its D-Bus connection disappears. Captures also hide residual GNOME actors when no helper is listening.
 - **Terminal enrichment** — `list_windows` cross-references each terminal window with its controlling TTY and the foreground process on that TTY, so `type_text` / `press_key` can target "the terminal where `pytest` is running" without the host ever knowing the window id.
 
 ## Security
@@ -458,10 +477,11 @@ files.
 Computer-use tooling is, by definition, a privilege-escalation surface. The threat model:
 
 - **`ydotoold` runs as a per-user service** with read/write access to `/dev/uinput`. `install.sh` automates this for systemd user sessions and prints manual supervisor guidance elsewhere. Any process that can connect to its socket (`/run/user/$UID/.ydotool_socket`, mode `0600` by default) can synthesize arbitrary input — keypresses, clicks, anything. Keep the socket in the user runtime dir (the default), not in `/tmp` or any world-readable location. Do not run `ydotoold` as root or as a system service.
-- **The screencast portal asks for permission once per session.** Granting it lets the calling MCP host capture the screen for the rest of the session. If you don't want that, decline the portal dialog and use `get_app_state` with `include_screenshot: false`.
+- **Screenshot consent depends on the portal and caller's app ID.** A saved permission can apply to multiple processes with that identity. Doctor only detects the available routes and does not request consent. Use `get_app_state` with `include_screenshot: false` to skip capture.
 - **Persisted remote control is opt-in.** `COMPUTER_USE_LINUX_PERSIST_REMOTE_DESKTOP=1` stores the portal's single-use restore token in the user state directory, mode `0600`. A same-user process that can read that file can restore remote control without a new prompt until the desktop revokes the grant. Leave the variable unset to keep a prompt on every new process.
 - **AT-SPI exposes window contents to any client on your session bus.** Enabling the AT-SPI bridge (`setup_accessibility`) is a prerequisite for this binary; it's also what screen readers use, and it shares the same trust boundary.
 - **The GNOME Shell extension** is loaded only into your user's GNOME Shell, runs in the Shell's JS sandbox, and exposes a single DBus interface on the user session bus. It does not request any extra permissions.
+- **The indicator shows what agents type.** `type_text` text (its last 64 characters) and `press_key` keycaps are drawn on screen unless the target is, or might be, a password field: both are replaced by a fixed mask when the focused element's AT-SPI role is a password role or cannot be read within 250 ms. Generic text boxes and terminal focus are also masked because their roles do not establish whether input is obscured. `set_value` values are always masked, because an app can change a field to a password after its accessibility snapshot. The text also travels over a socket in the user's private runtime directory. Set `COMPUTER_USE_LINUX_INDICATOR_HIDE_TEXT=1` to mask all typed text and keycaps, or `COMPUTER_USE_LINUX_INDICATOR=0` to turn the indicator off.
 - **No network.** This binary opens no TCP/UDP listener, makes no outbound Internet connections, and ships no telemetry. It does use local session transports such as DBus and the per-user `ydotoold` Unix socket.
 - **Mutating tools are explicit.** The MCP tool list annotates read-only versus mutating tools, and CI fails if the published tool annotations drift from the table above. Treat those annotations as hints; the host is still responsible for user approval and policy.
 

package/npm/README.md

@@ -79,7 +79,9 @@ mcp_servers:
 The package downloads the matching Linux x86_64 or aarch64 binary from the
 GitHub release for this package version and verifies the `.sha256` asset before
 installing it. It also installs the matching `computer-use-linux-cosmic` helper
-used for COSMIC desktop window targeting.
+used for COSMIC desktop window targeting and the `computer-use-linux-indicator`
+overlay that shows agent activity on screen (`COMPUTER_USE_LINUX_INDICATOR=0`
+turns it off).
 
 When installed through Pi, the package supplies native, dynamically loaded
 `computer_use_linux_*` tools. No separate MCP adapter or manual MCP

package/npm/bin/computer-use-linux.js

@@ -9,6 +9,7 @@ const binaryName = `computer-use-linux-${process.platform}-${process.arch}`;
 const bundledBinary = path.join(__dirname, binaryName);
 const binary = process.env.COMPUTER_USE_LINUX_BIN || bundledBinary;
 const bundledCosmicHelper = path.join(__dirname, 'computer-use-linux-cosmic');
+const bundledIndicator = path.join(__dirname, 'computer-use-linux-indicator');
 
 if (!fs.existsSync(binary)) {
   console.error(
@@ -24,6 +25,9 @@ const env = { ...process.env };
 if (!env.COMPUTER_USE_LINUX_COSMIC_HELPER && fs.existsSync(bundledCosmicHelper)) {
   env.COMPUTER_USE_LINUX_COSMIC_HELPER = bundledCosmicHelper;
 }
+if (!env.COMPUTER_USE_LINUX_INDICATOR_BIN && fs.existsSync(bundledIndicator)) {
+  env.COMPUTER_USE_LINUX_INDICATOR_BIN = bundledIndicator;
+}
 
 const child = spawn(binary, process.argv.slice(2), {
   stdio: 'inherit',

package/npm/install.js

@@ -17,6 +17,7 @@ const archToTarget = {
 const binDir = path.join(__dirname, 'bin');
 const binaryPath = path.join(binDir, `computer-use-linux-${process.platform}-${process.arch}`);
 const cosmicHelperPath = path.join(binDir, 'computer-use-linux-cosmic');
+const indicatorPath = path.join(binDir, 'computer-use-linux-indicator');
 
 function fail(message) {
   console.error(`[computer-use-linux] ${message}`);
@@ -31,6 +32,10 @@ function copyLocalBinary(source) {
     fs.copyFileSync(process.env.COMPUTER_USE_LINUX_LOCAL_COSMIC_HELPER, cosmicHelperPath);
     fs.chmodSync(cosmicHelperPath, 0o755);
   }
+  if (process.env.COMPUTER_USE_LINUX_LOCAL_INDICATOR) {
+    fs.copyFileSync(process.env.COMPUTER_USE_LINUX_LOCAL_INDICATOR, indicatorPath);
+    fs.chmodSync(indicatorPath, 0o755);
+  }
   console.log(`[computer-use-linux] installed local binary from ${source}`);
 }
 
@@ -51,7 +56,9 @@ function download(url, destination, redirects = 5) {
 
       if (response.statusCode !== 200) {
         response.resume();
-        reject(new Error(`download failed with HTTP ${response.statusCode}: ${url}`));
+        const error = new Error(`download failed with HTTP ${response.statusCode}: ${url}`);
+        error.statusCode = response.statusCode;
+        reject(error);
         return;
       }
 
@@ -64,6 +71,19 @@ function download(url, destination, redirects = 5) {
   });
 }
 
+// Releases before the overlay omit its asset; only HTTP 404 is optional.
+async function downloadOptional(url, destination) {
+  try {
+    await download(url, destination);
+    return true;
+  } catch (error) {
+    if (error.statusCode === 404) {
+      return false;
+    }
+    throw error;
+  }
+}
+
 function parseSha256(text) {
   const match = text.match(/\b[a-fA-F0-9]{64}\b/);
   if (!match) {
@@ -100,6 +120,7 @@ async function main() {
 
   const asset = `computer-use-linux-${targetArch}-unknown-linux-gnu`;
   const cosmicAsset = `computer-use-linux-cosmic-${targetArch}-unknown-linux-gnu`;
+  const indicatorAsset = `computer-use-linux-indicator-${targetArch}-unknown-linux-gnu`;
   const baseUrl =
     process.env.COMPUTER_USE_LINUX_DOWNLOAD_BASE ||
     `https://github.com/agent-sh/computer-use-linux/releases/download/v${pkg.version}`;
@@ -108,6 +129,8 @@ async function main() {
   const tmpSha = path.join(tmpDir, `${asset}.sha256`);
   const tmpCosmic = path.join(tmpDir, cosmicAsset);
   const tmpCosmicSha = path.join(tmpDir, `${cosmicAsset}.sha256`);
+  const tmpIndicator = path.join(tmpDir, indicatorAsset);
+  const tmpIndicatorSha = path.join(tmpDir, `${indicatorAsset}.sha256`);
 
   try {
     console.log(`[computer-use-linux] downloading ${asset} from ${baseUrl}`);
@@ -115,17 +138,29 @@ async function main() {
     await download(`${baseUrl}/${asset}.sha256`, tmpSha);
     await download(`${baseUrl}/${cosmicAsset}`, tmpCosmic);
     await download(`${baseUrl}/${cosmicAsset}.sha256`, tmpCosmicSha);
+    const hasIndicator = await downloadOptional(`${baseUrl}/${indicatorAsset}`, tmpIndicator);
+    if (hasIndicator) {
+      await download(`${baseUrl}/${indicatorAsset}.sha256`, tmpIndicatorSha);
+    }
 
     const expected = parseSha256(fs.readFileSync(tmpSha, 'utf8'));
     const actual = sha256File(tmpBinary);
     if (actual !== expected) {
-      fail(`sha256 mismatch for ${asset}: expected ${expected}, got ${actual}`);
+      throw new Error(`sha256 mismatch for ${asset}: expected ${expected}, got ${actual}`);
     }
 
     const expectedCosmic = parseSha256(fs.readFileSync(tmpCosmicSha, 'utf8'));
     const actualCosmic = sha256File(tmpCosmic);
     if (actualCosmic !== expectedCosmic) {
-      fail(`sha256 mismatch for ${cosmicAsset}: expected ${expectedCosmic}, got ${actualCosmic}`);
+      throw new Error(`sha256 mismatch for ${cosmicAsset}: expected ${expectedCosmic}, got ${actualCosmic}`);
+    }
+
+    if (hasIndicator) {
+      const expectedIndicator = parseSha256(fs.readFileSync(tmpIndicatorSha, 'utf8'));
+      const actualIndicator = sha256File(tmpIndicator);
+      if (actualIndicator !== expectedIndicator) {
+        throw new Error(`sha256 mismatch for ${indicatorAsset}: expected ${expectedIndicator}, got ${actualIndicator}`);
+      }
     }
 
     fs.mkdirSync(binDir, { recursive: true });
@@ -133,7 +168,13 @@ async function main() {
     fs.chmodSync(binaryPath, 0o755);
     fs.copyFileSync(tmpCosmic, cosmicHelperPath);
     fs.chmodSync(cosmicHelperPath, 0o755);
-    console.log(`[computer-use-linux] installed ${asset} and ${cosmicAsset}`);
+    if (hasIndicator) {
+      fs.copyFileSync(tmpIndicator, indicatorPath);
+      fs.chmodSync(indicatorPath, 0o755);
+    } else {
+      fs.rmSync(indicatorPath, { force: true });
+    }
+    console.log(`[computer-use-linux] installed ${asset} and ${cosmicAsset}${hasIndicator ? ` and ${indicatorAsset}` : ' (release has no indicator)'}`);
   } finally {
     fs.rmSync(tmpDir, { recursive: true, force: true });
   }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.7.10",
+  "version": "0.7.12",
   "description": "Linux desktop-control MCP server: AT-SPI accessibility trees, Wayland/X11 input, screenshots, and compositor window targeting.",
   "license": "MIT",
   "type": "commonjs",

package/pi/extension/THIRD_PARTY_NOTICES.txt

@@ -3,7 +3,7 @@ Third-party notices for pi/extension/mcp-client.bundle.cjs
 This file is generated by pi/scripts/build.mjs.
 
 ==============================================================================
-@modelcontextprotocol/client@2.0.0 (MIT)
+@modelcontextprotocol/client@2.3.1 (Apache-2.0)
 ==============================================================================
 The MCP project is undergoing a licensing transition from the MIT License to the Apache License, Version 2.0 ("Apache-2.0"). All new code and specification contributions to the project are licensed under Apache-2.0. Documentation contributions (excluding specifications) are licensed under CC-BY-4.0.
 
@@ -223,7 +223,7 @@ CC-BY-4.0. See https://creativecommons.org/licenses/by/4.0/legalcode for
 the full license text.
 
 ==============================================================================
-@modelcontextprotocol/core@2.0.0 (MIT)
+@modelcontextprotocol/core@2.3.1 (Apache-2.0)
 ==============================================================================
 The MCP project is undergoing a licensing transition from the MIT License to the Apache License, Version 2.0 ("Apache-2.0"). All new code and specification contributions to the project are licensed under Apache-2.0. Documentation contributions (excluding specifications) are licensed under CC-BY-4.0.
 

package/pi/extension/generated-tools.ts

@@ -7,9 +7,9 @@ export interface GeneratedMcpToolDefinition {
 	annotations: Record<string, unknown>;
 }
 
-export const GENERATED_SERVER_VERSION = "0.7.10";
-export const GENERATED_TOOL_CATALOG_HASH = "b65768d5f3112b43fcde1344223f5354557c871969987a0a49b58ebe5605eb70";
-export const GENERATED_SHELL_TOOL_CATALOG_HASH = "984f033a6045848f85209b7e74bf881f7810bcfe1fce820e2c3967fee52a6714";
+export const GENERATED_SERVER_VERSION = "0.7.12";
+export const GENERATED_TOOL_CATALOG_HASH = "666f4649084b7c1a240e6a0eacc01a55379b64a579355f50071211437e91229c";
+export const GENERATED_SHELL_TOOL_CATALOG_HASH = "554322e70886f5c1e1a3ad06869c5be91c140103f71350ad635160e128c1563d";
 export const GENERATED_MCP_TOOLS =
 [
   {

package/pi/extension/index.ts

@@ -226,6 +226,10 @@ function runtimeEnvironment(): Record<string, string> {
 		"NIX_LD_LIBRARY_PATH",
 		"RUST_LOG",
 		"COMPUTER_USE_LINUX_COSMIC_HELPER",
+		"COMPUTER_USE_LINUX_AGENT_NAME",
+		"COMPUTER_USE_LINUX_INDICATOR",
+		"COMPUTER_USE_LINUX_INDICATOR_BIN",
+		"COMPUTER_USE_LINUX_INDICATOR_HIDE_TEXT",
 		"COMPUTER_USE_LINUX_ENABLE_SHELL",
 		"COMPUTER_USE_LINUX_FORCE_PORTAL_KEYBOARD",
 		"COMPUTER_USE_LINUX_FORCE_PORTAL_POINTER",
@@ -277,6 +281,17 @@ function defaultFindBinary(): BinaryLaunch | null {
 		if (!env.COMPUTER_USE_LINUX_COSMIC_HELPER && executable(cosmicHelper)) {
 			env.COMPUTER_USE_LINUX_COSMIC_HELPER = cosmicHelper;
 		}
+		const indicator = join(
+			__dirname,
+			"..",
+			"..",
+			"npm",
+			"bin",
+			"computer-use-linux-indicator",
+		);
+		if (!env.COMPUTER_USE_LINUX_INDICATOR_BIN && executable(indicator)) {
+			env.COMPUTER_USE_LINUX_INDICATOR_BIN = indicator;
+		}
 		return { binaryPath: bundledBinary, env };
 	}