@agent-sh/computer-use-linux 0.3.1 → 0.4.0

package/README.md

@@ -36,11 +36,13 @@ The crate was extracted from [`codex-desktop-linux`](https://github.com/avifenes
 MCP tools exposed by the server:
 
 **Diagnostics**
+
 - `doctor` — single-shot JSON readiness report (platform, portals, accessibility, windowing, input, readiness summary, and a capability map of available backends)
 - `setup_accessibility` — enables GNOME's `org.gnome.desktop.interface toolkit-accessibility` setting so toolkit apps expose AT-SPI trees
 - `setup_window_targeting` — installs and enables the bundled GNOME Shell extension when `org.gnome.Shell.Introspect` is locked down
 
 **Discovery**
+
 - `list_apps` — running desktop apps visible to the AT-SPI registry
 - `list_windows` — compositor windows with title, app id, wm_class, focus state, client type (Wayland/X11), and bounds
 - `focused_window` — the window currently holding keyboard focus
@@ -50,6 +52,7 @@ MCP tools exposed by the server:
 Screenshot payloads are size-bounded by default before they are returned to the MCP host: max 1920 px width/height and 2 MiB image bytes, with hard caps even when callers request more. Agents that need more detail can pass `max_width`, `max_height`, `max_bytes`, `scale`, `format: "jpeg"`, or `quality`, preferably with a window target or crop. PNG remains the default; JPEG lets callers trade lossless pixels for a smaller payload before the byte cap forces further resizing. Returned screenshot metadata includes `coordinate_width`, `coordinate_height`, `scale`, `format`, and `quality` so callers can convert from a downscaled preview to desktop coordinate pixels.
 
 **Input**
+
 - `click` — by element index, semantic selector, or desktop coordinate pixels
 - `drag` — desktop coordinate drag (start / end)
 - `scroll` — page-based scroll on an element or at a pixel location
@@ -59,10 +62,12 @@ Screenshot payloads are size-bounded by default before they are returned to the
 Targeted `press_key`/`type_text` results append focused-element feedback from AT-SPI (role, name, editable) and warn when no editable element holds focus. Click/screenshot/input results warn when the target window or coordinate is partially or fully off-screen. `get_app_state` returns a compact readiness block by default; pass `verbose: true` for the full diagnostics report.
 
 **Semantic actions**
+
 - `perform_action` — invoke any AT-SPI action exposed by an element (`Press`, `Activate`, `Toggle`, …); defaults to the primary action
 - `set_value` — write to a settable accessibility element (text fields, sliders, spinners)
 
 **Navigation**
+
 - `activate_window` — focus a window by `window_id`, `pid`, `app_id`, `wm_class`, `title`, or terminal selectors
 - `move_window` / `resize_window` — reposition or resize a window in desktop coordinates (GNOME Shell extension backend); useful to recover windows that are partially off-screen
 
@@ -223,6 +228,24 @@ Edit `~/.config/Claude/claude_desktop_config.json`:
 
 Restart Claude Desktop. The tools should appear in the tools list.
 
+### Pi Coding Agent
+
+```bash
+pi install npm:pi-mcp-adapter
+pi install npm:@agent-sh/computer-use-linux
+```
+
+Restart pi or run `/reload`. The MCP proxy tool `mcp()` will have the desktop tools available:
+
+```
+mcp({ server: "computer-use-linux" })             # list all tools
+mcp({ search: "windows" })                         # search for window tools
+mcp({ tool: "computer_use_linux_doctor" })         # run readiness check
+mcp({ tool: "computer_use_linux_list_windows" })   # list desktop windows
+```
+
+The extension auto-registers the computer-use-linux MCP server into pi-mcp-adapter's config. If the binary is not found, check the [Pi setup guide](skills/computer-use-linux/references/pi-setup.md).
+
 ### Hermes Agent
 
 Install the companion Hermes skill so Hermes has the desktop-specific runbook:

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@agent-sh/computer-use-linux",
-  "version": "0.3.1",
+  "version": "0.4.0",
   "description": "Linux desktop-control MCP server: AT-SPI accessibility trees, Wayland/X11 input, screenshots, and compositor window targeting.",
   "license": "MIT",
   "type": "commonjs",
@@ -19,7 +19,8 @@
     "wayland",
     "accessibility",
     "hermes-agent",
-    "codex"
+    "codex",
+    "pi-package"
   ],
   "os": [
     "linux"
@@ -35,10 +36,16 @@
     "LICENSE",
     "README.md",
     "skills/computer-use-linux/SKILL.md",
+    "skills/computer-use-linux/references/",
     "npm/README.md",
     "npm/bin/computer-use-linux.js",
-    "npm/install.js"
+    "npm/install.js",
+    "pi/"
   ],
+  "pi": {
+    "extensions": ["./pi/extension/index.ts"],
+    "skills": ["./skills/computer-use-linux/SKILL.md"]
+  },
   "scripts": {
     "postinstall": "node npm/install.js",
     "pack:check": "npm pack --dry-run",

package/pi/extension/index.ts

@@ -0,0 +1,262 @@
+/**
+ * Pi coding agent extension for computer-use-linux.
+ *
+ * Discovers the computer-use-linux binary and registers it as an MCP server
+ * for pi-mcp-adapter by writing to ~/.pi/agent/mcp.json.
+ *
+ * Prerequisite: pi-mcp-adapter (npm:pi-mcp-adapter) must be installed.
+ */
+
+import type { ExtensionAPI } from "@earendil-works/pi-coding-agent";
+import {
+	accessSync,
+	constants,
+	existsSync,
+	mkdirSync,
+	readFileSync,
+	writeFileSync,
+} from "node:fs";
+import { homedir } from "node:os";
+import { dirname, join } from "node:path";
+import { env } from "node:process";
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+const PACKAGE_NAME = "@agent-sh/computer-use-linux";
+const MCP_SERVER_NAME = "computer-use-linux";
+
+/**
+ * Path to pi's agent-level MCP config. pi-mcp-adapter reads from this file
+ * when it starts up, alongside ~/.config/mcp/mcp.json and .mcp.json.
+ */
+function getPiAgentMcpConfigPath(): string {
+	const configured = env.PI_CODING_AGENT_DIR?.trim();
+	if (configured) {
+		return join(configured, "mcp.json");
+	}
+	return join(homedir(), ".pi", "agent", "mcp.json");
+}
+
+// ---------------------------------------------------------------------------
+// Binary discovery
+// ---------------------------------------------------------------------------
+
+/**
+ * Find the computer-use-linux binary. Cached after first successful lookup.
+ */
+let _binaryPath: string | null | undefined;
+
+function findBinary(): string | null {
+	if (_binaryPath !== undefined) return _binaryPath;
+
+	// 1) Environment variable override
+	const fromEnv = env.COMPUTER_USE_LINUX_BIN;
+	if (fromEnv && existsSync(fromEnv)) {
+		_binaryPath = fromEnv;
+		return fromEnv;
+	}
+
+	// 2) Bundled by the npm wrapper's postinstall
+	//    The npm wrapper downloads binaries to npm/bin/ next to the JS wrapper.
+	//    We derive the path relative to our own installed location.
+	const candidates = [
+		// Bundled binary for this platform
+		join(
+			__dirname,
+			"..",
+			"..",
+			"npm",
+			"bin",
+			`computer-use-linux-linux-${process.arch}`,
+		),
+		// Direct npm global install
+		join(__dirname, "..", "..", "npm", "bin", "computer-use-linux"),
+	];
+	for (const candidate of candidates) {
+		const resolved = join(candidate);
+		if (existsSync(resolved)) {
+			_binaryPath = resolved;
+			return resolved;
+		}
+	}
+
+	// 3) PATH
+	const pathDirs = (env.PATH || "").split(":");
+	for (const dir of pathDirs) {
+		const candidate = join(dir, "computer-use-linux");
+		try {
+			if (existsSync(candidate)) {
+				accessSync(candidate, constants.X_OK);
+				_binaryPath = candidate;
+				return candidate;
+			}
+		} catch {
+			// Skip inaccessible or non-executable entries
+		}
+	}
+
+	_binaryPath = null;
+	return null;
+}
+
+// ---------------------------------------------------------------------------
+// MCP config management
+// ---------------------------------------------------------------------------
+
+interface McpServerEntry {
+	command: string;
+	args: string[];
+	lifecycle?: string;
+}
+
+interface McpConfig {
+	mcpServers?: Record<string, McpServerEntry>;
+}
+
+function isRecord(value: unknown): value is Record<string, unknown> {
+	return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+
+function readMcpConfig(path: string): McpConfig | null {
+	// Only return empty if the file genuinely doesn't exist.
+	// If the file exists but can't be read or parsed, return null
+	// so the caller knows not to overwrite it.
+	if (!existsSync(path)) return {};
+
+	try {
+		const raw = readFileSync(path, "utf-8");
+		const parsed = JSON.parse(raw);
+		if (!isRecord(parsed)) {
+			console.error(
+				`${PACKAGE_NAME}: MCP config at ${path} contains a ${typeof parsed}` +
+					(Array.isArray(parsed) ? " (array)" : "") +
+					", expected a JSON object. Skipping.",
+			);
+			return null;
+		}
+		return parsed as McpConfig;
+	} catch (error) {
+		console.error(
+			`${PACKAGE_NAME}: failed to read existing MCP config at ${path}:`,
+			error instanceof Error ? error.message : String(error),
+		);
+		return null;
+	}
+}
+
+function writeMcpConfig(path: string, config: McpConfig): void {
+	mkdirSync(dirname(path), { recursive: true });
+	writeFileSync(path, JSON.stringify(config, null, 2) + "\n", "utf-8");
+}
+
+/**
+ * Result of ensuring the computer-use-linux entry in the MCP config.
+ * - "updated":   entry was written or updated successfully
+ * - "unchanged": entry already matches, no write needed
+ * - "failed":    config could not be read or written
+ */
+type EnsureResult = "updated" | "unchanged" | "failed";
+
+function ensureServerEntry(
+	configPath: string,
+	binaryPath: string,
+): EnsureResult {
+	const config = readMcpConfig(configPath);
+
+	// If config exists but can't be read, don't overwrite it
+	if (config === null) return "failed";
+
+	// Validate that mcpServers is a plain object before mutating
+	const servers =
+		config.mcpServers != null && isRecord(config.mcpServers)
+			? (config.mcpServers as Record<string, McpServerEntry>)
+			: {};
+
+	const existing = servers[MCP_SERVER_NAME];
+
+	// Check if entry already exists and matches
+	if (
+		existing &&
+		existing.command === binaryPath &&
+		existing.args?.length === 1 &&
+		existing.args[0] === "mcp"
+	) {
+		return "unchanged"; // No change needed
+	}
+
+	// Add or update the entry
+	servers[MCP_SERVER_NAME] = {
+		command: binaryPath,
+		args: ["mcp"],
+	};
+
+	try {
+		writeMcpConfig(configPath, { ...config, mcpServers: servers });
+		return "updated"; // Config was updated
+	} catch (error) {
+		console.error(
+			`${PACKAGE_NAME}: failed to write MCP config to ${configPath}:`,
+			error instanceof Error ? error.message : String(error),
+		);
+		return "failed"; // Write failed
+	}
+}
+
+// ---------------------------------------------------------------------------
+// Extension entry point
+// ---------------------------------------------------------------------------
+
+export default function (pi: ExtensionAPI) {
+	// Resolve binary path eagerly (before session_start) so it's cached.
+	const binaryPath = findBinary();
+
+	pi.on("session_start", async (_event, ctx) => {
+		try {
+			if (!binaryPath) {
+				ctx.ui.notify?.(
+					`${PACKAGE_NAME}: computer-use-linux binary not found. ` +
+						"Install it with 'npm install -g @agent-sh/computer-use-linux' " +
+						"or set COMPUTER_USE_LINUX_BIN.",
+					"warning",
+				);
+				return;
+			}
+
+			// Write/update the MCP config that pi-mcp-adapter reads
+			const configPath = getPiAgentMcpConfigPath();
+			const result = ensureServerEntry(configPath, binaryPath);
+
+			if (result === "updated" && ctx.hasUI) {
+				ctx.ui.notify?.(
+					`${PACKAGE_NAME}: MCP server configured at ${configPath}. ` +
+						"Run /reload if pi-mcp-adapter is already installed.",
+					"info",
+				);
+			}
+
+			if (result === "failed" && ctx.hasUI) {
+				ctx.ui.notify?.(
+					`${PACKAGE_NAME}: failed to configure MCP server at ${configPath}. ` +
+						"Check the console logs and ensure the file is writable.",
+					"error",
+				);
+			}
+
+			// Check that pi-mcp-adapter is available (its tools are registered)
+			if (!pi.getAllTools().some((t) => t.name === "mcp")) {
+				ctx.ui.notify?.(
+					`${PACKAGE_NAME}: pi-mcp-adapter not detected. ` +
+						"Install it with 'pi install npm:pi-mcp-adapter' then /reload.",
+					"warning",
+				);
+			}
+		} catch (error) {
+			console.error(
+				`${PACKAGE_NAME}: unexpected error in session_start handler:`,
+				error instanceof Error ? error.message : String(error),
+			);
+		}
+	});
+}

package/skills/computer-use-linux/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: computer-use-linux
-description: "Use when Hermes needs Linux desktop observation or control through the computer-use-linux MCP server."
+description: "Linux desktop observation and control via the computer-use-linux MCP server: accessibility trees, screenshots, window targeting, and input synthesis (click, type, scroll). Works with any MCP host."
 author: agent-sh
 license: MIT
 platforms: [linux]
@@ -8,28 +8,29 @@ platforms: [linux]
 
 # computer-use-linux
 
-Use `computer-use-linux` when Hermes needs to observe or operate a local Linux desktop through MCP: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions.
+Use `computer-use-linux` when an agent needs to observe or operate a local Linux desktop through MCP: inspect the accessibility tree, list/focus windows, take screenshots, click, scroll, type, press keys, or invoke AT-SPI actions.
 
 ## When to Use
 
 Use this skill when:
-- The user wants Hermes to control a Linux GUI app.
+
+- The user wants the agent to control a Linux GUI app.
 - You need desktop state from AT-SPI, screenshots, or compositor window metadata.
-- You are configuring the `computer-use-linux` MCP server for Hermes.
+- You are configuring the `computer-use-linux` MCP server for your agent.
 - A desktop action needs target-aware input instead of blind shell commands.
 
 Do not use this for remote browsers, websites, or headless automation when a browser-specific tool is available. Do not assume desktop actions are safe just because the MCP connection works.
 
 ## Install
 
-Preferred install for Hermes users:
+Preferred install:
 
 ```bash
 npm install -g @agent-sh/computer-use-linux
 computer-use-linux doctor | jq .readiness
 ```
 
-Rust users can install the same server from crates.io:
+Rust users can install from crates.io:
 
 ```bash
 cargo install computer-use-linux
@@ -47,42 +48,23 @@ computer-use-linux doctor | jq .readiness
 
 On GNOME Wayland, log out and back in after `setup-window-targeting` if the GNOME Shell extension was newly installed.
 
-## Configure Hermes
+## Configure Your Agent
 
-Add the server with the Hermes MCP CLI:
+The `computer-use-linux` binary is an MCP server. Configure it as a stdio MCP server in your agent of choice:
 
-```bash
-hermes mcp add computer-use-linux --command computer-use-linux --args mcp
-hermes mcp test computer-use-linux
-hermes mcp configure computer-use-linux
+```json
+{
+  "command": "computer-use-linux",
+  "args": ["mcp"]
+}
 ```
 
-`configure` opens Hermes' tool-selection UI for this MCP server.
-
-The generated config should look like this:
-
-```yaml
-mcp_servers:
-  computer-use-linux:
-    command: computer-use-linux
-    args: ["mcp"]
-    timeout: 120
-    connect_timeout: 30
-```
-
-If the binary is not on `PATH`, pass the absolute path to `--command`.
-
-Hermes registers tools using the `mcp_<server>_<tool>` pattern. With this config, tool names are prefixed as `mcp_computer_use_linux_`, for example:
+If the binary is not on `PATH`, use the absolute path (typically `~/.local/bin/computer-use-linux` or the npm global bin directory).
 
-| MCP tool | Hermes tool name |
-| --- | --- |
-| `doctor` | `mcp_computer_use_linux_doctor` |
-| `get_app_state` | `mcp_computer_use_linux_get_app_state` |
-| `list_windows` | `mcp_computer_use_linux_list_windows` |
-| `click` | `mcp_computer_use_linux_click` |
-| `type_text` | `mcp_computer_use_linux_type_text` |
+### Host-specific guides
 
-Restart Hermes after changing MCP config.
+- [Hermes setup](references/hermes-setup.md)
+- [Pi coding agent setup](references/pi-setup.md)
 
 ## Procedure
 
@@ -110,7 +92,6 @@ Run:
 
 ```bash
 computer-use-linux doctor | jq .readiness
-hermes chat --toolsets mcp-computer-use-linux -q "List the current desktop windows."
 ```
 
 Ready output should have:
@@ -121,4 +102,4 @@ Ready output should have:
 - `can_send_development_input: true`
 - `blockers: []`
 
-If Hermes does not expose the tools, check startup logs for MCP discovery errors and confirm the server name in `config.yaml` is exactly `computer-use-linux`.
+Then test with your agent by calling the `doctor` tool or asking the agent to list desktop windows.

package/skills/computer-use-linux/references/hermes-setup.md

@@ -0,0 +1,41 @@
+---
+name: hermes-setup
+description: "Hermes agent setup for the computer-use-linux MCP server."
+---
+
+# Hermes Setup
+
+Add the server with the Hermes MCP CLI:
+
+```bash
+hermes mcp add computer-use-linux --command computer-use-linux --args mcp
+hermes mcp test computer-use-linux
+hermes mcp configure computer-use-linux
+```
+
+`configure` opens Hermes' tool-selection UI for this MCP server.
+
+The generated config should look like this:
+
+```yaml
+mcp_servers:
+  computer-use-linux:
+    command: computer-use-linux
+    args: ["mcp"]
+    timeout: 120
+    connect_timeout: 30
+```
+
+If the binary is not on `PATH`, pass the absolute path to `--command`.
+
+Hermes registers tools using the `mcp_<server>_<tool>` pattern. With this config, tool names are prefixed as `mcp_computer_use_linux_`, for example:
+
+| MCP tool | Hermes tool name |
+| --- | --- |
+| `doctor` | `mcp_computer_use_linux_doctor` |
+| `get_app_state` | `mcp_computer_use_linux_get_app_state` |
+| `list_windows` | `mcp_computer_use_linux_list_windows` |
+| `click` | `mcp_computer_use_linux_click` |
+| `type_text` | `mcp_computer_use_linux_type_text` |
+
+Restart Hermes after changing MCP config.

package/skills/computer-use-linux/references/pi-setup.md

@@ -0,0 +1,57 @@
+---
+name: pi-setup
+description: "Pi coding agent setup for the computer-use-linux MCP server."
+---
+
+# Pi Setup
+
+## Prerequisites
+
+You need both of these installed:
+
+```bash
+pi install npm:pi-mcp-adapter
+pi install npm:@agent-sh/computer-use-linux
+```
+
+> **Note:** `pi-mcp-adapter` is a separate extension that provides MCP protocol support in pi. The `@agent-sh/computer-use-linux` package auto-registers the desktop MCP server into pi-mcp-adapter's config.
+
+After installing both, restart pi or run `/reload`.
+
+## Verify
+
+The `mcp()` proxy tool should now be available. Call it from any prompt:
+
+```bash
+mcp({ search: "doctor" })
+```
+
+Or start with the readiness check (note the server-prefixed tool name):
+
+```bash
+mcp({ tool: "computer_use_linux_doctor" })
+```
+
+Search for available tools:
+
+```bash
+mcp({ server: "computer-use-linux" })
+```
+
+## If the binary is not found
+
+The extension looks for `computer-use-linux` in this order:
+
+1. `COMPUTER_USE_LINUX_BIN` environment variable
+2. The npm-bundled binary (from `npm install -g @agent-sh/computer-use-linux`)
+3. `$PATH`
+
+If none are found, install it:
+
+```bash
+npm install -g @agent-sh/computer-use-linux
+# or
+cargo install computer-use-linux
+```
+
+Then restart pi or run `/reload`.