@vortex-os/computer-use 0.1.0

package/README.md

@@ -0,0 +1,74 @@
+# @vortex-os/computer-use
+
+Read-only **screen perception** for VortEX agents, exposed as an MCP server. It lets an agent *see* what is on screen — read a window's structure, capture a region as an image, and watch for on-screen changes — without ever moving the mouse or typing. It layers on `@vortex-os/base` but also works standalone.
+
+> **Status: 0.1.0, Windows-first, read-only.** Mouse/keyboard **control is intentionally out of scope** for this release — this package only *perceives*. macOS/Linux backends are not yet implemented.
+
+## What it is
+
+An MCP (Model Context Protocol) server that exposes six perception tools over stdio:
+
+| Tool | What it does | Cost |
+|---|---|---|
+| `probe` | Reports whether this environment can perceive the screen (displays, DPI, capture latency). Never captures real screen content. | ~0 |
+| `read_ui` | Reads the active/target window as a **structured accessibility tree** (UI Automation): element roles, coordinates, text. No image. | ~0 image tokens |
+| `capture_screen` | Pixel capture (PNG) for what structure can't reach — canvases, games, remote desktops. Target by window, region, monitor, or cursor box. | image |
+| `watch_capture` | Captures N frames at an interval in one process; with `changeOnly`, keeps only changed frames. | image(s) |
+| `poll_change` | One non-blocking "did it change?" probe; returns a change percentage and (optionally) an image. Poll it on an interval to watch without blocking. | metadata, image optional |
+| `beep` | A system beep, to get the user's attention while they look elsewhere. | — |
+
+The design favors **structure first, pixels as fallback**: `read_ui` is cheap and precise for ordinary apps; `capture_screen` is for content that has no accessibility tree (games, custom canvases).
+
+## What it is NOT
+
+- **Not control.** No clicking, typing, or app automation. Perception only.
+- **Not comprehensive secret protection.** See *Privacy & redaction* below — the denylist is the real control; field-level masking is best-effort and does not catch plaintext secrets sitting in arbitrary windows.
+- **Not cross-platform yet.** Windows only in 0.1.0.
+
+## Install
+
+```
+npm i @vortex-os/computer-use
+```
+
+Peer dependency: `@vortex-os/base` (`>=0.3.0 <1.0.0`). The MCP SDK (`@modelcontextprotocol/sdk`) is an optional dependency, loaded only when the server runs. No native build step.
+
+### Register the MCP server
+
+The package ships a `vortex-mcp-computer-use` bin that launches the stdio server. Register it with your agent host. For Claude Code, add it to `.mcp.json`:
+
+```json
+{
+  "mcpServers": {
+    "vortex-computer-use": {
+      "command": "npx",
+      "args": ["vortex-mcp-computer-use"]
+    }
+  }
+}
+```
+
+> Use a server name other than the reserved `computer-use` (e.g. `vortex-computer-use`) — some hosts reserve `computer-use` and will silently skip a server with that exact name. MCP servers load at session start, so **restart the agent** after adding it.
+
+## Privacy & redaction
+
+Whatever you point this at is sent to your AI model. Two controls reduce accidental exposure; both run in the backend **before** any pixels or text reach the model:
+
+1. **Denylist (the primary control).** List window titles or process names that must never be captured. If a listed window is visible anywhere inside a capture region, the whole capture is refused (`{ "redacted": true }` — no image, no text). This is the reliable defense against accidentally capturing a password manager or banking window during a watch.
+2. **Password-field masking.** In `read_ui`, fields the OS reports as password inputs are dropped (no value, no text, children not traversed).
+
+Copy `computer-use.config.example.json` to `computer-use.config.json` (next to the server) to configure the denylist, or set `VORTEX_CU_DENY_TITLES` / `VORTEX_CU_DENY_PROCS` (JSON arrays). **The denylist is read once at startup — restart the server after changing it.**
+
+**Honest limits.** This is *not* comprehensive secret-scanning. A plaintext token shown in a text editor or terminal (not a password field, not a denylisted window) will still be captured. Pixel-level password masking is intentionally out of scope for 0.1.0. Capture images are volatile — held only long enough to send, then deleted; they are never written to disk persistently.
+
+### Audit
+
+Each perception call appends one metadata line (timestamp, tool, output size, a keyed HMAC of the output, and an HMAC of the window title) to a daily JSONL log under your user-local app data (`%LOCALAPPDATA%\vortex-computer-use\audit\`) — outside the synced instance data. **No raw images and no plaintext window titles are stored.** If the audit key can't be set up, perception still works and a warning is printed.
+
+## Verify
+
+```
+npm run verify   # node scripts/verify.mjs — needs a desktop session; captures the real screen
+```
+
+Exercises every tool plus the redaction/audit gate (denylist blocking across all capture modes, no over-block, no title leak, audit written with no plaintext).

package/computer-use.config.example.json

@@ -0,0 +1,12 @@
+{
+  "_comment": "Copy to computer-use.config.json to enable redaction. Empty by default — nothing is blocked until you add entries. The denylist is the primary control: any listed window/process that appears inside a capture region makes the whole capture fail-closed (no image, no structured text). Matching is case-insensitive substring. This is NOT comprehensive secret-scanning: plaintext secrets visible in non-listed windows (editors, terminals) are still captured. Env overrides: VORTEX_CU_DENY_TITLES / VORTEX_CU_DENY_PROCS (JSON arrays).",
+  "_restart": "The denylist is read once at server start; RESTART the MCP server (restart the agent / reload its MCP servers) after changing this file or the env vars for the change to take effect.",
+  "redaction": {
+    "denyWindowTitles": [],
+    "denyProcesses": []
+  },
+  "_examples": {
+    "denyWindowTitles": ["Bitwarden", "1Password", "KeePass", "Online Banking"],
+    "denyProcesses": ["Bitwarden", "1Password", "KeePassXC", "keeper"]
+  }
+}

package/package.json

@@ -0,0 +1,50 @@
+{
+  "name": "@vortex-os/computer-use",
+  "version": "0.1.0",
+  "description": "Add-on — read-only screen perception (structured UIA tree + pixel fallback + change watch) exposed as an MCP server, layered on @vortex-os/base. Windows-first. Control (mouse/keyboard) is intentionally out of scope.",
+  "license": "MIT",
+  "author": "vortex-os-project",
+  "homepage": "https://github.com/vortex-os-project/vortex#readme",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/vortex-os-project/vortex.git",
+    "directory": "modules/computer-use"
+  },
+  "type": "module",
+  "files": [
+    "scripts/mcp-stdio.mjs",
+    "scripts/worker.ps1",
+    "scripts/lib.ps1",
+    "scripts/probe.ps1",
+    "scripts/read-ui.ps1",
+    "scripts/point-to-ask.ps1",
+    "computer-use.config.example.json",
+    "README.md"
+  ],
+  "bin": {
+    "vortex-mcp-computer-use": "scripts/mcp-stdio.mjs"
+  },
+  "scripts": {
+    "verify": "node scripts/verify.mjs"
+  },
+  "peerDependencies": {
+    "@vortex-os/base": ">=0.3.0 <1.0.0"
+  },
+  "peerDependenciesMeta": {
+    "@vortex-os/base": {
+      "optional": false
+    }
+  },
+  "optionalDependencies": {
+    "@modelcontextprotocol/sdk": "^1.21.0"
+  },
+  "engines": {
+    "node": ">=22"
+  },
+  "os": [
+    "win32"
+  ],
+  "publishConfig": {
+    "access": "public"
+  }
+}