better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/features/computer-use.md

@@ -0,0 +1,636 @@
+---
+title: Computer Use
+sidebar_position: 16
+---
+
+# Computer Use
+
+Hermes Agent can drive your desktop — clicking, typing, scrolling,
+dragging — in the **background** on **macOS, Windows, and Linux**. Your
+cursor doesn't move, keyboard focus doesn't change, and your virtual
+desktops / Spaces don't switch on you. You and the agent co-work on the
+same machine.
+
+Unlike most computer-use integrations, this works with **any tool-capable
+model** — Claude, GPT, Gemini, or an open model on a local
+OpenAI-compatible endpoint. There's no Anthropic-native schema to worry
+about.
+
+## How it works
+
+The built-in `computer_use` toolset is the recommended Hermes integration. It
+speaks MCP over stdio to
+[`cua-driver`](https://github.com/trycua/cua), an open-source background
+computer-use driver. Each platform uses the appropriate accessibility +
+input stack under the hood:
+
+| Platform | Accessibility tree | Input dispatch |
+|---|---|---|
+| macOS | AX (private SkyLight SPIs) | `SLPSPostEventRecordTo` — pid-scoped, no cursor warp |
+| Windows | UIAutomation | `SendInput` + `PostMessage` — no focus steal |
+| Linux | AT-SPI (X11 + Wayland) | XTest (X11) / virtual-keyboard (Wayland) |
+
+The result is the same on every platform: the agent can read the
+accessibility tree of any visible window AND post synthesized events
+without bringing it to front, switching virtual desktops, or moving the
+real OS cursor.
+
+For the underlying contract — *why* background mode matters, the
+no-foreground invariant, click-dispatch internals — see
+**[cua.ai/docs/explanation/the-no-foreground-contract](https://cua.ai/docs/explanation/the-no-foreground-contract)**.
+
+## Enabling
+
+**Fresh installs already have the driver.** The Hermes installer
+(`install.sh` / `install.ps1`) pre-installs `cua-driver` (best-effort;
+pass `--skip-computer-use` / `-SkipComputerUse` to opt out), so enabling
+Computer Use is just a config flip:
+
+- **`hermes tools`** → pick `🖱️  Computer Use` — installs the driver
+  automatically if it's still missing.
+- **Dashboard / desktop app** → toggle the Computer Use toolset — if the
+  driver is missing, the toggle kicks off the install in the background
+  automatically (watch progress in the toolset panel).
+
+**Manual fallback (older installs, skipped installer step):**
+
+```
+hermes computer-use install
+```
+
+This fetches and runs the upstream cua-driver installer — `install.sh`
+on macOS/Linux, `install.ps1` on Windows. Use `hermes computer-use
+status` to verify the install.
+
+Already have cua-driver? Hermes reuses it when it supports the 0.20 runtime
+contract. During setup, toolset enablement, `hermes update`, and the first
+`computer_use` call of a session, Hermes checks the local version and
+manifest. It repairs an old or incomplete standard installation through
+the upstream installer (at most once per session at runtime). A binary
+selected with `HERMES_CUA_DRIVER_CMD` stays
+under your control, so Hermes reports the incompatibility and leaves it
+unchanged.
+
+If you install Cua Driver first, `cua-driver skills install` installs Cua's
+skill pack under `~/.cua-driver/skills/cua-driver`. Hermes autodetection is a
+planned cua-driver follow-up, so currently point Hermes at that directory or
+symlink it into your skill space. You can also register raw Cua MCP tools as a
+custom MCP server, but that is an alternative for users who need the low-level
+interface. The built-in toolset provides Hermes actions, configuration,
+approvals, and diagnostics.
+
+After installing, regardless of which path you took, grant the
+platform-appropriate prereqs:
+
+| Platform | Prereqs |
+|---|---|
+| **macOS** | System Settings → Privacy & Security → **Accessibility** + **Screen Recording**. Grant the identity named by `hermes computer-use doctor`. Standard mode uses CuaDriver.app; bounded and unrestricted modes use the Hermes host identity. |
+| **Windows** | None at install time. If you're driving over SSH (not RDP / console), you need the autostart pattern — see [cua.ai/docs/how-to-guides/driver/windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh) for the Session 0 ↔ Session 1+ proxy. |
+| **Linux** | A reachable display server: `DISPLAY` set for X11, or `XDG_SESSION_TYPE=wayland`. Wayland sessions need an XWayland bridge for capture. AT-SPI must be on (default on GNOME/KDE/Xfce). |
+
+Then start a session with the toolset enabled:
+
+```
+hermes -t computer_use chat
+```
+
+or add `computer_use` to your enabled toolsets in `~/.hermes/config.yaml`.
+
+## Permission modes and logged-in browser profiles
+
+Hermes maps its existing approval UX onto cua-driver's immutable runtime
+modes. Permission mode and capability manifest approval are launch settings.
+They cannot change after the runtime starts:
+
+| Hermes session | cua-driver mode | Human intervention |
+|---|---|---|
+| Manual or smart approvals (default) | `standard` | Normal Hermes approvals; Cua stops at its protected boundary |
+| `computer_use.permission_mode: bounded` + reviewed manifest | private `bounded` daemon | You review and approve the capability manifest once, at launch |
+| `--yolo`, `/yolo`, or `approvals.mode: off` | private `unrestricted` daemon | One explicit Hermes risk acceptance; no runtime Cua prompts |
+
+Browser work — including pages in a signed-in profile — goes through the
+`browser` toolset (`browser_exec`), not `computer_use`. The former
+`computer_use.grant_existing_profile` opt-in was removed along with the typed
+browser route; a leftover key in config.yaml is ignored.
+
+### Bounded mode for repeatable automation
+
+For recurring browser automation (cron jobs, scheduled research against an
+authenticated app), `bounded` mode uses a capability manifest you review once:
+
+```yaml
+# config.yaml
+computer_use:
+  permission_mode: bounded
+  capability_manifest: ~/.hermes/cua-manifest.yaml
+```
+
+The manifest names the apps, browser profile kinds, allowed origins, and
+typed tools the session may use (see the
+[cua-driver permission modes reference](https://cua.ai/docs/reference/cua-driver/permission-modes)
+for the format). Hermes launches a private runtime with
+`--capability-manifest ... --approve-capability-manifest`; anything outside
+the manifest fails closed inside cua-driver. A missing or unreadable manifest
+fails loudly at session start rather than silently downgrading. Session YOLO
+still overrides bounded for that one session.
+
+On macOS, private-session daemons launch through the installed
+`CuaDriver.app` bundle (so permission grants attribute to the driver's own
+identity instead of resetting with every Hermes build), and Hermes verifies
+the bundle's code signature — exact `com.trycua.driver` identifier and the
+official signing team — before launching it. If you build cua-driver from
+source (unsigned), opt in explicitly:
+
+```yaml
+# config.yaml
+computer_use:
+  allow_unsigned_driver: true   # local driver development only
+```
+
+Each MCP transport owns a private lifecycle session inside its runtime. A
+public session name is only a label for cursor identity and session-scoped
+state. It does not select, share, or keep a runtime alive. Turning `/yolo` off,
+resetting or closing the Hermes session, cancellation cleanup, or process exit
+closes that transport session. Hermes also stops private runtimes that it
+launched for bounded or unrestricted access. One Hermes
+conversation cannot change another runtime's mode or grants. Bounded and
+unrestricted modes use a private
+embedded service under the Hermes host identity.
+
+`smart` approval remains `standard`: an LLM classification cannot stand in for
+a reviewed manifest.
+
+<div class="alert alert--warning">
+
+YOLO/unrestricted mode does not protect against prompt injection or unintended
+input. Use it only in a disposable VM or with accounts and data whose full
+compromise you accept.
+
+</div>
+
+## `hermes computer-use doctor` — your first triage stop
+
+`hermes computer-use doctor` runs cua-driver's structured
+`health_report` MCP tool and prints a per-check matrix. It's the single
+fastest way to find out *why* an action isn't working.
+
+```
+$ hermes computer-use doctor
+⚠️  cua-driver VERSION on darwin: degraded
+  ✅ binary_version: cua-driver VERSION
+  ✅ platform_supported: macOS 26.4.1 (arm64)
+  ✅ session_active: MCP session is active.
+  ❌ bundle_identity: Process has no CFBundleIdentifier.
+      → Run the binary inside CuaDriver.app so TCC grants attribute correctly.
+  ✅ tcc_accessibility: Accessibility is granted.
+  ✅ tcc_screen_recording: Screen Recording is granted.
+  ✅ ax_capability: AX is trusted and reachable.
+  ✅ screen_capture_capability: ScreenCaptureKit reachable; 1 display(s) shareable.
+```
+
+- **Exit code 0** when overall is `ok` — everything's wired up.
+- **Exit code 1** when `degraded` or `failed` — at least one check failed; the hint on each failure tells you what to fix.
+- **Exit code 2** when the cua-driver binary itself isn't reachable.
+
+Useful flags:
+
+- `--include CHECK` — run only the listed checks (repeat for multiple)
+- `--skip CHECK` — skip a check (wins over `--include`)
+- `--json` — emit the raw structured payload, same shape as the
+  `tools/call health_report` MCP response
+
+The check matrix is platform-aware: `bundle_identity` / `tcc_*` are
+`skip` on Windows + Linux because those concepts don't apply.
+`ax_capability` checks AX on macOS, UIA on Windows, AT-SPI on Linux —
+each with the right diagnostic hint when it can't reach.
+
+## The agent cursor and sessions
+
+When the agent acts, you'll see a **tinted overlay cursor** glide
+across the screen to where each click / type / scroll lands. The real
+OS cursor never moves. The overlay shows where the agent is acting. Each
+Hermes run declares a public cua-driver **session name** (something like
+`hermes-3a7b9c14d2e8`). The name labels cursor identity and related state, so
+concurrent runs and subagents get distinct cursors. The MCP transport owns the
+private lifecycle session inside the runtime; the public name does not.
+
+The overlay cursor is cosmetic — captures, clicks, and typing all work
+without it. Hermes disables it automatically where it is a known failure
+mode: macOS (idle CPU burn), headless Linux / WSL2 / containers, and
+**Linux X11 desktops** (the overlay is a fullscreen always-on-top window
+that can get stuck over every workspace after an unclean session end,
+wedging desktop input). Linux Wayland and Windows keep the overlay. Set
+`computer_use.no_overlay: false` in `config.yaml` to force the cursor on
+(or `true` to force it off) on any platform.
+
+Tune the cursor with `cua-driver`'s CLI flags or the runtime
+`set_agent_cursor_style` MCP tool — see
+[cua.ai/docs/how-to-guides/driver/personalize-cursor](https://cua.ai/docs/how-to-guides/driver/personalize-cursor)
+for the full menu (built-in `arrow` vs `teardrop` silhouette, custom
+SVG / PNG / ICO via `--cursor-icon`, runtime gradient colors, bloom
+halo).
+
+## Going deeper — the cua-driver skill pack
+
+Hermes keeps its wrapper skill (`skills/autonomous-ai-agents/computer-use/SKILL.md`)
+focused on the Hermes-side `computer_use` workflow and action vocabulary. For
+platform details, recording semantics, browser page interaction, and other
+deep Cua behavior, install the skill pack that the cua-driver team ships and
+maintains directly:
+
+```
+cua-driver skills install
+```
+
+The command links the pack into `~/.hermes/skills/cua-driver` (Hermes is one
+of the agents `cua-driver skills status` reports). The wrapper remains the
+workflow layer: the pack documents the driver's own MCP vocabulary
+(`get_window_state`, `element_token`, `snapshot_id`), which the `computer_use`
+wrapper translates to for you — keep calling `computer_use(action=...)`. The
+pack contains:
+
+| File | Topic |
+|---|---|
+| `SKILL.md` | The cross-platform core (snapshot invariant, no-foreground contract, click dispatch, AX-tree mechanics) |
+| `MACOS.md` | macOS specifics: no-foreground contract, AXMenuBar navigation, SkyLight click dispatch, Apple Events JS bridge |
+| `WINDOWS.md` | Windows specifics: UIA tree, UWP / `ApplicationFrameHost` hosting, Session 0 isolation, autostart pattern |
+| `LINUX.md` | Linux specifics: AT-SPI tree, X11 / Wayland, terminal-emulator detection |
+| `RECORDING.md` | Trajectory + video recording semantics |
+| `WEB_APPS.md` | Browser-page interaction tips |
+| `TESTS.md` | Replay-by-trajectory workflow |
+
+These are **platform deep dives, not duplicates of the Hermes skill** —
+when an agent reports "on Windows, my click landed on the wrong
+element," it reads `WINDOWS.md` for the UIA / UWP context that
+explains why and what to do differently.
+
+`cua-driver skills status` shows what's installed and which agent
+harnesses it's linked into. Today the autodetect list covers Claude
+Code, Codex, OpenCode, OpenClaw, and Antigravity; **Hermes
+autodetection is planned as a follow-up in `trycua/cua`** — until
+then, run `cua-driver skills install` once and point your harness at
+the resulting `~/.cua-driver/skills/cua-driver` directory (or symlink
+it into your usual skill space).
+
+## Quick example
+
+User prompt: *"Find my latest email from Stripe and summarise what they want me to do."*
+
+The agent's plan (this is the same shape on macOS / Windows / Linux —
+the model substitutes the platform's idiomatic shortcut and app name):
+
+1. `computer_use(action="capture", mode="som", app="Mail")` — gets a
+   screenshot of the email app with every sidebar item, toolbar button,
+   and message row numbered.
+2. `computer_use(action="click", element=14)` — clicks the search field.
+3. `computer_use(action="type", text="from:stripe")`
+4. `computer_use(action="key", keys="return", capture_after=True)` —
+   submit and get the new screenshot.
+5. Click the top result, read the body, summarise.
+
+During all of this, your cursor stays wherever you left it and the email
+app never comes to front.
+
+## Receiving the actual screenshot
+
+Screenshots taken during computer control are normally internal — they exist
+so the model can see the screen, and the agent replies in text. But every
+image capture also saves a bounded, shareable copy under Hermes' image cache
+and reports its path, so on attachment-capable surfaces (Telegram, Discord,
+Desktop, and other gateway platforms) you can simply ask:
+
+> *"Send me a screenshot of my screen."*
+
+and the agent delivers the real image as a native attachment, not just a
+description. On the CLI there is no attachment channel, so the agent gives
+you the saved file's path instead.
+
+Only the 20 most recent capture files are kept, and screenshots are never
+sent automatically — only when you ask for one.
+
+### Whole screen vs. desktop surface
+
+"Screenshot my screen" captures **everything currently displayed** — a
+composited grab of all visible windows, like pressing PrtScn. This image has
+no clickable elements, so to *act* on something in it the agent re-captures
+the specific app.
+
+Asking for the **desktop** instead targets the OS shell surface itself —
+wallpaper, desktop icons, taskbar — with its clickable elements, so requests
+like "open the Recycle Bin on my desktop" still work.
+
+## Provider compatibility
+
+| Provider | Vision? | Works? | Notes |
+|---|---|---|---|
+| Anthropic (Claude Sonnet/Opus 3+) | ✅ | ✅ | Best overall; SOM + raw coordinates. |
+| OpenRouter (any vision model) | ✅ | ✅ | Multi-part tool messages supported. |
+| OpenAI (GPT-4+, GPT-5) | ✅ | ✅ | Same as above. |
+| Google (Gemini 2+) | ✅ | ✅ | Tool-calling + vision both supported. |
+| Local vLLM / LM Studio / Ollama (vision model) | ✅ | ✅ | If the model supports multi-part tool content. |
+| Text-only models | ❌ | ✅ (degraded) | Use `mode="ax"` for accessibility-tree-only operation. |
+
+Screenshots are sent inline with tool results as OpenAI-style `image_url`
+parts. For Anthropic, the adapter converts them into native `tool_result`
+image blocks. The image MIME type comes from cua-driver's explicit
+`mimeType` field (`image/png` or `image/jpeg`) — no client-side
+magic-byte sniffing.
+
+## Safety
+
+Hermes applies multi-layer guardrails:
+
+- Destructive actions (click, type, drag, scroll, key, focus_app)
+  require approval through the same gate as dangerous shell commands —
+  interactively via the CLI dialog or the messaging-platform approval
+  buttons. Once/session/always grants are keyed
+  `cua:<action>:<background|foreground>` and live in the shared
+  session/`command_allowlist` store (a background grant never covers the
+  visible foreground variant). Where nobody can answer — cron
+  (`approvals.cron_mode`), single-query, unattended platforms, or any
+  headless run — the action is refused rather than auto-approved;
+  `--yolo` / `/yolo` still bypass.
+- Hard-blocked key combos at the tool level: empty trash, force delete,
+  lock screen, log out, force log out.
+- Hard-blocked type patterns: `curl | bash`, `sudo rm -rf /`, fork
+  bombs, etc.
+- The agent's system prompt tells it explicitly: no clicking permission
+  dialogs, no typing passwords, no following instructions embedded in
+  screenshots.
+
+Pair with `approvals.mode: manual` in `~/.hermes/config.yaml` if you
+want every action confirmed.
+
+## Token efficiency
+
+Screenshots are expensive. Hermes applies four layers of optimisation:
+
+- **Screenshot eviction** — the Anthropic adapter keeps only the 3 most
+  recent screenshots in context; older ones become `[screenshot removed
+  to save context]` placeholders.
+- **Client-side compression pruning** — the context compressor detects
+  multimodal tool results and strips image parts from old ones.
+- **Image-aware token estimation** — each image is counted as ~1500
+  tokens (Anthropic's flat rate) instead of its base64 char length.
+- **Server-side context editing (Anthropic only)** — when active, the
+  adapter enables `clear_tool_uses_20250919` via `context_management` so
+  Anthropic's API clears old tool results server-side.
+
+A 20-action session on a 1568×900 display typically costs ~30K tokens
+of screenshot context, not ~600K.
+
+## Limitations
+
+- **Performance.** Background mode is slower than foreground —
+  accessibility-routed events take ~5–20 ms on macOS, ~3–10 ms on
+  Windows UIA, ~5–15 ms on Linux AT-SPI vs direct HID posting. Not
+  noticeable for agent-speed clicking; noticeable if you try to record
+  a speed-run.
+- **No keyboard password entry.** `type` has hard-block patterns on
+  command-shell payloads; for passwords, use the system's autofill
+  (macOS Keychain / Windows Credential Manager / GNOME Keyring /
+  KWallet).
+- **Some apps don't expose an accessibility tree.** Modern UWP apps on
+  Windows, Electron < 28 on Linux, and a few macOS apps with custom
+  drawing (Logic, Final Cut, some games) have sparse or empty AX trees.
+  Fall back to pixel coordinates if the tree is empty — or skip the
+  task entirely.
+- **Windows: elevated (admin) windows can't be driven from a normal
+  agent.** Windows UIPI (User Interface Privilege Isolation) enforces
+  integrity-level boundaries: a Medium-integrity process (the default
+  Hermes agent) cannot enumerate the UIA tree of, or inject mouse input
+  into, a window owned by a High-integrity (Administrator) process.
+  Symptom: `capture(mode='som')` returns 0 elements and `click(...)`
+  reports success while doing nothing, even though the screenshot
+  renders fine (GDI capture sits below the integrity check). Keyboard
+  events partially bypass UIPI, so Tab / Enter can still navigate an
+  elevated dialog. This is an OS constraint, not a cua-driver bug — it
+  affects every Windows automation stack. To drive elevated windows,
+  run the Hermes agent itself at High integrity (launch from an
+  elevated terminal); otherwise target non-elevated windows.
+- **Windows: `hermes computer-use doctor` fails with "Access is denied"
+  while the tool works.** A cua-driver installed under
+  `C:\Program Files\WindowsApps` cannot be executed by the Hermes venv
+  interpreter (WinError 5 from `CreateProcess`), even though the shell
+  resolves the same binary fine. The doctor now reports this as a
+  diagnosis instead of a traceback. Fix once: reinstall with the upstream
+  installer (lands under your user profile) or set
+  `HERMES_CUA_DRIVER_CMD` to a copy outside `WindowsApps`.
+- **Platform-specific deployment gotchas:**
+  - **macOS** uses private SkyLight SPIs. Apple can change them in any
+    OS update. Hermes warns when the installed cua-driver is older than
+    the version it was tested against.
+  - **Windows** SSH sessions run in **Session 0**, which has no
+    interactive desktop. Drive Hermes from inside the RDP / console
+    session, or set up cua-driver's autostart Scheduled Task —
+    [windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh)
+    has the recipe.
+  - **Linux** requires a reachable display server. Headless servers
+    need Xvfb (`Xvfb :99 -screen 0 1920x1080x24`) before
+    `computer_use` can capture or inject events. Pure Wayland sessions
+    need an XWayland bridge for screen capture (cua-driver's Wayland
+    inject path handles input independently).
+
+For cross-platform GUI automation without the desktop overhead (and
+without TCC / Session 0 / X11 setup), the `browser` toolset uses a
+real headless Chromium and is the right answer for web-only tasks.
+
+## Configuration
+
+Permission mode and manifest (see
+[Permission modes](#permission-modes-and-logged-in-browser-profiles) above):
+
+```yaml
+computer_use:
+  permission_mode: standard        # standard (default) | bounded
+  capability_manifest: ""          # capability manifest path, required for bounded
+```
+
+On Linux, native Wayland support remains an explicit opt-in. Hermes passes the
+opt-in to every cua-driver process, including gateway sessions, only when that
+process also has `WAYLAND_DISPLAY`:
+
+```yaml
+computer_use:
+  native_wayland: true
+```
+
+Restart a running gateway after changing this setting.
+
+Override the driver binary path (tests / CI / local builds):
+
+```
+HERMES_CUA_DRIVER_CMD=/path/to/your/cua-driver
+```
+
+Swap the backend entirely (for testing):
+
+```
+HERMES_COMPUTER_USE_BACKEND=noop   # records calls, no side effects
+```
+
+### Telemetry
+
+cua-driver ships with anonymous usage telemetry (PostHog) enabled by default
+upstream. **Hermes disables it for you** — on every cua-driver invocation
+(the MCP backend, `status`, `doctor`, and install) Hermes sets
+`CUA_DRIVER_RS_TELEMETRY_ENABLED=0` in the driver's environment.
+
+To opt back in (let cua-driver use its own default and send telemetry), set
+this in `config.yaml`:
+
+```yaml
+computer_use:
+  cua_telemetry: true   # default: false (telemetry off)
+```
+
+When it's on, `hermes computer-use doctor` reports `telemetry: enabled`;
+when off (the default), it reports `telemetry: disabled via
+CUA_DRIVER_RS_TELEMETRY_ENABLED`.
+
+## Testing against a local cua-driver build
+
+When you're developing cua-driver itself — or want to test an
+unreleased fix — point Hermes at a binary you built from source instead
+of the published release. Hermes resolves the driver with
+`shutil.which("cua-driver")` and **does not enforce
+`HERMES_CUA_DRIVER_VERSION`**, so a local build (reported as
+`0.0.0-local-*`) is accepted as-is. Two approaches:
+
+### Option A — `install-local` (build + put it on PATH)
+
+From your `trycua/cua` checkout, run the upstream local installer. It
+builds the Rust backend in release mode and drops `cua-driver` into the
+same install layout the production installer uses, adding its bin dir
+to your PATH:
+
+```powershell
+# Windows (PowerShell), from the cua repo root
+./libs/cua-driver/scripts/install-local.ps1 -NoAutoStart
+```
+
+```bash
+# macOS / Linux, from the cua repo root  (defaults to a debug build without --release)
+./libs/cua-driver/scripts/install-local.sh --release
+```
+
+- Windows stages the build under `%USERPROFILE%\.cua-driver\packages\…`
+  and junctions
+  `%LOCALAPPDATA%\Programs\Cua\cua-driver\bin` (added to your User
+  PATH) to it. macOS/Linux symlinks `cua-driver` into `~/.local/bin`
+  (override with `--bin-dir <path>`).
+- `-NoAutoStart` skips registering the `cua-driver-serve` logon daemon
+  — you don't need it for Hermes testing (see notes).
+
+Then open a fresh shell (so the PATH change is visible) and confirm:
+
+```
+cua-driver --version                 # local builds report 0.0.0-local-release
+# Windows:      (Get-Command cua-driver).Source
+# macOS/Linux:  which cua-driver
+```
+
+### Option B — point Hermes straight at the built binary (fastest loop)
+
+Skip the install ceremony entirely: `cargo build` and set
+`HERMES_CUA_DRIVER_CMD` to the resulting binary. Best for rapid
+edit/build/test.
+
+```bash
+cargo build -p cua-driver            # add --release for a release build; run from libs/cua-driver/rust
+```
+
+```
+# Windows (.env)
+HERMES_CUA_DRIVER_CMD=C:\path\to\cua\libs\cua-driver\rust\target\debug\cua-driver.exe
+# macOS / Linux (.env)
+HERMES_CUA_DRIVER_CMD=/path/to/cua/libs/cua-driver/rust/target/debug/cua-driver
+```
+
+### Confirm Hermes is using your build
+
+- `hermes computer-use status` prints the resolved binary path and
+  version.
+- `hermes computer-use doctor` confirms the binary is reachable and
+  exercises the full MCP path end-to-end.
+- In a session, `computer_use(action="capture")` exercises the spawned
+  `cua-driver mcp` child process.
+
+### Notes & gotchas
+
+- **Hermes spawns a `cua-driver mcp` stdio proxy.** In a normal session the
+  proxy connects to (and may start) the standard machine daemon. In explicit
+  Hermes YOLO, Hermes instead owns a private `cua-driver serve --embedded`
+  child and points the proxy at its private socket or named pipe. The Windows
+  autostart/UIAccess pattern still matters for interactive Session 1+ input
+  from SSH — see the Limitations section.
+- **Locked binary on Windows.** A running `cua-driver-serve` daemon can
+  hold `cua-driver.exe` and block an overwrite on rebuild.
+  `install-local.ps1` renames the locked binary out of the way
+  automatically; if you `cargo build` manually (Option B), stop it
+  first with `cua-driver autostart disable` (or `schtasks /End /TN
+  cua-driver-serve`).
+- **Rebuild loop.** After editing cua-driver source, re-run
+  `install-local` (rebuilds, restages, flips the `current` junction)
+  for Option A, or just re-`cargo build` for Option B — no Hermes
+  change needed either way.
+- **Local builds skip the version check.** Hermes warns when the
+  installed cua-driver is older than its per-OS tested baseline, but
+  exempts `0.0.0-local-*` dev builds — so your local build never
+  triggers that warning.
+
+## Troubleshooting
+
+**First action when anything's off: run `hermes computer-use doctor`.**
+The structured per-check matrix tells you (and any agent helping you
+debug) exactly what's wrong.
+
+Specific failure modes the doctor doesn't catch:
+
+**`computer_use backend unavailable: cua-driver is not installed`** —
+Run `hermes computer-use install` to fetch the cua-driver binary, or
+run `hermes tools` and enable the Computer Use toolset.
+
+**Clicks seem to have no effect** — Capture and verify. A modal you
+didn't see may be blocking input. Dismiss it with `escape` or the close
+button.
+
+**Element indices are stale** — SOM indices are only valid until the
+next `capture`. Re-capture after any state-changing action. The
+wrapper carries opaque `element_token`s for stale detection — you'll
+see an explicit error rather than a wrong click.
+
+**"blocked pattern in type text"** — The text you tried to `type`
+matches the dangerous-shell-pattern list. Break the command up or
+reconsider.
+
+**Empty captures on Linux** — `DISPLAY` not set, or you're on pure
+Wayland without an XWayland bridge. `hermes computer-use doctor` will
+flag this as `ax_capability: fail` with a `Set DISPLAY (X11)…` hint.
+
+**Empty captures on Windows over SSH** — You're in Session 0 (the
+services session). Drive from RDP / console directly, or set up the
+autostart pattern — see
+[cua.ai/docs/how-to-guides/driver/windows-ssh](https://cua.ai/docs/how-to-guides/driver/windows-ssh).
+
+## See also
+
+- **Hermes-side skill** — `skills/autonomous-ai-agents/computer-use/SKILL.md` — teaches the
+  Hermes `computer_use` action vocabulary; this is what the agent loads.
+- **cua-driver skill pack** — for platform-specific deep dives
+  (macOS no-foreground contract, Windows UIA + Session 0, Linux AT-SPI
+  + X11/Wayland, recording, browser pages), run
+  `cua-driver skills install` and read `MACOS.md` / `WINDOWS.md` /
+  `LINUX.md` / `RECORDING.md` / `WEB_APPS.md`. Hermes autodetection is a
+  planned follow-up; currently point Hermes at the installed pack directory
+  or symlink it into your skill space.
+- **cua.ai/docs** — the cua-driver project's documentation:
+  - [What is computer use?](https://cua.ai/docs/explanation/what-is-computer-use) — concept intro
+  - [The no-foreground contract](https://cua.ai/docs/explanation/the-no-foreground-contract) — *why* background mode matters
+  - [Install reference](https://cua.ai/docs/how-to-guides/driver/install) — cross-platform install details
+  - [Personalize the agent cursor](https://cua.ai/docs/how-to-guides/driver/personalize-cursor) — built-in shapes, custom assets, runtime overrides
+  - [Drive Windows over SSH](https://cua.ai/docs/how-to-guides/driver/windows-ssh) — the Session 0 → Session 1+ autostart pattern
+  - [Keep cua-driver running](https://cua.ai/docs/how-to-guides/driver/keep-running) — autostart / daemon lifecycle
+  - [Connect your agent](https://cua.ai/docs/how-to-guides/driver/connect-your-agent) — register cua-driver with various harnesses (Hermes among them)
+- [cua-driver source (trycua/cua)](https://github.com/trycua/cua)
+- [Browser automation](./browser.md) for cross-platform web tasks where you don't need to drive native apps.