@phnx-labs/computer-cli 0.1.5 → 0.1.8

package/CHANGELOG.md

@@ -1,5 +1,37 @@
 # Changelog
 
+## 0.1.8
+
+- Add `scripts/generate-cli-docs.sh` with `--check` and `--out-dir`, downloadable command references in PR/release-branch CI, and a freshness check that also covers releases with tests/build skipped.
+- Calling `computer` directly records the same session as `agents computer`: it now reads `CLAUDE_SESSION_ID` and `AGENTS_RUN_ID`, prefers `CODEX_THREAD_ID` and `CLAUDE_CODE_SESSION_ID` over `AGENTS_SESSION_ID` / `AGENT_SESSION_ID`, and skips empty values. An inherited context session still wins. Target admissions use the same resolver.
+- `computer run` exits 1 when it stops at `--max-steps` without a completion, in both human and `--json` output; the JSON result is unchanged. `--max-steps` and `--max-tokens` reject values that are not positive integers (such as `0`, `abc` or `1.5`) before any request.
+
+## 0.1.7
+
+- `computer sessions` uses the shared `@phnx-labs/tui` picker, list and stdout writer (PHNX-4227), so its controls and output match standalone `sessions` and `browser sessions`. The picker still lists every run.
+- Ages read `just now`, `3 min ago`, `2 hours ago`, `4 days ago`, then a calendar date (`Sep 28`), in the picker, its preview and the table, instead of `3m ago` / `2h ago` / `5d ago`.
+- The table's footer moves from stdout to stderr and reads `Showing 50 of 120 runs. --limit 120 or --json to see all, --search or --since to narrow.` instead of `… (70 more; --limit 120 …)`, so piped stdout carries rows only. Table columns are as wide as their widest cell, and a long task ends in `…`.
+- Typing a filter in the picker closes the preview pane, as in `sessions`; clear the filter and press space to reopen it.
+- `--json` output is unchanged.
+
+## 0.1.6
+
+- `computer sessions --since <when>` and `--until <when>` filter runs by last activity, with the same values as `sessions` (`2h`, `7d`, `4w`, `1mo`, `1y` or an ISO date). They apply to the picker, whose header now lists the active filters, the table and `--json`. An unreadable value exits 2.
+- `computer sessions --limit <n>` now also caps `--json` when given (`--json` stays unbounded without it), and a value that is not a positive integer exits 2 instead of being ignored.
+- Preserve 365-day computer run summaries independently of Agents and Sessions, with safe adoption of existing history, total action counts and saved screenshot locations.
+- Browse runs and captures with consistent search, previews and Escape navigation; use `sessions --search` for scripts or `--open` to open a saved screenshot. Existing action and live desktop task commands remain available.
+- Piped output is no longer cut off at 64 KiB: `computer sessions --json | …` and every other command now deliver all of their output to a pipe. The picker's prompt library loads only when a picker opens.
+- `computer sessions` opens its picker in a terminal instead of failing with `Cannot find package '@inquirer/prompts'`.
+- The legacy import skips runs older than the 365-day retention, and expired runs are pruned before a listing reads them. A legacy database with no computer table, or one that cannot be read, is checked once rather than on every listing; an unreadable one is reported once, and `computer sessions migrate` retries it.
+- Add a generated command reference: `docs/command-reference.html` (searchable, command tree in a sidebar), `docs/command-index.md` and `docs/command-index.json`, built from the registered command tree by `bun scripts/gen-docs.ts`. `scripts/test.sh` fails when they are stale. `computer task start`, `task list` and `task stop` now have help descriptions.
+- Return compact post-action screenshots in one CLI or MCP tool call, with optional local Tesseract OCR and original-desktop word coordinates. Preserve successful action results when capture fails.
+- Add `computer mcp` image tool results and actual image inputs in the opt-in `computer run` model loop.
+- Add Linux desktop tasks with private capability handles, reconnect, per-desktop operation serialization and owned-resource cleanup. New tasks default to isolated X displays; save `desktopMode` in `~/.agents/.computer/config.json` or override with `--desktop-mode`.
+- Keep persistent VNC input responsive by consuming incremental framebuffer updates, and remove the two-second timer left behind after a closed VNC connection.
+- macOS helper: coordinate clicks, drags and scrolls are gated on the app that owns the window under the point, not only on the named pid, and `screenshot --display` leaves out windows of apps outside the allow list. A coordinate paste restores the previous clipboard.
+- Both helpers: element ids are never reused, so an id from an earlier `describe` returns `element_stale` instead of acting on a different element, and input from concurrent connections is serialized. The Windows helper drops an unauthenticated connection after 10 seconds or a 4 KiB first line.
+- `stop --host` signals the recorded tunnel pid only while it is still that ssh tunnel, and `start --host` notices an ssh that exited during startup. Default screenshot captures are pruned to the newest 200 from the last 7 days, and VNC rejects rectangles outside the framebuffer.
+
 ## 0.1.5
 
 - Saved screenshots carry their actual output path, name and byte count in action history, so session viewers can open the capture. Failed writes and window listings do not claim a saved capture.

package/README.md

@@ -1,34 +1,309 @@
-# Computer CLI
+# computer-cli
 
-Desktop automation through macOS Accessibility, Windows UI Automation and VNC.
+[![macOS: arm64 and x64](https://img.shields.io/badge/macOS-arm64%20%7C%20x64-24292f?logo=apple&logoColor=white)](#platform-support)
+[![Linux: arm64 and x64](https://img.shields.io/badge/Linux-arm64%20%7C%20x64-24292f?logo=linux&logoColor=white)](#platform-support)
+[![Windows: x64](https://img.shields.io/badge/Windows-x64-0078D4?logo=data%3Aimage%2Fsvg%2Bxml%3Bbase64%2CPHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHZpZXdCb3g9IjAgMCAxNiAxNiI%2BPHBhdGggZmlsbD0id2hpdGUiIGQ9Ik0wIDBoN3Y3SDB6bTkgMGg3djdIOXptLTkgOWg3djdIMHptOSAwaDd2N0g5eiIvPjwvc3ZnPg%3D%3D)](#platform-support)
+
+`computer` drives desktop applications from the command line — macOS apps
+through Accessibility, Windows apps through UI Automation over SSH, and any GUI
+desktop over VNC. It reads the accessibility tree, clicks elements by id or
+coordinate, types text, sends key chords, takes screenshots, and can run an
+autonomous task with an embedded model loop. It talks to a local native helper
+daemon on macOS, an SSH tunnel to the Windows driver, or the RFB protocol for
+VNC targets.
+
+## Install
 
 ```sh
 npm install -g @phnx-labs/computer-cli
-computer --help
-computer apps
+computer --version
+```
+
+npm installs a small Node launcher plus one compiled binary for your platform
+(macOS arm64/x64, Linux arm64/x64, or Windows x64). No TypeScript sources or
+source maps ship in the package.
+
+## First run (macOS)
+
+```sh
+computer setup                      # download and verify the signed native helper
+computer start                      # activate the helper daemon via launchd
+computer apps                       # list apps the daemon may drive
+computer screenshot -o desktop.png  # capture the display
+computer describe                   # dump the accessibility tree of the front app
+computer click --id <element>       # click an element from the tree
+computer type --id <field> "hello"  # set a field value
+computer key "cmd+shift+s"          # send a key chord
+computer stop                       # deactivate the daemon
+```
+
+`setup` downloads the signed native helper to `/Applications/`. macOS requires
+the user to grant Accessibility and Screen Recording access; setup cannot do
+that automatically. `start` activates the helper daemon via launchd; `stop`
+deactivates it. `computer status` reports installation, connection and trust
+state.
+
+## Remote targets
+
+`--host` selects a remote desktop using the address grammar shared by every
+Phoenix CLI:
+
+```sh
+# Windows: native UI Automation driver over SSH
+computer screenshot --host ssh://user@winbox
+computer click --host ssh://user@winbox --id <element>
+
+# VNC: any GUI desktop (Linux, macOS, Windows) over RFB
+computer screenshot --host vnc://host:5900
+computer click --host vnc://host:5900 --x 512 --y 384
+computer type-text --host vnc://host:5900 --text "hello"
+computer key --host vnc://host:5900 --keys enter
+```
+
+VNC provides screenshot and coordinate input (`click --x --y`, `type-text`,
+`key`). Accessibility verbs (`describe`, `get-text`, `click --id`, `type --id`)
+require a native driver and are not available over VNC.
+
+`computer run` drives an app from a natural-language task using an embedded model
+loop over the computer verbs.
+
+## Agent tools and isolated Linux desktops
+
+`computer mcp` serves tools over stdio to your existing agent. It does not make
+model API calls. Input tools return a compact screenshot as an actual MCP image
+alongside action metadata. `computer run` remains the opt-in embedded model loop;
+it sends the latest screenshot as an image to its configured provider.
+
+On Linux, each new MCP task gets its own desktop by default. Install `Xvfb`,
+`x11vnc` (with Unix-socket support), `xauth`, and util-linux (`flock`, `setpriv`).
+These displays start empty; launch an application with `launch --path`.
+
+```sh
+computer mcp                           # create an isolated Linux task
+computer task start                    # create a task for CLI use; prints its ID
+computer launch --path /path/to/app --desktop-task TASK_ID
+computer click --x 400 --y 300 --screenshot --desktop-task TASK_ID
+computer mcp --desktop-task TASK_ID     # reconnect another agent to that task
+computer task list
+computer task stop TASK_ID
+```
+
+Save the default in `~/.agents/.computer/config.json`, or override it when creating
+a task. An override never changes the saved setting or an existing task's mode.
+
+```sh
+computer config desktopMode shared
+computer task start --desktop-mode isolated
+computer mcp --desktop-mode shared --host vnc://localhost:5901
+```
+
+One private local service coordinates Linux tasks. Clients sharing a task hold
+its desktop lock through target selection, input and capture. Separate isolated
+desktops run independently; shared-mode tasks are serialized. Direct commands
+without `--desktop-task` do not participate in that coordination. Disconnecting
+one client does not stop another client's task. Idle tasks expire after 15 minutes.
+
+Isolation covers the X display, windows, focus and pointer, not the host user,
+filesystem, application profiles or credentials. Applications that reuse a
+global profile may require their own separate-profile launch configuration.
+Private task handles prevent accidental cross-task access; they are not a
+security boundary against hostile code running as the same OS user. No TCP port
+is opened for an isolated desktop. Native macOS shared-mode MCP uses the existing
+helper; managed task provisioning currently requires Linux.
+
+## Action observations and local OCR
+
+```sh
+computer click --host vnc://localhost:5901 --x 400 --y 300 --screenshot result.png --max-dimension 1280 --json
+computer screenshot --host vnc://localhost:5901 --ocr --out screen.png --json
+```
+
+`--ocr` uses local Tesseract and includes text, word bounds and confidence, without
+a model call. For input commands it implies a post-action capture. OCR coordinates
+refer to the original desktop; screenshot `scale` and `scale_y` map that desktop
+into the resized image. Images preserve aspect ratio and default to a maximum
+edge of 1280 pixels for action/MCP observations. A screenshot is an observation,
+not a guarantee that an application's asynchronous work has finished.
+
+If capture fails after a successful action, the action remains successful and
+the capture error is returned separately. Retry observation, not the action.
+Native compact capture requires a helper supporting `max_dimension`; an older
+helper that ignores the requested bound reports `helper_update_required`.
+
+## How fast are action observations?
+
+In a September 15, 2026 Linux arm64 benchmark, the installed **0.1.5** CLI was
+compared with the **0.1.6 candidate** at `60ca877`. The candidate was not released
+or installed at measurement time. Gedit displayed the real 729-line
+`src/commands/actions.ts`; each sample sent Page Down and captured the result on
+the same owned, isolated 1600 × 1000 desktop.
+
+| Workflow | Median | Observed range | Samples |
+| --- | ---: | ---: | ---: |
+| 0.1.5: separate action and screenshot | 4,494 ms | 4,453–4,552 ms | 8 |
+| Candidate: separate action and screenshot | 515 ms | 454–691 ms | 8 |
+| Candidate: combined action and screenshot | 712 ms | 693–947 ms | 8 |
+| Candidate: MCP image response | 533 ms | 496–578 ms | 8 |
+| Candidate: MCP image + local OCR | 1,205 ms | 1,111–1,364 ms | 8 |
+
+The candidate MCP workflow had **88.1% lower median latency (8.4× speedup)** than the released
+separate-call workflow. Most of that improvement also exists in the candidate's
+separate CLI calls, following the connection-close timer fix; it is not an
+MCP-only speedup. Combined capture was **38% slower** than candidate separate
+calls here, and MCP was not demonstrably faster than candidate separate calls.
+OCR added **672 ms** to the MCP median.
+
+Separate screenshots were full-size; combined/MCP observations used the default
+1280 × 800 image and a 150 ms post-input receive window. MCP PNGs were **21.3%
+smaller** (93,406 versus 118,695 bytes), excluding protocol/base64 overhead.
+These compare default workflows, not identical capture settings.
+
+Each mode had one warmup; measured order rotated between rounds. Timings include
+CLI process start/exit or a tool request on an already-connected MCP client.
+Desktop/app startup, the initial 67 ms MCP connection and harness verification
+are excluded. All 40 captures matched the expected source section using
+case-insensitive OCR anchors; original case-sensitive misses were retained in
+the raw results. The busy host and eight samples per mode make this a directional
+comparison, not a production tail-latency estimate. No model calls, token savings,
+concurrent-task throughput, macOS latency or Windows latency were measured.
+
+See the [benchmark evidence on PR #14](https://github.com/phnx-labs/computer-cli/pull/14#issuecomment-5683895205)
+for the recorded run and delivery limits. Source links require repository access.
+
+## Command reference
+
+```bash
+scripts/generate-cli-docs.sh                              # refresh docs/
+scripts/generate-cli-docs.sh --check                      # fail on stale docs
+scripts/generate-cli-docs.sh --out-dir .agents/scratch/cli-docs  # preview
 ```
 
-The executable is `computer`. Agents CLI consumes it through `agents computer`, adding fleet device discovery and agent-session history.
+Run after `bun install --frozen-lockfile`. The script works from any directory;
+`--out-dir` is relative to your current directory. Open `command-reference.html`
+from the output folder to browse the tree and search commands.
+CI uploads a `computer-cli-docs` artifact on pull requests and pushes to `main`
+or `release/**`. It contains the HTML reference plus Markdown and JSON indexes.
+Tests and releases check the committed files; commit regenerated `docs/` with
+command changes. Release checks also run with `--skip-tests` / `--skip-build`.
+
+[docs/command-reference.html](docs/command-reference.html) lists every command
+with its arguments and options, searchable, with the command tree in a sidebar.
+[docs/command-index.md](docs/command-index.md) is the one-line-per-command index,
+and [docs/command-index.json](docs/command-index.json) the structured form.
+`scripts/generate-cli-docs.sh` regenerates all three from the registered command
+tree; `scripts/test.sh` fails when they are stale.
+
+## Using it through agents-cli
+
+Agents CLI ships `agents computer` as a thin consumer of this binary. It
+forwards every verb unchanged and adds fleet device discovery (`--device` becomes
+a resolved `--host`), agent-session identity, and cross-machine session history.
+The seam between the two is documented in
+[docs/integration.md](docs/integration.md). Transports, addresses and verbs are
+explained in [docs/computer.md](docs/computer.md).
 
-On first use, macOS downloads and verifies the signed native helper. macOS still requires the user to grant Accessibility and Screen Recording access. Setup cannot grant those permissions automatically. An existing helper is reused; its application identity and policy files are preserved. `computer start` also installs a missing helper. Use `computer setup` to explicitly replace it.
+## Platform support
 
-Windows automation uses the native Windows driver in the interactive desktop session, reached through SSH. VNC provides screenshot and coordinate input for desktops without native Accessibility support. See [command reference](https://github.com/phnx-labs/computer-cli/blob/main/docs/computer.md) and [integration contract](https://github.com/phnx-labs/computer-cli/blob/main/docs/integration.md).
+| Platform | CLI architectures | Desktop driver |
+| --- | --- | --- |
+| macOS | arm64, x64 | Native Accessibility helper (local) |
+| Windows | x64 | UI Automation driver over SSH |
+| Linux | arm64, x64 | VNC; isolated task desktops via Xvfb (no native Accessibility) |
 
-Source lives in the private `phnx-labs/computer-cli` repository. npm ships a small Node launcher and an optional compiled package for your platform: macOS arm64/x64, Linux arm64/x64, or Windows x64. TypeScript, source maps and intermediate bundles are excluded. Before compilation, our application code is obfuscated offline with a pinned tool and fixed seed: identifiers, string lookups and selected control flow are transformed. Dependencies are bundled without that transformation to limit runtime overhead.
+These badges describe platform support, not feature parity. Managed isolated
+task provisioning is Linux-only. The measurements above cover Linux/VNC, not
+the native macOS or Windows paths.
 
-Bun executables retain JavaScript alongside bytecode. The transformed code can still be extracted, analyzed and deobfuscated; this build raises inspection effort and does not provide secrecy or an authorization boundary. Bytecode alone is a startup optimization, not source protection. No source is sent to an obfuscation service, and the build adds no debugger interference or runtime network checks. Keep credentials out of binaries.
+### Do macOS and Windows need native helpers?
 
-Native helpers retain their separate, immutable `computer-mac/v*` and `computer-win/v*` release channels. An ordinary CLI release does not rebuild them. This JavaScript hardening does not transform the helpers; the Windows helper is a managed .NET single-file application and remains decompilable.
+Yes, for native element-level automation, and **both helpers already live in
+this repository**, extracted from Agents CLI. Installing Agents CLI is not
+required.
 
-Development:
+- **macOS:** the [Swift helper](native/computer-mac) supplies Accessibility,
+  ScreenCaptureKit capture and CoreGraphics input. Its signed app bundle keeps a
+  stable identity for the user's Accessibility and Screen Recording grants.
+  The CLI downloads/verifies it during setup and manages it through launchd.
+- **Windows:** the [C#/.NET helper](native/computer-win) supplies UI Automation,
+  screen capture and `SendInput`. Setup provisions a scheduled task in the
+  logged-in interactive desktop session. The CLI reaches its loopback service
+  over an authenticated SSH tunnel; an SSH login alone is not a GUI session.
+- **Linux/VNC:** no macOS or Windows helper is needed. A VNC server supplies
+  screenshot and coordinate input; isolated Linux tasks additionally need the
+  dependencies listed [above](#agent-tools-and-isolated-linux-desktops).
+
+Native helpers retain their separate, immutable `computer-mac/v*` and
+`computer-win/v*` release channels. An ordinary CLI release does not rebuild
+them.
+
+## Troubleshooting
+
+- **Accessibility not trusted (macOS).** `computer status` shows whether the
+  helper has Accessibility and Screen Recording access. Grant both in System
+  Settings > Privacy & Security. The helper binary path must match the grant;
+  `computer setup` after a helper upgrade may require re-granting.
+- **VNC password.** Pass `--vnc-password <pw>` or set
+  `COMPUTER_HELPER_VNC_PASSWORD`. A VNC server with no authentication needs
+  neither.
+- **`unsupported_over_vnc`.** Accessibility verbs (`describe`, `get-text`,
+  `click --id`, `type --id`) require a native driver. Over VNC, use coordinate
+  input (`click --x --y`, `type-text --text`, `key --keys`).
+- **Windows SSH tunnel.** `computer start --host ssh://user@winbox` opens the
+  tunnel. The Windows driver runs in the interactive desktop session, so the SSH
+  user must have an active logon session.
+
+## Development
 
 ```sh
-bash scripts/install.sh
-bash scripts/test.sh
-bash scripts/build.sh
-# Run dist/<platform>-<arch>/computer directly for development.
-bash scripts/release.sh                 # print the release plan
-bash scripts/release.sh --confirm       # publish the reviewed commit
+bash scripts/install.sh   # install dependencies
+bash scripts/test.sh      # typecheck + tests
+bash scripts/build.sh     # compile this platform's binary into dist/
 ```
 
-Run publishing through an existing npm credential bundle, for example `secrets exec npmjs.com -- bash scripts/release.sh --confirm`. Release requires a clean linked worktree at the reviewed remote main commit. It verifies packed file lists, checks immutable package integrity on retries, and checks a registry installation. Live desktop verification remains part of delivery.
+`scripts/build.sh` bundles and minifies `src/`, obfuscates the bundle offline
+with a pinned tool and fixed seed, then compiles a bytecode binary per target.
+Run `dist/<platform>-<arch>/computer` directly during development.
+
+Release with the repository's own script:
+
+```sh
+bash scripts/release.sh                 # print the release plan (dry run)
+secrets exec npmjs.com -- bash scripts/release.sh --confirm
+```
+
+Release requires a clean linked worktree at the reviewed `origin/main` commit. It
+builds every platform, verifies packed file lists and immutable package
+integrity, publishes the launcher and per-platform packages, and checks a
+registry install. Live desktop verification remains part of delivery.
+
+Minification, obfuscation and bytecode compilation raise the bar for casual
+inspection; they are not a guarantee against reverse engineering. Keep
+credentials out of the binary.
+
+## License
+
+Functional Source License 1.1 with an Apache 2.0 future grant
+([FSL-1.1-Apache-2.0](LICENSE)).
+
+## Browse run history
+
+`computer sessions` opens a searchable picker: arrows move, Enter shows run
+details and saved screenshots, and Escape goes back. `--search <text>` also
+filters scripted output, `--json` remains a row array, and `--no-interactive`
+prints the table. `--since <when>` and `--until <when>` keep runs whose last
+activity falls in that window and accept the same values as `sessions`: `2h`,
+`7d`, `4w`, `1mo`, `1y` or an ISO date. An unreadable value exits 2. `--limit
+<n>` caps the table (default 50) and, when given, `--json`; when rows are left
+out, the count and the `--limit` that shows them go to stderr, so piped stdout
+carries rows only. The picker header
+lists the active filters. `--open latest` (or a screenshot filename) opens a recorded
+capture. Live Linux desktops remain under `computer task start/list/stop`.
+
+Computer owns 365-day run summaries under `.history/computer/history.db`.
+Counts survive the 30-day action ledger; summaries retain the latest 120
+actions and 120 screenshot locations per run. Typed text is never stored.
+Existing ledger entries and legacy Agents summaries are adopted without
+modifying their source. Legacy runs older than 365 days are skipped. The
+automatic import runs once; if the legacy database cannot be read it is reported
+once, and `computer sessions migrate [--from /path/to/sessions.db]` retries.
+The command safely repeats the summary import. Sessions need not be installed.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@phnx-labs/computer-cli",
-  "version": "0.1.5",
+  "version": "0.1.8",
   "description": "Native desktop automation over Accessibility, UI Automation and VNC",
   "type": "module",
   "bin": {
@@ -25,14 +25,16 @@
     "test": "bash scripts/test.sh"
   },
   "optionalDependencies": {
-    "@phnx-labs/computer-cli-darwin-arm64": "0.1.5",
-    "@phnx-labs/computer-cli-darwin-x64": "0.1.5",
-    "@phnx-labs/computer-cli-linux-arm64": "0.1.5",
-    "@phnx-labs/computer-cli-linux-x64": "0.1.5",
-    "@phnx-labs/computer-cli-win32-x64": "0.1.5"
+    "@phnx-labs/computer-cli-darwin-arm64": "0.1.8",
+    "@phnx-labs/computer-cli-darwin-x64": "0.1.8",
+    "@phnx-labs/computer-cli-linux-arm64": "0.1.8",
+    "@phnx-labs/computer-cli-linux-x64": "0.1.8",
+    "@phnx-labs/computer-cli-win32-x64": "0.1.8"
   },
   "devDependencies": {
-    "@inquirer/prompts": "8.5.2",
+    "@modelcontextprotocol/sdk": "1.30.0",
+    "@phnx-labs/cli-docs": "0.1.1",
+    "@phnx-labs/tui": "file:vendor/phnx-labs-tui-0.1.0.tgz",
     "@types/bun": "1.3.10",
     "@types/node": "26.1.0",
     "bun": "1.4.1",