@phnx-labs/computer-cli 0.1.10 → 0.1.12
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +14 -0
- package/README.md +311 -226
- package/package.json +6 -6
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.1.12
|
|
4
|
+
|
|
5
|
+
- A VNC screenshot scaled to `--max-dimension` (the default for action and MCP captures) takes about half as long: 194 ms instead of 380 ms for 1920x1080 to 1280x720 on a GNOME desktop, so a click with its screenshot returns in about 534 ms instead of 721 ms. The box filter no longer allocates per output pixel; the image is the same.
|
|
6
|
+
|
|
7
|
+
## 0.1.11
|
|
8
|
+
|
|
9
|
+
- `computer mcp` over VNC (a `vnc://` host, an isolated Linux task, or a task created on a `vnc://` host; a task handle records which, so a `--desktop-task` reconnect keeps the same tools) lists only what works there: `screenshot`, `click`, `right-click`, `drag`, `scroll`, `type-text`, `key` and `wait`, without the `bundle`, `pid` and `windowId` parameters. It listed `apps`, `describe`, `get-text` and `launch`, which VNC refuses, and had no `drag` or `right-click`.
|
|
10
|
+
- Over VNC, MCP coordinates are pixels of the latest screenshot it returned. On a 1920x1080 desktop the model sees 1280x720 images; it used to have to divide by `scale` itself, and now the server maps `x`/`y`, `from`/`to` and scroll deltas, and returns OCR word bounds in the same pixels. Screenshots it returns no longer carry `origin`, `scale` or `scale_y`.
|
|
11
|
+
- `computer mcp` gains `right-click`, `drag` and `click` `count` on every backend.
|
|
12
|
+
- `type-text` over VNC types a newline (`\n`, `\r\n` or `\r`, each once) as Enter, a tab as Tab and a backspace as BackSpace. They were sent as Unicode keysyms that X ignores, so `"Total\tExpenses\n=SUM(A1:A9)"` landed as `TotalExpenses=SUM(A1:A9)` in one cell.
|
|
13
|
+
- VNC MCP input tools take `screenshot: false` to skip the screen after them. Several calls sent together each returned an image; now only the last needs to.
|
|
14
|
+
- `computer mcp --max-dimension <px>` sets the longest edge of every screenshot it returns (default 1280).
|
|
15
|
+
- `computer mcp --host …` (or `--device`) uses the shared desktop, even when the config file says `isolated`. It failed with "An isolated task creates a new local Linux desktop" unless `--desktop-mode shared` was added.
|
|
16
|
+
|
|
3
17
|
## 0.1.10
|
|
4
18
|
|
|
5
19
|
- Over VNC, `click`, `right-click`, `type`, `type-text`, `key`, `scroll` and `drag` return once the screen has stopped changing (250 ms with no changed pixel, at most 1.5 s), instead of as soon as the events are written. Text typed right after a click no longer lands before the field has focus, and the next screenshot shows the result without a `sleep`.
|
package/README.md
CHANGED
|
@@ -1,256 +1,358 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
1
3
|
# computer-cli
|
|
2
4
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
5
|
+
**Let your coding agent use desktop apps.**<br>
|
|
6
|
+
Mac apps by element, Windows apps over SSH, and any Linux, macOS or Windows desktop over VNC.
|
|
7
|
+
|
|
8
|
+
[](https://www.npmjs.com/package/@phnx-labs/computer-cli)
|
|
9
|
+
[](#platform-support)
|
|
10
|
+
[](LICENSE)
|
|
11
|
+
|
|
12
|
+
[Install](#install) · [Benchmark](#benchmark) · [How it compares](#how-it-compares) · [Driving a desktop](#driving-a-desktop) · [Remote desktops](#remote-desktops) · [Command reference](docs/command-index.md)
|
|
6
13
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
+
</div>
|
|
15
|
+
|
|
16
|
+
<picture>
|
|
17
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/how-it-works-dark.svg">
|
|
18
|
+
<img src="docs/assets/how-it-works-light.svg" alt="An agent calls the computer CLI, which drives Mac apps through Accessibility, Windows apps through UI Automation over SSH, and any desktop over VNC" width="100%">
|
|
19
|
+
</picture>
|
|
20
|
+
|
|
21
|
+
<picture>
|
|
22
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/benchmark-strip-dark.svg">
|
|
23
|
+
<img src="docs/assets/benchmark-strip-light.svg" alt="Model calls per task on Cua's cua-bench-basic: computer-cli 0.1.10 5.38, Cua MCP server 6.71, Cua do CLI 8.58" width="100%">
|
|
24
|
+
</picture>
|
|
14
25
|
|
|
15
26
|
## Install
|
|
16
27
|
|
|
17
28
|
```sh
|
|
18
29
|
npm install -g @phnx-labs/computer-cli
|
|
19
|
-
computer
|
|
30
|
+
npx skills add phnx-labs/.agents --skill computer # teaches your agents to use it
|
|
20
31
|
```
|
|
21
32
|
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
33
|
+
Or paste this into your agent:
|
|
34
|
+
|
|
35
|
+
```text
|
|
36
|
+
Install computer-cli (npm install -g @phnx-labs/computer-cli) and the computer
|
|
37
|
+
skill (npx skills add phnx-labs/.agents --skill computer). Then take a
|
|
38
|
+
screenshot of my desktop and tell me which apps are open.
|
|
39
|
+
```
|
|
25
40
|
|
|
26
|
-
|
|
41
|
+
On a Mac, run `computer setup` once to install the signed helper; see [First run on a Mac](#first-run-on-a-mac).
|
|
42
|
+
|
|
43
|
+
<p align="center">
|
|
44
|
+
<img src="docs/assets/demo.gif" alt="An agent takes one screenshot, then fills and submits a contact form in Firefox with one chained shell call: click and type-text for name, email and age, key U,K,Return for the country, two clicks, the comments and submit. Each frame is that step's own --screenshot." width="820">
|
|
45
|
+
</p>
|
|
46
|
+
|
|
47
|
+
## Why computer-cli
|
|
48
|
+
|
|
49
|
+
- **Every desktop, one command.** Mac apps through Accessibility, Windows apps
|
|
50
|
+
through UI Automation over SSH, and any Linux, macOS or Windows desktop over
|
|
51
|
+
VNC. The verbs are the same everywhere: `screenshot`, `click`, `type-text`,
|
|
52
|
+
`key`, `scroll`, `drag`.
|
|
53
|
+
- **Fewer model calls.** Each command is a shell call the agent can chain, and
|
|
54
|
+
over VNC every input returns once the screen has stopped changing, so a whole
|
|
55
|
+
form goes in one call with no `sleep`. `--screenshot` on an action captures the
|
|
56
|
+
result in the same call. On Cua's own benchmark that came to 20% fewer model
|
|
57
|
+
calls than Cua's MCP server; see [Benchmark](#benchmark).
|
|
58
|
+
- **Elements, not pixels, on a Mac.** `describe` lists the app's Accessibility
|
|
59
|
+
tree, and `click --id @e12` presses the element through Accessibility without
|
|
60
|
+
bringing the app forward or moving your cursor, so you keep working while the
|
|
61
|
+
agent does.
|
|
62
|
+
- **Isolated desktops for parallel agents.** On Linux, each MCP task gets its
|
|
63
|
+
own X display, so several agents can drive GUI apps at once without
|
|
64
|
+
stealing each other's focus.
|
|
65
|
+
- **Deny by default on a Mac.** The Mac helper drives only the apps you allow,
|
|
66
|
+
and screenshots of the whole display leave out windows of apps outside that
|
|
67
|
+
list. A Windows host over SSH, a VNC desktop and an Xvfb task have no such
|
|
68
|
+
list: give the agent a desktop that holds only what it should touch.
|
|
69
|
+
- **Your agent, no model of its own.** `computer` and `computer mcp` make no
|
|
70
|
+
model calls; Claude Code, Codex, Cursor or any MCP client does the thinking.
|
|
71
|
+
`computer run` is an opt-in built-in loop for when you want one.
|
|
72
|
+
|
|
73
|
+
## Benchmark
|
|
74
|
+
|
|
75
|
+
<picture>
|
|
76
|
+
<source media="(prefers-color-scheme: dark)" srcset="docs/assets/benchmark-dark.svg">
|
|
77
|
+
<img src="docs/assets/benchmark-light.svg" alt="Model calls, cost and seconds per task for computer-cli 0.1.10, Cua's MCP server and Cua's do CLI over 12 tasks; computer-cli makes the fewest model calls at 5.38 and costs least at $0.055, Cua's MCP server is fastest at 14.2 s" width="100%">
|
|
78
|
+
</picture>
|
|
79
|
+
|
|
80
|
+
- **Fewest model calls:** 5.38 per task, against 6.71 for Cua's MCP server and
|
|
81
|
+
8.58 for its `cua do` CLI.
|
|
82
|
+
- **Lowest cost:** $0.055 per task, 21% less than Cua's MCP server ($0.070)
|
|
83
|
+
and 18% less than `cua do` ($0.068).
|
|
84
|
+
- **Every run passed:** 24 of 24. Cua's MCP server missed a toggle switch and
|
|
85
|
+
`cua do` a button click.
|
|
86
|
+
- **Not the fastest:** Cua's MCP server took 14.2 s per task against 16.5 s.
|
|
87
|
+
That gap includes the benchmark's wrapper, which runs every `computer` call
|
|
88
|
+
through `docker exec` and copies screenshots back out of the container.
|
|
89
|
+
- **Why:** on the multi-step tasks, a date and a contact form, computer-cli took
|
|
90
|
+
10 tool calls (half of them screenshot reads) where the MCP server took 13 to
|
|
91
|
+
27, one per action. `cua do` chains too, but writes each screenshot to a new
|
|
92
|
+
timestamped file, so every look costs a second model call to read it.
|
|
93
|
+
|
|
94
|
+
<details>
|
|
95
|
+
<summary>Method and every run</summary>
|
|
96
|
+
|
|
97
|
+
Claude Sonnet 5.5 ran each task headless in Claude Code (`claude -p`) with one
|
|
98
|
+
desktop tool and that tool's skill, on 2026-10-20. Tasks are variants 1 and 2 of
|
|
99
|
+
Cua's [cua-bench-basic](https://github.com/trycua/cua), one run each, each in
|
|
100
|
+
its own 1280 × 800 Linux container and scored by the task's own check.
|
|
101
|
+
computer-cli 0.1.10 ran over VNC with a draft of the "Linux and VNC desktops"
|
|
102
|
+
section later published in the computer skill
|
|
103
|
+
([phnx-labs/.agents#533](https://github.com/phnx-labs/.agents/pull/533)); Cua
|
|
104
|
+
ran with its stock GUI-automation skill. Every run started with a cold prompt
|
|
105
|
+
cache and used a subscription login. Cost is Claude Code's own per-run
|
|
106
|
+
estimate. A model call is one model response; one response can make several
|
|
107
|
+
tool calls.
|
|
108
|
+
|
|
109
|
+
Each cell gives the two runs' model calls; ✗ marks a failed run.
|
|
110
|
+
|
|
111
|
+
| Task | computer-cli 0.1.10 | Cua MCP server | Cua `do` CLI |
|
|
112
|
+
| --- | --- | --- | --- |
|
|
113
|
+
| Click a button | 4 / 4 | 5 / 4 | 5 / 7 ✗ |
|
|
114
|
+
| Click an icon | 4 / 5 | 5 / 5 | 6 / 5 |
|
|
115
|
+
| Pick a color | 4 / 4 | 5 / 5 | 7 / 7 |
|
|
116
|
+
| Set a date | 8 / 8 | 10 / 7 | 13 / 15 |
|
|
117
|
+
| Drag a slider to a value | 3 / 5 | 3 / 6 | 3 / 13 |
|
|
118
|
+
| Fill and submit a contact form | 8 / 7 | 10 / 14 | 12 / 13 |
|
|
119
|
+
| Choose from a right-click menu | 6 / 6 | 7 / 7 | 9 / 9 |
|
|
120
|
+
| Choose from a dropdown | 5 / 5 | 6 / 6 | 7 / 7 |
|
|
121
|
+
| Edit a spreadsheet cell | 4 / 5 | 5 / 5 | 5 / 5 |
|
|
122
|
+
| Flip a toggle switch | 4 / 4 | 8 / 5 ✗ | 7 / 7 |
|
|
123
|
+
| Type into a field | 5 / 5 | 5 / 5 | 7 / 7 |
|
|
124
|
+
| Operate a video player | 6 / 10 | 9 / 14 | 13 / 17 |
|
|
125
|
+
| Drag an item into a box (left out) | 13 ✗ / 13 ✗ | 23 ✗ / 24 ✗ | 41 ✗ / 5 ✗ |
|
|
126
|
+
| **Mean, 12 tasks** | **5.38 calls, 16.5 s, 24/24** | **6.71 calls, 14.2 s, 23/24** | **8.58 calls, 19.5 s, 23/24** |
|
|
127
|
+
|
|
128
|
+
The 13th task, drag-drop, is left out: every tool failed every run, and so does
|
|
129
|
+
the benchmark's own reference solution, because the page's HTML5 drop never
|
|
130
|
+
fires under synthetic X input in this container. On variant 0 with
|
|
131
|
+
computer-cli 0.1.9, before input waited for the screen to settle, computer-cli
|
|
132
|
+
and Cua's MCP server were level at 6.04 model calls each. With one run per
|
|
133
|
+
variant, read single rows as a direction. More measurements, including action
|
|
134
|
+
latency over VNC, are in [docs/benchmarks.md](docs/benchmarks.md).
|
|
135
|
+
|
|
136
|
+
</details>
|
|
137
|
+
|
|
138
|
+
## How it compares
|
|
139
|
+
|
|
140
|
+
| Tool | Drives | Element ids | Without taking your cursor | Other machines | A desktop per agent | Interface |
|
|
141
|
+
| --- | --- | --- | --- | --- | --- | --- |
|
|
142
|
+
| **computer-cli** | **macOS (Accessibility), Windows (UI Automation), any desktop over VNC** | **yes on macOS and Windows; coordinates over VNC** | **yes for elements that take a press or a value** | **Windows over SSH; any VNC server (its own RFB client)** | **yes: an Xvfb display per MCP task on Linux** | **CLI, MCP** |
|
|
143
|
+
| Cua (`cua-driver`, `cua do`) | macOS, Windows, Linux (AT-SPI) | yes in cua-driver; `cua do` is mostly coordinates | yes, events posted to the app | its own sandboxes and hosts through `cua-spacesd`; VNC for its VMs | yes: a VM or container per sandbox | CLI, MCP, SDKs |
|
|
144
|
+
| Peekaboo | macOS only | yes | yes, background by default | no | no | CLI, MCP |
|
|
145
|
+
| Windows-MCP | Windows only | labels, clicked as coordinates | no: moves the cursor and brings the window forward | MCP over HTTP with a key | no | MCP |
|
|
146
|
+
| terminator | Windows only | selectors | `invoke` and `set_value` yes; `click` moves the cursor | MCP over HTTP with a token | no | MCP, CLI, SDKs |
|
|
147
|
+
| usecomputer | macOS, Windows, Linux (X11) | no, coordinates | no | no | no | CLI, library |
|
|
148
|
+
| Anthropic computer-use demo | a Linux desktop in Docker | no, coordinates | its own virtual display | no | one desktop per container | Streamlit app, Python loop |
|
|
149
|
+
|
|
150
|
+
Read from each tool's source at a pinned commit on 2026-10-20: trycua/cua
|
|
151
|
+
`a4b5fec`, openclaw/Peekaboo `213a071`, CursorTouch/Windows-MCP `5b24a8a`,
|
|
152
|
+
mediar-ai/terminator `73a381c`, remorses/usecomputer `711a429`,
|
|
153
|
+
anthropics/anthropic-quickstarts `9ec32b9`. computer-cli is the only one that
|
|
154
|
+
drives Windows over SSH and the only one whose own VNC client is its desktop
|
|
155
|
+
driver for agents. Others reach further elsewhere: Cua reads the Linux
|
|
156
|
+
accessibility tree through AT-SPI, where computer-cli has coordinates only, and
|
|
157
|
+
runs agents in full VMs. OpenAI's Codex and Anthropic's Claude apps also offer
|
|
158
|
+
computer use on macOS and Windows; they are closed source and drive the machine
|
|
159
|
+
they run on.
|
|
160
|
+
|
|
161
|
+
## Security and privacy
|
|
162
|
+
|
|
163
|
+
- **An agent with `computer` acts as you.** It can click and type anywhere an
|
|
164
|
+
allowed app lets you. Allow only the apps a task needs.
|
|
165
|
+
- **Deny by default on a Mac.** No app is drivable through the Mac helper
|
|
166
|
+
until you add it to `~/.agents/permissions/groups/<name>.yaml`;
|
|
167
|
+
see [docs/computer.md](docs/computer.md#authorize-the-apps-it-may-drive).
|
|
168
|
+
`--display` screenshots leave out windows of apps outside the list, and a
|
|
169
|
+
coordinate click on a Mac is refused unless the app under the point is allowed.
|
|
170
|
+
- **Windows and VNC desktops are driven whole.** The Windows helper has no
|
|
171
|
+
allow-list; access to it is your SSH login. Over `vnc://` and on isolated
|
|
172
|
+
Xvfb tasks, the agent can click anything on that display. Point it at a
|
|
173
|
+
machine or task desktop that holds only what it should touch.
|
|
174
|
+
- **Nothing leaves your machine.** The CLI sends no telemetry. It talks to the
|
|
175
|
+
local helper, to the machines you name over your own SSH config, and to VNC
|
|
176
|
+
servers you point it at. It downloads only its signed helpers, checked
|
|
177
|
+
against published SHA-256 sums. Only `computer run` calls a model provider.
|
|
178
|
+
- **What ships.** npm installs a small launcher and one compiled binary per
|
|
179
|
+
platform, published from this repository's release workflow through npm
|
|
180
|
+
trusted publishing (OIDC), so no npm token exists to leak.
|
|
181
|
+
|
|
182
|
+
## First run on a Mac
|
|
27
183
|
|
|
28
184
|
```sh
|
|
29
|
-
computer setup
|
|
30
|
-
computer start
|
|
31
|
-
computer
|
|
32
|
-
computer
|
|
33
|
-
computer
|
|
34
|
-
computer click --id <element> # click an element from the tree
|
|
35
|
-
computer type --id <field> "hello" # set a field value
|
|
36
|
-
computer key "cmd+shift+s" # send a key chord
|
|
37
|
-
computer stop # deactivate the daemon
|
|
185
|
+
computer setup # install the signed helper
|
|
186
|
+
computer start # start its daemon (launchd)
|
|
187
|
+
computer status # install, policy, trust
|
|
188
|
+
computer apps # what it may drive
|
|
189
|
+
computer screenshot --out desktop.png --display # the whole screen
|
|
38
190
|
```
|
|
39
191
|
|
|
40
|
-
`setup`
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
192
|
+
`setup` installs the signed helper in `/Applications/`. macOS asks you to grant
|
|
193
|
+
it Accessibility and Screen Recording; setup cannot grant them for you. `stop`
|
|
194
|
+
deactivates the daemon.
|
|
195
|
+
|
|
196
|
+
## Driving a desktop
|
|
197
|
+
|
|
198
|
+
The loop an agent runs: look, act, look at the result.
|
|
45
199
|
|
|
46
|
-
|
|
200
|
+
```sh
|
|
201
|
+
computer describe --bundle com.apple.notes # element ids @e1…
|
|
202
|
+
computer click --bundle com.apple.notes --id @e12 # press an element
|
|
203
|
+
computer type --bundle com.apple.notes --id @e7 --text "Groceries"
|
|
204
|
+
computer key --keys "cmd+s" # or several: "Right,1,Tab"
|
|
205
|
+
computer click --x 640 --y 300 --screenshot after.png # act, then look
|
|
206
|
+
computer screenshot --host vnc://box --out s.png --ocr # VNC: local OCR
|
|
207
|
+
```
|
|
47
208
|
|
|
48
|
-
|
|
49
|
-
|
|
209
|
+
- **Elements first.** `describe` ids feed `click`, `type`, `focus`, `get-text`
|
|
210
|
+
and `ax-action`. They act through Accessibility (AXPress, setting the value),
|
|
211
|
+
so the app stays where it is and your cursor does not move. An element with
|
|
212
|
+
no press action is clicked at its centre instead, which moves the cursor
|
|
213
|
+
unless you pass `--background`. Ids are never reused; after a new
|
|
214
|
+
`describe`, older ids answer `element_stale`.
|
|
215
|
+
- **Coordinates when there is no tree:** VM guests, canvas apps, games and VNC.
|
|
216
|
+
Coordinate input moves the real cursor and needs the app in front, and
|
|
217
|
+
`computer` prints a note whenever an action will cost you focus.
|
|
218
|
+
- **Chain known steps.** Over VNC, every input returns once the screen has
|
|
219
|
+
stopped changing (250 ms with no changed pixel, at most 1.5 s), so
|
|
220
|
+
`click … && type-text … && key …` needs no `sleep`. Add `--screenshot <path>`
|
|
221
|
+
to the last action to see the result in the same call; on a Mac or Windows,
|
|
222
|
+
look before relying on a slow app having caught up.
|
|
223
|
+
- **Local OCR over VNC.** `screenshot --ocr` returns text, word boxes and
|
|
224
|
+
confidence from a local `tesseract`, with no model call.
|
|
225
|
+
- **Screenshots report their scale.** A screenshot's pixels are click
|
|
226
|
+
coordinates when `scale` is 1; otherwise divide by it. The `saved:` line and
|
|
227
|
+
`--json` both print it.
|
|
228
|
+
|
|
229
|
+
`computer sessions` browses what was driven, one row per command grouped by
|
|
230
|
+
run; `--since 2h`, `--search <text>` and `--open latest` (the recorded
|
|
231
|
+
screenshot) narrow it down.
|
|
232
|
+
|
|
233
|
+
Run `computer <command> --help` for every option, or browse
|
|
234
|
+
[docs/command-reference.html](docs/command-reference.html).
|
|
235
|
+
|
|
236
|
+
## Remote desktops
|
|
237
|
+
|
|
238
|
+
`--host` selects a remote desktop:
|
|
50
239
|
|
|
51
240
|
```sh
|
|
52
|
-
# Windows:
|
|
53
|
-
computer
|
|
54
|
-
computer
|
|
241
|
+
# Windows: the UI Automation driver over SSH
|
|
242
|
+
computer setup --host ssh://user@winbox
|
|
243
|
+
computer start --host ssh://user@winbox
|
|
244
|
+
computer describe --host ssh://user@winbox
|
|
245
|
+
computer click --host ssh://user@winbox --id @e4
|
|
55
246
|
|
|
56
247
|
# VNC: any GUI desktop (Linux, macOS, Windows) over RFB
|
|
57
|
-
computer screenshot --host vnc://
|
|
58
|
-
computer click --host vnc://
|
|
59
|
-
computer type-text --host vnc://
|
|
60
|
-
computer
|
|
248
|
+
computer screenshot --host vnc://build-box:5900 --out desk.png
|
|
249
|
+
computer click --host vnc://build-box:5900 --x 512 --y 384 --screenshot after.png
|
|
250
|
+
computer type-text --host vnc://build-box:5900 --text "hello"
|
|
251
|
+
computer drag --host vnc://build-box:5900 --from 200,300 --to 600,300
|
|
61
252
|
```
|
|
62
253
|
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
254
|
+
- **Windows** runs the helper in the logged-in interactive session, reached over
|
|
255
|
+
an authenticated SSH tunnel; an SSH login alone is not a GUI session.
|
|
256
|
+
- **VNC** gives screenshots and coordinate input (`click`, `right-click`,
|
|
257
|
+
`type-text`, `key`, `scroll`, `drag`). `describe`, `get-text` and `--id` need
|
|
258
|
+
a native driver and answer `unsupported_over_vnc`. Pass `--vnc-password` or
|
|
259
|
+
set `COMPUTER_HELPER_VNC_PASSWORD` for a server with a password.
|
|
69
260
|
|
|
70
261
|
## Agent tools and isolated Linux desktops
|
|
71
262
|
|
|
72
|
-
`computer mcp` serves
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
263
|
+
`computer mcp` serves the same verbs as MCP tools over stdio. Input tools return
|
|
264
|
+
a compact screenshot as an MCP image with the action's result, and it makes no
|
|
265
|
+
model calls.
|
|
266
|
+
|
|
267
|
+
```sh
|
|
268
|
+
claude mcp add computer -- computer mcp # Claude Code; any MCP client works the same way
|
|
269
|
+
claude mcp add computer -- computer mcp --host vnc://localhost:5901 # a VNC desktop (shared)
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
Over VNC (a `vnc://` host, an isolated Linux task, or a task created on a
|
|
273
|
+
`vnc://` host) MCP lists only `screenshot`, `click` (with `count`),
|
|
274
|
+
`right-click`, `drag`, `scroll`, `type-text`, `key` and `wait`. Their `x`/`y`
|
|
275
|
+
are pixels of the latest screenshot it returned; the server maps them to the
|
|
276
|
+
desktop, so the model never applies `scale`. Input tools take
|
|
277
|
+
`screenshot: false` to skip the image when several are sent together, and
|
|
278
|
+
`--max-dimension` sets the screenshot size (default 1280).
|
|
76
279
|
|
|
77
280
|
On Linux, each new MCP task gets its own desktop by default. Install `Xvfb`,
|
|
78
|
-
`x11vnc` (with Unix-socket support), `xauth
|
|
79
|
-
|
|
281
|
+
`x11vnc` (with Unix-socket support), `xauth` and util-linux (`flock`,
|
|
282
|
+
`setpriv`). The displays start empty; launch an app with `launch --path`.
|
|
80
283
|
|
|
81
284
|
```sh
|
|
82
|
-
computer
|
|
83
|
-
computer
|
|
84
|
-
computer launch --path /path/to/app --desktop-task TASK_ID
|
|
285
|
+
computer task start # a task for CLI use; prints its id
|
|
286
|
+
computer launch --path /usr/bin/gedit --desktop-task TASK_ID
|
|
85
287
|
computer click --x 400 --y 300 --screenshot --desktop-task TASK_ID
|
|
86
|
-
computer mcp --desktop-task TASK_ID
|
|
288
|
+
computer mcp --desktop-task TASK_ID # connect another agent to that task
|
|
87
289
|
computer task list
|
|
88
290
|
computer task stop TASK_ID
|
|
89
291
|
```
|
|
90
292
|
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
```
|
|
293
|
+
`computer config desktopMode shared` makes tasks share one desktop instead;
|
|
294
|
+
`--desktop-mode` overrides it for one task. Isolation covers the X display,
|
|
295
|
+
windows, focus and pointer, not the user, filesystem, app profiles or
|
|
296
|
+
credentials, and task handles are not a security boundary against code running
|
|
297
|
+
as the same user. No TCP port is opened for an isolated desktop. Idle tasks
|
|
298
|
+
expire after 15 minutes. Managed desktops are Linux-only; on a Mac, MCP uses
|
|
299
|
+
the helper.
|
|
99
300
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
without `--desktop-task` do not participate in that coordination. Disconnecting
|
|
104
|
-
one client does not stop another client's task. Idle tasks expire after 15 minutes.
|
|
301
|
+
`computer run --task "open Notes and write a haiku"` drives an app with a
|
|
302
|
+
built-in model loop, for when there is no agent around it. It is the only
|
|
303
|
+
command that calls a model provider.
|
|
105
304
|
|
|
106
|
-
|
|
107
|
-
filesystem, application profiles or credentials. Applications that reuse a
|
|
108
|
-
global profile may require their own separate-profile launch configuration.
|
|
109
|
-
Private task handles prevent accidental cross-task access; they are not a
|
|
110
|
-
security boundary against hostile code running as the same OS user. No TCP port
|
|
111
|
-
is opened for an isolated desktop. Native macOS shared-mode MCP uses the existing
|
|
112
|
-
helper; managed task provisioning currently requires Linux.
|
|
305
|
+
## Agent skill
|
|
113
306
|
|
|
114
|
-
|
|
307
|
+
The computer skill teaches an agent the loop above, when to use elements or
|
|
308
|
+
coordinates, and the VNC rules. It lives in the
|
|
309
|
+
[phnx-labs/.agents](https://github.com/phnx-labs/.agents/tree/main/skills/computer)
|
|
310
|
+
skill collection:
|
|
115
311
|
|
|
116
312
|
```sh
|
|
117
|
-
|
|
118
|
-
computer screenshot --host vnc://localhost:5901 --ocr --out screen.png --json
|
|
119
|
-
```
|
|
120
|
-
|
|
121
|
-
`--ocr` uses local Tesseract and includes text, word bounds and confidence, without
|
|
122
|
-
a model call. For input commands it implies a post-action capture. OCR coordinates
|
|
123
|
-
refer to the original desktop; screenshot `scale` and `scale_y` map that desktop
|
|
124
|
-
into the resized image. Images preserve aspect ratio and default to a maximum
|
|
125
|
-
edge of 1280 pixels for action/MCP observations. A screenshot is an observation,
|
|
126
|
-
not a guarantee that an application's asynchronous work has finished.
|
|
127
|
-
|
|
128
|
-
If capture fails after a successful action, the action remains successful and
|
|
129
|
-
the capture error is returned separately. Retry observation, not the action.
|
|
130
|
-
Native compact capture requires a helper supporting `max_dimension`; an older
|
|
131
|
-
helper that ignores the requested bound reports `helper_update_required`.
|
|
132
|
-
|
|
133
|
-
## How fast are action observations?
|
|
134
|
-
|
|
135
|
-
In a September 15, 2026 Linux arm64 benchmark, the installed **0.1.5** CLI was
|
|
136
|
-
compared with the **0.1.6 candidate** at `60ca877`. The candidate was not released
|
|
137
|
-
or installed at measurement time. Gedit displayed the real 729-line
|
|
138
|
-
`src/commands/actions.ts`; each sample sent Page Down and captured the result on
|
|
139
|
-
the same owned, isolated 1600 × 1000 desktop.
|
|
140
|
-
|
|
141
|
-
| Workflow | Median | Observed range | Samples |
|
|
142
|
-
| --- | ---: | ---: | ---: |
|
|
143
|
-
| 0.1.5: separate action and screenshot | 4,494 ms | 4,453–4,552 ms | 8 |
|
|
144
|
-
| Candidate: separate action and screenshot | 515 ms | 454–691 ms | 8 |
|
|
145
|
-
| Candidate: combined action and screenshot | 712 ms | 693–947 ms | 8 |
|
|
146
|
-
| Candidate: MCP image response | 533 ms | 496–578 ms | 8 |
|
|
147
|
-
| Candidate: MCP image + local OCR | 1,205 ms | 1,111–1,364 ms | 8 |
|
|
148
|
-
|
|
149
|
-
The candidate MCP workflow had **88.1% lower median latency (8.4× speedup)** than the released
|
|
150
|
-
separate-call workflow. Most of that improvement also exists in the candidate's
|
|
151
|
-
separate CLI calls, following the connection-close timer fix; it is not an
|
|
152
|
-
MCP-only speedup. Combined capture was **38% slower** than candidate separate
|
|
153
|
-
calls here, and MCP was not demonstrably faster than candidate separate calls.
|
|
154
|
-
OCR added **672 ms** to the MCP median.
|
|
155
|
-
|
|
156
|
-
Separate screenshots were full-size; combined/MCP observations used the default
|
|
157
|
-
1280 × 800 image and a 150 ms post-input receive window. MCP PNGs were **21.3%
|
|
158
|
-
smaller** (93,406 versus 118,695 bytes), excluding protocol/base64 overhead.
|
|
159
|
-
These compare default workflows, not identical capture settings.
|
|
160
|
-
|
|
161
|
-
Each mode had one warmup; measured order rotated between rounds. Timings include
|
|
162
|
-
CLI process start/exit or a tool request on an already-connected MCP client.
|
|
163
|
-
Desktop/app startup, the initial 67 ms MCP connection and harness verification
|
|
164
|
-
are excluded. All 40 captures matched the expected source section using
|
|
165
|
-
case-insensitive OCR anchors; original case-sensitive misses were retained in
|
|
166
|
-
the raw results. The busy host and eight samples per mode make this a directional
|
|
167
|
-
comparison, not a production tail-latency estimate. No model calls, token savings,
|
|
168
|
-
concurrent-task throughput, macOS latency or Windows latency were measured.
|
|
169
|
-
|
|
170
|
-
See the [benchmark evidence on PR #14](https://github.com/phnx-labs/computer-cli/pull/14#issuecomment-5683895205)
|
|
171
|
-
for the recorded run and delivery limits. Source links require repository access.
|
|
172
|
-
|
|
173
|
-
## Command reference
|
|
174
|
-
|
|
175
|
-
```bash
|
|
176
|
-
scripts/generate-cli-docs.sh # refresh docs/
|
|
177
|
-
scripts/generate-cli-docs.sh --check # fail on stale docs
|
|
178
|
-
scripts/generate-cli-docs.sh --out-dir .agents/scratch/cli-docs # preview
|
|
313
|
+
npx skills add phnx-labs/.agents --skill computer
|
|
179
314
|
```
|
|
180
315
|
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
from the output folder to browse the tree and search commands.
|
|
184
|
-
CI uploads a `computer-cli-docs` artifact on pull requests and pushes to `main`
|
|
185
|
-
or `release/**`. It contains the HTML reference plus Markdown and JSON indexes.
|
|
186
|
-
Tests and releases check the committed files; commit regenerated `docs/` with
|
|
187
|
-
command changes. Release checks also run with `--skip-tests` / `--skip-build`.
|
|
188
|
-
|
|
189
|
-
[docs/command-reference.html](docs/command-reference.html) lists every command
|
|
190
|
-
with its arguments and options, searchable, with the command tree in a sidebar.
|
|
191
|
-
[docs/command-index.md](docs/command-index.md) is the one-line-per-command index,
|
|
192
|
-
and [docs/command-index.json](docs/command-index.json) the structured form.
|
|
193
|
-
`scripts/generate-cli-docs.sh` regenerates all three from the registered command
|
|
194
|
-
tree; `scripts/test.sh` fails when they are stale.
|
|
316
|
+
The skill lists no flags. Agents read `computer <command> --help`, which always
|
|
317
|
+
matches the installed binary.
|
|
195
318
|
|
|
196
319
|
## Using it through agents-cli
|
|
197
320
|
|
|
198
|
-
Agents CLI ships `agents computer` as a thin consumer of this binary. It
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
The seam between the two is
|
|
202
|
-
[docs/
|
|
203
|
-
explained in [docs/computer.md](docs/computer.md).
|
|
321
|
+
Agents CLI ships `agents computer` as a thin consumer of this binary. It forwards
|
|
322
|
+
every verb unchanged and adds fleet device discovery (`--device` becomes a
|
|
323
|
+
resolved `--host`), agent-session identity and cross-machine session history.
|
|
324
|
+
The seam between the two is in [docs/integration.md](docs/integration.md), and
|
|
325
|
+
transports, addresses and verbs in [docs/computer.md](docs/computer.md).
|
|
204
326
|
|
|
205
327
|
## Platform support
|
|
206
328
|
|
|
207
|
-
| Platform | CLI
|
|
329
|
+
| Platform | CLI | Drives |
|
|
208
330
|
| --- | --- | --- |
|
|
209
|
-
| macOS | arm64, x64 |
|
|
210
|
-
|
|
|
211
|
-
|
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
Yes, for native element-level automation, and **both helpers already live in
|
|
220
|
-
this repository**, extracted from Agents CLI. Installing Agents CLI is not
|
|
221
|
-
required.
|
|
222
|
-
|
|
223
|
-
- **macOS:** the [Swift helper](native/computer-mac) supplies Accessibility,
|
|
224
|
-
ScreenCaptureKit capture and CoreGraphics input. Its signed app bundle keeps a
|
|
225
|
-
stable identity for the user's Accessibility and Screen Recording grants.
|
|
226
|
-
The CLI downloads/verifies it during setup and manages it through launchd.
|
|
227
|
-
- **Windows:** the [C#/.NET helper](native/computer-win) supplies UI Automation,
|
|
228
|
-
screen capture and `SendInput`. Setup provisions a scheduled task in the
|
|
229
|
-
logged-in interactive desktop session. The CLI reaches its loopback service
|
|
230
|
-
over an authenticated SSH tunnel; an SSH login alone is not a GUI session.
|
|
231
|
-
- **Linux/VNC:** no macOS or Windows helper is needed. A VNC server supplies
|
|
232
|
-
screenshot and coordinate input; isolated Linux tasks additionally need the
|
|
233
|
-
dependencies listed [above](#agent-tools-and-isolated-linux-desktops).
|
|
234
|
-
|
|
235
|
-
Native helpers retain their separate, immutable `computer-mac/v*` and
|
|
236
|
-
`computer-win/v*` release channels. An ordinary CLI release does not rebuild
|
|
237
|
-
them.
|
|
331
|
+
| macOS | arm64, x64 | Mac apps through the signed Accessibility helper; any desktop over VNC |
|
|
332
|
+
| Linux | arm64, x64 | Any desktop over VNC; isolated task desktops through Xvfb |
|
|
333
|
+
| Windows | x64 | Any desktop over VNC |
|
|
334
|
+
| Remote Windows | — | Windows apps through the UI Automation helper over SSH |
|
|
335
|
+
|
|
336
|
+
Both native helpers live in this repository: the [Swift helper](native/computer-mac)
|
|
337
|
+
(Accessibility, ScreenCaptureKit, CoreGraphics) and the [C#/.NET helper](native/computer-win)
|
|
338
|
+
(UI Automation, screen capture, `SendInput`). They ship on their own release
|
|
339
|
+
channels (`computer-mac/v*`, `computer-win/v*`); a CLI release does not rebuild
|
|
340
|
+
them. Linux needs no helper.
|
|
238
341
|
|
|
239
342
|
## Troubleshooting
|
|
240
343
|
|
|
241
344
|
- **Accessibility not trusted (macOS).** `computer status` shows whether the
|
|
242
345
|
helper has Accessibility and Screen Recording access. Grant both in System
|
|
243
|
-
Settings > Privacy & Security.
|
|
244
|
-
|
|
346
|
+
Settings > Privacy & Security. After a helper upgrade, `computer setup` may
|
|
347
|
+
need the grants again.
|
|
348
|
+
- **An app is missing from `computer apps`.** Add it to an allow group and run
|
|
349
|
+
`computer reload`; see [docs/computer.md](docs/computer.md#authorize-the-apps-it-may-drive).
|
|
350
|
+
- **`unsupported_over_vnc`.** `describe`, `get-text` and `--id` need a native
|
|
351
|
+
driver. Over VNC, use `click --x --y`, `type-text --text` and `key --keys`.
|
|
245
352
|
- **VNC password.** Pass `--vnc-password <pw>` or set
|
|
246
|
-
`COMPUTER_HELPER_VNC_PASSWORD`. A
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
`click --id`, `type --id`) require a native driver. Over VNC, use coordinate
|
|
250
|
-
input (`click --x --y`, `type-text --text`, `key --keys`).
|
|
251
|
-
- **Windows SSH tunnel.** `computer start --host ssh://user@winbox` opens the
|
|
252
|
-
tunnel. The Windows driver runs in the interactive desktop session, so the SSH
|
|
253
|
-
user must have an active logon session.
|
|
353
|
+
`COMPUTER_HELPER_VNC_PASSWORD`. A server with no authentication needs neither.
|
|
354
|
+
- **Windows over SSH.** `computer start --host ssh://user@winbox` opens the
|
|
355
|
+
tunnel. The SSH user needs an active logon session on that machine.
|
|
254
356
|
|
|
255
357
|
## Development
|
|
256
358
|
|
|
@@ -262,7 +364,17 @@ bash scripts/build.sh # compile this platform's binary into dist/
|
|
|
262
364
|
|
|
263
365
|
`scripts/build.sh` bundles and minifies `src/`, obfuscates the bundle offline
|
|
264
366
|
with a pinned tool and fixed seed, then compiles a bytecode binary per target.
|
|
265
|
-
Run `dist/<platform>-<arch>/computer` directly during development.
|
|
367
|
+
Run `dist/<platform>-<arch>/computer` directly during development. Minification,
|
|
368
|
+
obfuscation and bytecode compilation raise the bar for casual inspection; they
|
|
369
|
+
are not a guarantee against reverse engineering. Keep credentials out of the
|
|
370
|
+
binary.
|
|
371
|
+
|
|
372
|
+
`scripts/generate-cli-docs.sh` regenerates [docs/command-reference.html](docs/command-reference.html),
|
|
373
|
+
[docs/command-index.md](docs/command-index.md) and [docs/command-index.json](docs/command-index.json)
|
|
374
|
+
from the registered command tree; `--check` fails when they are stale, and
|
|
375
|
+
`scripts/test.sh` runs it. Commit regenerated `docs/` with command changes. CI
|
|
376
|
+
uploads the same files as the `computer-cli-docs` artifact on pull requests and
|
|
377
|
+
pushes to `main` or `release/**`.
|
|
266
378
|
|
|
267
379
|
Release with the repository's own script:
|
|
268
380
|
|
|
@@ -279,34 +391,7 @@ workflow in the `npm-publish` environment through trusted publishing, so no npm
|
|
|
279
391
|
stored anywhere. The script then checks a registry install. Live desktop verification
|
|
280
392
|
remains part of delivery.
|
|
281
393
|
|
|
282
|
-
Minification, obfuscation and bytecode compilation raise the bar for casual
|
|
283
|
-
inspection; they are not a guarantee against reverse engineering. Keep
|
|
284
|
-
credentials out of the binary.
|
|
285
|
-
|
|
286
394
|
## License
|
|
287
395
|
|
|
288
396
|
Functional Source License 1.1 with an Apache 2.0 future grant
|
|
289
397
|
([FSL-1.1-Apache-2.0](LICENSE)).
|
|
290
|
-
|
|
291
|
-
## Browse run history
|
|
292
|
-
|
|
293
|
-
`computer sessions` opens a searchable picker: arrows move, Enter shows run
|
|
294
|
-
details and saved screenshots, and Escape goes back. `--search <text>` also
|
|
295
|
-
filters scripted output, `--json` remains a row array, and `--no-interactive`
|
|
296
|
-
prints the table. `--since <when>` and `--until <when>` keep runs whose last
|
|
297
|
-
activity falls in that window and accept the same values as `sessions`: `2h`,
|
|
298
|
-
`7d`, `4w`, `1mo`, `1y` or an ISO date. An unreadable value exits 2. `--limit
|
|
299
|
-
<n>` caps the table (default 50) and, when given, `--json`; when rows are left
|
|
300
|
-
out, the count and the `--limit` that shows them go to stderr, so piped stdout
|
|
301
|
-
carries rows only. The picker header
|
|
302
|
-
lists the active filters. `--open latest` (or a screenshot filename) opens a recorded
|
|
303
|
-
capture. Live Linux desktops remain under `computer task start/list/stop`.
|
|
304
|
-
|
|
305
|
-
Computer owns 365-day run summaries under `.history/computer/history.db`.
|
|
306
|
-
Counts survive the 30-day action ledger; summaries retain the latest 120
|
|
307
|
-
actions and 120 screenshot locations per run. Typed text is never stored.
|
|
308
|
-
Existing ledger entries and legacy Agents summaries are adopted without
|
|
309
|
-
modifying their source. Legacy runs older than 365 days are skipped. The
|
|
310
|
-
automatic import runs once; if the legacy database cannot be read it is reported
|
|
311
|
-
once, and `computer sessions migrate [--from /path/to/sessions.db]` retries.
|
|
312
|
-
The command safely repeats the summary import. Sessions need not be installed.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@phnx-labs/computer-cli",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.12",
|
|
4
4
|
"description": "Native desktop automation over Accessibility, UI Automation and VNC",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"bin": {
|
|
@@ -25,11 +25,11 @@
|
|
|
25
25
|
"test": "bash scripts/test.sh"
|
|
26
26
|
},
|
|
27
27
|
"optionalDependencies": {
|
|
28
|
-
"@phnx-labs/computer-cli-darwin-arm64": "0.1.
|
|
29
|
-
"@phnx-labs/computer-cli-darwin-x64": "0.1.
|
|
30
|
-
"@phnx-labs/computer-cli-linux-arm64": "0.1.
|
|
31
|
-
"@phnx-labs/computer-cli-linux-x64": "0.1.
|
|
32
|
-
"@phnx-labs/computer-cli-win32-x64": "0.1.
|
|
28
|
+
"@phnx-labs/computer-cli-darwin-arm64": "0.1.12",
|
|
29
|
+
"@phnx-labs/computer-cli-darwin-x64": "0.1.12",
|
|
30
|
+
"@phnx-labs/computer-cli-linux-arm64": "0.1.12",
|
|
31
|
+
"@phnx-labs/computer-cli-linux-x64": "0.1.12",
|
|
32
|
+
"@phnx-labs/computer-cli-win32-x64": "0.1.12"
|
|
33
33
|
},
|
|
34
34
|
"devDependencies": {
|
|
35
35
|
"@modelcontextprotocol/sdk": "1.30.0",
|