@agent-sh/computer-use-linux 0.7.42 → 0.7.44
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +90 -87
- package/npm/README.md +27 -17
- package/package.json +7 -4
- package/pi/extension/generated-tools.ts +119 -27
- package/pi/extension/index.ts +245 -97
- package/skills/computer-use-linux/SKILL.md +27 -17
- package/skills/computer-use-linux/references/hermes-setup.md +6 -3
- package/skills/computer-use-linux/references/pi-setup.md +18 -8
package/npm/README.md
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# @agent-sh/computer-use-linux
|
|
2
2
|
|
|
3
|
-
Native Linux desktop observation and control for AI agents
|
|
3
|
+
Native Linux desktop observation and control for AI agents, published as
|
|
4
4
|
[`@agent-sh/computer-use-linux`](https://www.npmjs.com/package/@agent-sh/computer-use-linux).
|
|
5
5
|
|
|
6
6
|
Provides:
|
|
@@ -14,7 +14,7 @@ Provides:
|
|
|
14
14
|
> `ToolAnnotations` so hosts can distinguish read-only observation from mutating
|
|
15
15
|
> desktop actions.
|
|
16
16
|
|
|
17
|
-
On labwc, window targeting uses the Selkies IPC patch's `labwc.sock` when it exists. Stock labwc and other wlroots compositors that advertise wlr-foreign-toplevel-management use that protocol instead, which reports no pids or bounds. Activation requires a unique app-id/title match. Window IDs include the title and must be refreshed after it changes. Ambiguous windows are rejected. Window bounds and relative coordinate targeting are unavailable until output scaling can be verified.
|
|
17
|
+
On labwc, window targeting uses the Selkies IPC patch's `labwc.sock` when it exists. Stock labwc and other wlroots compositors that advertise wlr-foreign-toplevel-management use that protocol instead, which reports no pids or bounds. Activation requires a unique app-id/title match. Window IDs include the title and must be refreshed after it changes. Ambiguous windows are rejected. Window bounds and relative coordinate targeting are unavailable until output scaling can be verified. Window-targeted screenshots, including `screenshot_after: "window"`, fail for the same reason; capture the full screen instead.
|
|
18
18
|
|
|
19
19
|
Native X11 clicks, scrolls and drags use xdotool/XTEST on `DISPLAY`, ahead of uinput and portal input. Install xdotool for this route. Missing or failed commands return an error without a system-wide fallback. Drags send paced intermediate motion and always attempt release after a failed press or motion; verify the drop target afterward.
|
|
20
20
|
|
|
@@ -47,9 +47,12 @@ pi install npm:@agent-sh/computer-use-linux
|
|
|
47
47
|
```
|
|
48
48
|
|
|
49
49
|
When installed through Pi, the package supplies native, dynamically loaded
|
|
50
|
-
`computer_use_linux_*` tools.
|
|
50
|
+
`computer_use_linux_*` tools. When npm skipped the install script, the first
|
|
51
|
+
real tool call downloads the binary. No separate MCP adapter or manual MCP
|
|
51
52
|
configuration is required. Native tools require Pi 0.84.4 or newer; the
|
|
52
|
-
standalone CLI wrapper retains Node.js 18 support.
|
|
53
|
+
standalone CLI wrapper retains Node.js 18 support. Each native tool call times
|
|
54
|
+
out after 60 seconds; set `COMPUTER_USE_LINUX_TIMEOUT_MS` in Pi's environment to
|
|
55
|
+
change it.
|
|
53
56
|
|
|
54
57
|
In Pi codemode, native tools return `{ text, images, isError }`. Pass image
|
|
55
58
|
blocks directly to `image(result.images[0])` and check `isError` before using
|
|
@@ -82,15 +85,17 @@ hermes mcp test computer-use-linux
|
|
|
82
85
|
hermes mcp configure computer-use-linux
|
|
83
86
|
```
|
|
84
87
|
|
|
85
|
-
The generated Hermes config
|
|
88
|
+
The generated Hermes config holds `command`, `args` and `enabled`, plus
|
|
89
|
+
`tools.include` when you pick a subset of tools. Hermes defaults to a 300-second
|
|
90
|
+
tool-call timeout and a 60-second connect timeout; add `timeout` or
|
|
91
|
+
`connect_timeout` under the server to change them.
|
|
86
92
|
|
|
87
93
|
```yaml
|
|
88
94
|
mcp_servers:
|
|
89
95
|
computer-use-linux:
|
|
90
96
|
command: computer-use-linux
|
|
91
97
|
args: ["mcp"]
|
|
92
|
-
|
|
93
|
-
connect_timeout: 30
|
|
98
|
+
enabled: true
|
|
94
99
|
```
|
|
95
100
|
|
|
96
101
|
## What the package installs
|
|
@@ -100,11 +105,14 @@ GitHub release for this package version and verifies the `.sha256` asset before
|
|
|
100
105
|
installing it. It also installs the matching `computer-use-linux-cosmic` helper
|
|
101
106
|
used for COSMIC desktop window targeting and the `computer-use-linux-indicator`
|
|
102
107
|
overlay that shows agent activity on screen (`COMPUTER_USE_LINUX_INDICATOR=0`
|
|
103
|
-
turns it off).
|
|
108
|
+
turns it off). When npm skips the install script (npm 12 does for scripts it was
|
|
109
|
+
not told to allow), the wrapper runs the same download on first launch, with its
|
|
110
|
+
output on stderr. A package directory that cannot be written is refused.
|
|
104
111
|
|
|
105
112
|
If you already built or installed the binary yourself, set
|
|
106
113
|
`COMPUTER_USE_LINUX_BIN=/path/to/computer-use-linux` to make the wrapper use
|
|
107
|
-
that executable instead.
|
|
114
|
+
that executable instead. The Pi extension uses it only when it is an absolute
|
|
115
|
+
path to an executable.
|
|
108
116
|
|
|
109
117
|
## Post-action observation & completion cue
|
|
110
118
|
|
|
@@ -145,7 +153,9 @@ does not yet forward the flag or include the tool in its catalog.
|
|
|
145
153
|
environment to reuse a restore token instead. The first dialog still appears.
|
|
146
154
|
Leave it unset to keep the prompt. The repository README records where the
|
|
147
155
|
token is stored and who can read it.
|
|
148
|
-
- **niri**:
|
|
156
|
+
- **niri**: When the host does not pass `NIRI_SOCKET`, the server reads it from its
|
|
157
|
+
parent processes or the systemd user environment; without it, discovery needs
|
|
158
|
+
an unambiguous socket matching `WAYLAND_DISPLAY`. Window listing and
|
|
149
159
|
exact focus use `niri msg` with a direct IPC fallback. Missing positions or
|
|
150
160
|
unknown/mixed output scaling leave window-relative coordinates unavailable;
|
|
151
161
|
focus the target and inspect a fresh full-screen screenshot before using
|
|
@@ -157,13 +167,13 @@ does not yet forward the flag or include the tool in its catalog.
|
|
|
157
167
|
accept an optional app target (`pid`, `window_id`, `app_id`, `wm_class`,
|
|
158
168
|
`window_title`). With one, an `element_index` or selector from another app's
|
|
159
169
|
`get_app_state` snapshot is refused. When the window backend reports no pid
|
|
160
|
-
for the target window (i3
|
|
161
|
-
for that window. `perform_action` and
|
|
162
|
-
|
|
163
|
-
not the target pid.
|
|
164
|
-
- **Click vs. semantic actions**: Plain left `click` by element index or selector
|
|
165
|
-
|
|
166
|
-
avoiding pointer-coordinate conversion
|
|
170
|
+
for the target window (i3 without `xprop`, COSMIC, other wlroots compositors),
|
|
171
|
+
the snapshot must have been taken for that window. `perform_action` and
|
|
172
|
+
`set_value` do not focus the window; an `element_identifier` is refused only
|
|
173
|
+
when its bus owner is known and is not the target pid.
|
|
174
|
+
- **Click vs. semantic actions**: Plain left `click` by element index or selector invokes the
|
|
175
|
+
element's first AT-SPI action instead of using toolkit bounds when that action
|
|
176
|
+
is `click`, `press`, or `toggle`, avoiding pointer-coordinate conversion. Explicit
|
|
167
177
|
`activate`/`jump` requests belong in `perform_action`, not `click`, even when
|
|
168
178
|
bounds are missing. Explicit
|
|
169
179
|
`x`/`y`, right clicks, and multi-clicks retain pointer semantics. Explicit
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agent-sh/computer-use-linux",
|
|
3
|
-
"version": "0.7.
|
|
3
|
+
"version": "0.7.44",
|
|
4
4
|
"description": "Native Linux desktop control for AI agents (MCP server, Pi tools, and CLI): semantic AT-SPI accessibility trees, screenshots, compositor window targeting, Wayland/X11 input, and an on-screen activity indicator.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "commonjs",
|
|
@@ -30,6 +30,9 @@
|
|
|
30
30
|
"x64",
|
|
31
31
|
"arm64"
|
|
32
32
|
],
|
|
33
|
+
"libc": [
|
|
34
|
+
"glibc"
|
|
35
|
+
],
|
|
33
36
|
"bin": {
|
|
34
37
|
"computer-use-linux": "npm/bin/computer-use-linux.js"
|
|
35
38
|
},
|
|
@@ -57,14 +60,14 @@
|
|
|
57
60
|
"pack:check": "node scripts/check_npm_package.mjs",
|
|
58
61
|
"test:pi": "npm test --prefix pi",
|
|
59
62
|
"typecheck:pi": "npm run typecheck --prefix pi",
|
|
60
|
-
"test:wrapper": "node npm/bin/computer-use-linux.js --help"
|
|
63
|
+
"test:wrapper": "COMPUTER_USE_LINUX_BIN=target/debug/computer-use-linux node npm/bin/computer-use-linux.js --help"
|
|
61
64
|
},
|
|
62
65
|
"engines": {
|
|
63
66
|
"node": ">=18"
|
|
64
67
|
},
|
|
65
68
|
"peerDependencies": {
|
|
66
|
-
"@earendil-works/pi-ai": "
|
|
67
|
-
"@earendil-works/pi-coding-agent": "
|
|
69
|
+
"@earendil-works/pi-ai": ">=0.84.4",
|
|
70
|
+
"@earendil-works/pi-coding-agent": ">=0.84.4",
|
|
68
71
|
"typebox": "*"
|
|
69
72
|
},
|
|
70
73
|
"peerDependenciesMeta": {
|