@colton.padden/pi-agent-desktop 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +42 -10
- package/package.json +2 -2
- package/src/index.ts +5 -3
- package/src/runtime.ts +40 -7
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.1.1] - 2026-11-17
|
|
4
|
+
|
|
5
|
+
- Validate against Pi 1.5.2 and add opt-in real CLI 1.3.1 metadata/help contract checks without GUI actions.
|
|
6
|
+
- Update the external agent-desktop CLI reference/install recommendation to 1.3.1 (unchanged 2.6 envelope).
|
|
7
|
+
- Scope desktop commands to `pi-<Pi session ID>` instead of a fixed `pi` namespace.
|
|
8
|
+
- Enable a labeled presentation-only cursor overlay lazily, without changing headless delivery.
|
|
9
|
+
- Warn if cursor rendering is unavailable and disable the overlay on reload/shutdown.
|
|
10
|
+
|
|
3
11
|
## [0.1.0]
|
|
4
12
|
|
|
5
13
|
Initial release of the desktop accessibility extension.
|
package/README.md
CHANGED
|
@@ -13,14 +13,21 @@ Package: `@colton.padden/pi-agent-desktop`, under the
|
|
|
13
13
|
Install the CLI separately; the extension never downloads or installs anything:
|
|
14
14
|
|
|
15
15
|
```sh
|
|
16
|
-
npm install -g agent-desktop@1.3.
|
|
16
|
+
npm install -g agent-desktop@1.3.1
|
|
17
17
|
agent-desktop status
|
|
18
18
|
agent-desktop permissions --guide
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
The
|
|
22
|
-
|
|
23
|
-
|
|
21
|
+
The reference version is **1.3.1 CLI / 2.6 JSON envelope**, updated from 1.3.0.
|
|
22
|
+
[Upstream 1.3.1](https://github.com/vercel-labs/agent-desktop/compare/v1.3.0...v1.3.1)
|
|
23
|
+
fixes macOS sandbox lifetime and updates Linux's rustix dependency for a security
|
|
24
|
+
advisory; it does not change the adapter's commands or cursor-overlay API.
|
|
25
|
+
The adapter typechecks and its loader integration tests pass against **Pi 1.5.2**.
|
|
26
|
+
An opt-in contract test also checks the real macOS arm64 **1.3.1** binary's version
|
|
27
|
+
and 2.6 envelope, plus command/overlay flags through `--help`, without operating
|
|
28
|
+
apps. This is not end-to-end GUI validation: Slack screenshots/navigation were
|
|
29
|
+
exercised with 1.3.0; cursor rendering reported unavailable. Other CLI versions
|
|
30
|
+
are not blocked, but may change behavior or payloads. If npm's
|
|
24
31
|
`min-release-age` policy excludes the pinned release, wait for it to become
|
|
25
32
|
eligible rather than assuming an older CLI has the same contract.
|
|
26
33
|
|
|
@@ -58,7 +65,19 @@ the extension never requests permissions itself.
|
|
|
58
65
|
## Tool
|
|
59
66
|
|
|
60
67
|
One tool named `desktop`, with explicit operations and no arbitrary argument or
|
|
61
|
-
JavaScript escape hatch
|
|
68
|
+
JavaScript escape hatch. Its namespace is `pi-<Pi session ID>`, obtained from Pi's
|
|
69
|
+
session manager rather than inherited shell state.
|
|
70
|
+
|
|
71
|
+
On the first valid tool call, the extension enables a **presentation-only cursor
|
|
72
|
+
overlay**, labeled with that namespace. It does not enable physical input or take
|
|
73
|
+
focus: actions remain headless unless explicitly given `headed: true`. Cursor
|
|
74
|
+
setup is best-effort (with a separate 5-second deadline); unsupported or unavailable
|
|
75
|
+
renderers produce a warning without blocking desktop commands. Setup is attempted
|
|
76
|
+
once per runtime; `/reload` retries it. Shutdown/reload attempts to disable the
|
|
77
|
+
overlay before deleting private state. No multi-agent mode or concurrent desktop
|
|
78
|
+
delegation is enabled.
|
|
79
|
+
|
|
80
|
+
Available operations:
|
|
62
81
|
|
|
63
82
|
| Command | Inputs |
|
|
64
83
|
| --- | --- |
|
|
@@ -109,8 +128,10 @@ persistence; upstream's `disposition` and verification fields remain intact.
|
|
|
109
128
|
external CLI calls, or a human interacting with the same desktop. Avoid
|
|
110
129
|
concurrent desktop delegation. Do not batch dependent actions using refs that
|
|
111
130
|
may become stale before they execute.
|
|
112
|
-
- Each instance uses a private temporary `AGENT_DESKTOP_HOME` and explicit
|
|
113
|
-
namespace. Inherited agent/session identities are removed.
|
|
131
|
+
- Each instance uses a private temporary `AGENT_DESKTOP_HOME` and explicit
|
|
132
|
+
`pi-<Pi session ID>` namespace. Inherited agent/session identities are removed.
|
|
133
|
+
Separate Pi sessions get separate cursor overlays and snapshot namespaces, not
|
|
134
|
+
separate OS pointers or desktops. Snapshot refs do not
|
|
114
135
|
survive `/reload`, new/resumed sessions, or process restarts; always resnapshot.
|
|
115
136
|
- Abort, deadline and output limits terminate the CLI. A cancellation cannot undo
|
|
116
137
|
an already-delivered action. Upstream helpers may have their own lifecycles.
|
|
@@ -166,10 +187,21 @@ installed as runtime dependencies for consumers. Installed dependencies and
|
|
|
166
187
|
local package archives are gitignored.
|
|
167
188
|
|
|
168
189
|
Tests cover command construction, stdin/flag isolation, structured errors,
|
|
169
|
-
screenshots, truncation/cleanup, serialization/cancellation,
|
|
190
|
+
screenshots, truncation/cleanup, serialization/cancellation, cursor setup/failure/
|
|
191
|
+
shutdown, Pi session namespacing, subprocess limits,
|
|
170
192
|
real Pi subfolder-manifest discovery, and the npm tarball allowlist. Integration
|
|
171
193
|
tests resolve the loader from the pinned development dependency. No model
|
|
172
|
-
requests, installed desktop CLI, permissions, or live GUI interaction required
|
|
194
|
+
requests, installed desktop CLI, permissions, or live GUI interaction required
|
|
195
|
+
for the default suite. The real-CLI contract test is skipped unless explicitly
|
|
196
|
+
configured; it only runs metadata/help commands, not desktop actions:
|
|
197
|
+
|
|
198
|
+
```sh
|
|
199
|
+
PI_AGENT_DESKTOP_TEST_BIN=/absolute/path/to/agent-desktop npm run check
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
Install/download the reference CLI separately; this test never installs it or
|
|
203
|
+
changes your global CLI. It asserts CLI 1.3.1 / envelope 2.6 so a future reference
|
|
204
|
+
upgrade requires an explicit compatibility review.
|
|
173
205
|
The fake executable fixture currently targets macOS/Linux.
|
|
174
206
|
|
|
175
207
|
For an opt-in macOS smoke test after installation: `/reload`, ask Pi to check
|
|
@@ -208,7 +240,7 @@ development lockfile are not shipped.
|
|
|
208
240
|
After publication, install the matching version with Pi, for example:
|
|
209
241
|
|
|
210
242
|
```sh
|
|
211
|
-
pi install npm:@colton.padden/pi-agent-desktop@0.1.
|
|
243
|
+
pi install npm:@colton.padden/pi-agent-desktop@0.1.1
|
|
212
244
|
```
|
|
213
245
|
|
|
214
246
|
Published code is TypeScript loaded directly by Pi. The package does not bundle
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@colton.padden/pi-agent-desktop",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.1",
|
|
4
4
|
"description": "A small Pi extension for native desktop accessibility and computer use through agent-desktop",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"license": "MIT",
|
|
@@ -40,7 +40,7 @@
|
|
|
40
40
|
"typebox": "*"
|
|
41
41
|
},
|
|
42
42
|
"devDependencies": {
|
|
43
|
-
"@earendil-works/pi-coding-agent": "1.5.
|
|
43
|
+
"@earendil-works/pi-coding-agent": "1.5.2",
|
|
44
44
|
"@types/node": "^24.0.0",
|
|
45
45
|
"typebox": "1.3.27",
|
|
46
46
|
"typescript": "5.9.3"
|
package/src/index.ts
CHANGED
|
@@ -8,8 +8,8 @@ function stringEnum<const T extends readonly string[]>(values: T) {
|
|
|
8
8
|
}
|
|
9
9
|
|
|
10
10
|
export default function (pi: ExtensionAPI) {
|
|
11
|
-
|
|
12
|
-
pi.on("session_shutdown", () => runtime
|
|
11
|
+
let runtime: DesktopRuntime | undefined;
|
|
12
|
+
pi.on("session_shutdown", () => runtime?.close());
|
|
13
13
|
|
|
14
14
|
pi.registerTool({
|
|
15
15
|
name: "desktop",
|
|
@@ -22,7 +22,8 @@ export default function (pi: ExtensionAPI) {
|
|
|
22
22
|
"press (keys, app/windowId); scroll (ref, optional direction/amount); screenshot (app/windowId).",
|
|
23
23
|
"No raw CLI args, shell, or JavaScript. Text output is capped at 50 KiB/2000 lines;",
|
|
24
24
|
"overflow JSON is saved privately until reload/shutdown. Screenshots attach PNGs, max 8 MiB.",
|
|
25
|
-
"
|
|
25
|
+
"Uses a pi-<session-id> namespace and enables a presentation-only cursor overlay on first use.",
|
|
26
|
+
"Requires an installed agent-desktop CLI (reference version 1.3.1).",
|
|
26
27
|
].join(" "),
|
|
27
28
|
promptSnippet: "Observe and operate native desktop apps using accessibility refs; capture window screenshots",
|
|
28
29
|
promptGuidelines: [
|
|
@@ -51,6 +52,7 @@ export default function (pi: ExtensionAPI) {
|
|
|
51
52
|
timeoutMs: Type.Optional(Type.Integer({ minimum: 100, maximum: 60_000, description: "Whole CLI process deadline, default 15000 ms. Does not extend upstream's internal action/traversal deadlines." })),
|
|
52
53
|
}, { additionalProperties: false }),
|
|
53
54
|
async execute(_toolCallId, input, signal, _onUpdate, ctx) {
|
|
55
|
+
runtime ??= new DesktopRuntime(ctx.sessionManager.getSessionId());
|
|
54
56
|
return runtime.execute(input, ctx.cwd, signal);
|
|
55
57
|
},
|
|
56
58
|
});
|
package/src/runtime.ts
CHANGED
|
@@ -36,7 +36,7 @@ type Envelope = {
|
|
|
36
36
|
};
|
|
37
37
|
export type DesktopResult = {
|
|
38
38
|
content: ({ type: "text"; text: string } | { type: "image"; data: string; mimeType: string })[];
|
|
39
|
-
details: { command: string; fullOutputPath?: string; truncated?: boolean };
|
|
39
|
+
details: { command: string; fullOutputPath?: string; truncated?: boolean; cursorWarning?: string };
|
|
40
40
|
isError?: boolean;
|
|
41
41
|
};
|
|
42
42
|
export type ProcessOptions = {
|
|
@@ -112,7 +112,8 @@ function required(value: unknown, field: string, allowEmpty = false): asserts va
|
|
|
112
112
|
}
|
|
113
113
|
}
|
|
114
114
|
|
|
115
|
-
export function buildInvocation(input: DesktopInput): { args: string[]; stdin?: string } {
|
|
115
|
+
export function buildInvocation(input: DesktopInput, session: string): { args: string[]; stdin?: string } {
|
|
116
|
+
if (!/^pi-[A-Za-z0-9_-]+$/.test(session)) throw new Error("Expected a pi-<session-id> namespace");
|
|
116
117
|
if (!COMMANDS.includes(input.command)) throw new Error("Unsupported desktop command");
|
|
117
118
|
const allowed = new Set(["command", "timeoutMs", ...FIELDS[input.command]]);
|
|
118
119
|
for (const [key, value] of Object.entries(input)) {
|
|
@@ -154,7 +155,7 @@ export function buildInvocation(input: DesktopInput): { args: string[]; stdin?:
|
|
|
154
155
|
if (Buffer.byteLength(input.text) > limit) throw new Error(`text exceeds ${limit} bytes`);
|
|
155
156
|
}
|
|
156
157
|
|
|
157
|
-
const args = [
|
|
158
|
+
const args = [`--session=${session}`, ...(input.headed ? ["--headed"] : []), input.command];
|
|
158
159
|
// '=' prevents a value such as '--headed' from being interpreted as a flag.
|
|
159
160
|
if (input.app) args.push(`--app=${input.app}`);
|
|
160
161
|
if (input.windowId) args.push(`--window-id=${input.windowId}`);
|
|
@@ -188,11 +189,15 @@ export class DesktopRuntime {
|
|
|
188
189
|
private shutdown = new AbortController();
|
|
189
190
|
private directory?: string;
|
|
190
191
|
private closePromise?: Promise<void>;
|
|
192
|
+
private cursorOptions?: ProcessOptions;
|
|
193
|
+
private cursorWarning?: string;
|
|
194
|
+
private session: string;
|
|
191
195
|
|
|
192
196
|
private binary: string;
|
|
193
197
|
private run: RunProcess;
|
|
194
198
|
|
|
195
|
-
constructor(binary = process.env.PI_AGENT_DESKTOP_BIN || "agent-desktop", run: RunProcess = runProcess) {
|
|
199
|
+
constructor(sessionId: string, binary = process.env.PI_AGENT_DESKTOP_BIN || "agent-desktop", run: RunProcess = runProcess) {
|
|
200
|
+
this.session = `pi-${sessionId}`;
|
|
196
201
|
this.binary = binary;
|
|
197
202
|
this.run = run;
|
|
198
203
|
}
|
|
@@ -202,19 +207,36 @@ export class DesktopRuntime {
|
|
|
202
207
|
const task = this.tail.then(async () => {
|
|
203
208
|
try {
|
|
204
209
|
combined.throwIfAborted();
|
|
205
|
-
const invocation = buildInvocation(input);
|
|
210
|
+
const invocation = buildInvocation(input, this.session);
|
|
206
211
|
this.directory ??= await mkdtemp(join(tmpdir(), "pi-agent-desktop-"));
|
|
207
212
|
combined.throwIfAborted();
|
|
208
213
|
const env: NodeJS.ProcessEnv = { ...process.env, AGENT_DESKTOP_HOME: join(this.directory, "state") };
|
|
209
214
|
// Do not join a caller's snapshot namespace / subagent identity.
|
|
210
215
|
delete env.AGENT_DESKTOP_SESSION;
|
|
211
216
|
delete env.AGENT_DESKTOP_AGENT_ID;
|
|
217
|
+
if (!this.cursorOptions) {
|
|
218
|
+
// Presentation only: never opt into headed delivery to draw the overlay.
|
|
219
|
+
// Retain cleanup options even when enable times out after starting a renderer.
|
|
220
|
+
this.cursorOptions = { cwd, env, signal: new AbortController().signal, timeout: 5000, maxBuffer: 64 * 1024 };
|
|
221
|
+
try {
|
|
222
|
+
const cursor = await this.run(this.binary, [
|
|
223
|
+
`--session=${this.session}`, "cursor-overlay", "enable", `--label=${this.session}`,
|
|
224
|
+
], { ...this.cursorOptions, signal: combined });
|
|
225
|
+
const envelope = JSON.parse(cursor.stdout) as Envelope;
|
|
226
|
+
if (cursor.code !== 0 || cursor.stopped || !envelope?.ok || envelope.command !== "cursor-overlay" || envelope.data?.rendered !== true) {
|
|
227
|
+
this.cursorWarning = "Cursor overlay unavailable; desktop delivery policy is unchanged. Reload to retry cursor setup.";
|
|
228
|
+
}
|
|
229
|
+
} catch {
|
|
230
|
+
this.cursorWarning = "Cursor overlay unavailable; desktop delivery policy is unchanged. Reload to retry cursor setup.";
|
|
231
|
+
}
|
|
232
|
+
}
|
|
233
|
+
combined.throwIfAborted();
|
|
212
234
|
const result = await this.run(this.binary, invocation.args, {
|
|
213
235
|
cwd, env, signal: combined, timeout: input.timeoutMs ?? 15_000,
|
|
214
236
|
maxBuffer: input.command === "screenshot" ? 12 * 1024 * 1024 : 2 * 1024 * 1024,
|
|
215
237
|
stdin: invocation.stdin,
|
|
216
238
|
});
|
|
217
|
-
if (result.missing) throw new Error("agent-desktop not found. Install agent-desktop@1.3.
|
|
239
|
+
if (result.missing) throw new Error("agent-desktop not found. Install agent-desktop@1.3.1 or set PI_AGENT_DESKTOP_BIN to an executable path; then retry status.");
|
|
218
240
|
if (result.stopped || combined.aborted) {
|
|
219
241
|
throw new Error("Desktop command cancelled, timed out, or exceeded its output limit. Delivery may be uncertain: observe the UI before repeating any action.");
|
|
220
242
|
}
|
|
@@ -245,6 +267,10 @@ export class DesktopRuntime {
|
|
|
245
267
|
}
|
|
246
268
|
if (result.code !== 0 && envelope.ok) text += `\n[CLI exited with code ${result.code}; treat this result as a failure.]`;
|
|
247
269
|
content.unshift({ type: "text", text });
|
|
270
|
+
if (this.cursorWarning) {
|
|
271
|
+
details.cursorWarning = this.cursorWarning;
|
|
272
|
+
content.push({ type: "text", text: `[${this.cursorWarning}]` });
|
|
273
|
+
}
|
|
248
274
|
return { content, details, isError: !envelope.ok || result.code !== 0 };
|
|
249
275
|
} catch (error) {
|
|
250
276
|
// Never stringify subprocess exceptions: they can include command text.
|
|
@@ -262,7 +288,14 @@ export class DesktopRuntime {
|
|
|
262
288
|
this.shutdown.abort();
|
|
263
289
|
return this.closePromise ??= (async () => {
|
|
264
290
|
await this.tail;
|
|
265
|
-
|
|
291
|
+
try {
|
|
292
|
+
if (this.cursorOptions) {
|
|
293
|
+
// Shutdown has aborted tool work; cleanup needs its own bounded signal.
|
|
294
|
+
await this.run(this.binary, [`--session=${this.session}`, "cursor-overlay", "disable"], this.cursorOptions);
|
|
295
|
+
}
|
|
296
|
+
} finally {
|
|
297
|
+
if (this.directory) await rm(this.directory, { recursive: true, force: true });
|
|
298
|
+
}
|
|
266
299
|
})();
|
|
267
300
|
}
|
|
268
301
|
}
|