@colton.padden/pi-agent-desktop 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,5 +1,13 @@
1
1
  # Changelog
2
2
 
3
+ ## [0.1.1] - 2026-11-17
4
+
5
+ - Validate against Pi 1.5.2 and add opt-in real CLI 1.3.1 metadata/help contract checks without GUI actions.
6
+ - Update the external agent-desktop CLI reference/install recommendation to 1.3.1 (unchanged 2.6 envelope).
7
+ - Scope desktop commands to `pi-<Pi session ID>` instead of a fixed `pi` namespace.
8
+ - Enable a labeled presentation-only cursor overlay lazily, without changing headless delivery.
9
+ - Warn if cursor rendering is unavailable and disable the overlay on reload/shutdown.
10
+
3
11
  ## [0.1.0]
4
12
 
5
13
  Initial release of the desktop accessibility extension.
package/README.md CHANGED
@@ -13,14 +13,21 @@ Package: `@colton.padden/pi-agent-desktop`, under the
13
13
  Install the CLI separately; the extension never downloads or installs anything:
14
14
 
15
15
  ```sh
16
- npm install -g agent-desktop@1.3.0
16
+ npm install -g agent-desktop@1.3.1
17
17
  agent-desktop status
18
18
  agent-desktop permissions --guide
19
19
  ```
20
20
 
21
- The adapter was written against the **1.3.0 CLI / 2.6 JSON envelope** source
22
- contract. Tests use fake executables; real desktop behavior is not yet validated.
23
- Other CLI versions are not blocked, but may change behavior or payloads. If npm's
21
+ The reference version is **1.3.1 CLI / 2.6 JSON envelope**, updated from 1.3.0.
22
+ [Upstream 1.3.1](https://github.com/vercel-labs/agent-desktop/compare/v1.3.0...v1.3.1)
23
+ fixes macOS sandbox lifetime and updates Linux's rustix dependency for a security
24
+ advisory; it does not change the adapter's commands or cursor-overlay API.
25
+ The adapter typechecks and its loader integration tests pass against **Pi 1.5.2**.
26
+ An opt-in contract test also checks the real macOS arm64 **1.3.1** binary's version
27
+ and 2.6 envelope, plus command/overlay flags through `--help`, without operating
28
+ apps. This is not end-to-end GUI validation: Slack screenshots/navigation were
29
+ exercised with 1.3.0; cursor rendering reported unavailable. Other CLI versions
30
+ are not blocked, but may change behavior or payloads. If npm's
24
31
  `min-release-age` policy excludes the pinned release, wait for it to become
25
32
  eligible rather than assuming an older CLI has the same contract.
26
33
 
@@ -58,7 +65,19 @@ the extension never requests permissions itself.
58
65
  ## Tool
59
66
 
60
67
  One tool named `desktop`, with explicit operations and no arbitrary argument or
61
- JavaScript escape hatch:
68
+ JavaScript escape hatch. Its namespace is `pi-<Pi session ID>`, obtained from Pi's
69
+ session manager rather than inherited shell state.
70
+
71
+ On the first valid tool call, the extension enables a **presentation-only cursor
72
+ overlay**, labeled with that namespace. It does not enable physical input or take
73
+ focus: actions remain headless unless explicitly given `headed: true`. Cursor
74
+ setup is best-effort (with a separate 5-second deadline); unsupported or unavailable
75
+ renderers produce a warning without blocking desktop commands. Setup is attempted
76
+ once per runtime; `/reload` retries it. Shutdown/reload attempts to disable the
77
+ overlay before deleting private state. No multi-agent mode or concurrent desktop
78
+ delegation is enabled.
79
+
80
+ Available operations:
62
81
 
63
82
  | Command | Inputs |
64
83
  | --- | --- |
@@ -109,8 +128,10 @@ persistence; upstream's `disposition` and verification fields remain intact.
109
128
  external CLI calls, or a human interacting with the same desktop. Avoid
110
129
  concurrent desktop delegation. Do not batch dependent actions using refs that
111
130
  may become stale before they execute.
112
- - Each instance uses a private temporary `AGENT_DESKTOP_HOME` and explicit session
113
- namespace. Inherited agent/session identities are removed. Snapshot refs do not
131
+ - Each instance uses a private temporary `AGENT_DESKTOP_HOME` and explicit
132
+ `pi-<Pi session ID>` namespace. Inherited agent/session identities are removed.
133
+ Separate Pi sessions get separate cursor overlays and snapshot namespaces, not
134
+ separate OS pointers or desktops. Snapshot refs do not
114
135
  survive `/reload`, new/resumed sessions, or process restarts; always resnapshot.
115
136
  - Abort, deadline and output limits terminate the CLI. A cancellation cannot undo
116
137
  an already-delivered action. Upstream helpers may have their own lifecycles.
@@ -166,10 +187,21 @@ installed as runtime dependencies for consumers. Installed dependencies and
166
187
  local package archives are gitignored.
167
188
 
168
189
  Tests cover command construction, stdin/flag isolation, structured errors,
169
- screenshots, truncation/cleanup, serialization/cancellation, subprocess limits,
190
+ screenshots, truncation/cleanup, serialization/cancellation, cursor setup/failure/
191
+ shutdown, Pi session namespacing, subprocess limits,
170
192
  real Pi subfolder-manifest discovery, and the npm tarball allowlist. Integration
171
193
  tests resolve the loader from the pinned development dependency. No model
172
- requests, installed desktop CLI, permissions, or live GUI interaction required.
194
+ requests, installed desktop CLI, permissions, or live GUI interaction required
195
+ for the default suite. The real-CLI contract test is skipped unless explicitly
196
+ configured; it only runs metadata/help commands, not desktop actions:
197
+
198
+ ```sh
199
+ PI_AGENT_DESKTOP_TEST_BIN=/absolute/path/to/agent-desktop npm run check
200
+ ```
201
+
202
+ Install/download the reference CLI separately; this test never installs it or
203
+ changes your global CLI. It asserts CLI 1.3.1 / envelope 2.6 so a future reference
204
+ upgrade requires an explicit compatibility review.
173
205
  The fake executable fixture currently targets macOS/Linux.
174
206
 
175
207
  For an opt-in macOS smoke test after installation: `/reload`, ask Pi to check
@@ -208,7 +240,7 @@ development lockfile are not shipped.
208
240
  After publication, install the matching version with Pi, for example:
209
241
 
210
242
  ```sh
211
- pi install npm:@colton.padden/pi-agent-desktop@0.1.0
243
+ pi install npm:@colton.padden/pi-agent-desktop@0.1.1
212
244
  ```
213
245
 
214
246
  Published code is TypeScript loaded directly by Pi. The package does not bundle
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@colton.padden/pi-agent-desktop",
3
- "version": "0.1.0",
3
+ "version": "0.1.1",
4
4
  "description": "A small Pi extension for native desktop accessibility and computer use through agent-desktop",
5
5
  "type": "module",
6
6
  "license": "MIT",
@@ -40,7 +40,7 @@
40
40
  "typebox": "*"
41
41
  },
42
42
  "devDependencies": {
43
- "@earendil-works/pi-coding-agent": "1.5.0",
43
+ "@earendil-works/pi-coding-agent": "1.5.2",
44
44
  "@types/node": "^24.0.0",
45
45
  "typebox": "1.3.27",
46
46
  "typescript": "5.9.3"
package/src/index.ts CHANGED
@@ -8,8 +8,8 @@ function stringEnum<const T extends readonly string[]>(values: T) {
8
8
  }
9
9
 
10
10
  export default function (pi: ExtensionAPI) {
11
- const runtime = new DesktopRuntime();
12
- pi.on("session_shutdown", () => runtime.close());
11
+ let runtime: DesktopRuntime | undefined;
12
+ pi.on("session_shutdown", () => runtime?.close());
13
13
 
14
14
  pi.registerTool({
15
15
  name: "desktop",
@@ -22,7 +22,8 @@ export default function (pi: ExtensionAPI) {
22
22
  "press (keys, app/windowId); scroll (ref, optional direction/amount); screenshot (app/windowId).",
23
23
  "No raw CLI args, shell, or JavaScript. Text output is capped at 50 KiB/2000 lines;",
24
24
  "overflow JSON is saved privately until reload/shutdown. Screenshots attach PNGs, max 8 MiB.",
25
- "Requires an installed agent-desktop CLI (reference version 1.3.0).",
25
+ "Uses a pi-<session-id> namespace and enables a presentation-only cursor overlay on first use.",
26
+ "Requires an installed agent-desktop CLI (reference version 1.3.1).",
26
27
  ].join(" "),
27
28
  promptSnippet: "Observe and operate native desktop apps using accessibility refs; capture window screenshots",
28
29
  promptGuidelines: [
@@ -51,6 +52,7 @@ export default function (pi: ExtensionAPI) {
51
52
  timeoutMs: Type.Optional(Type.Integer({ minimum: 100, maximum: 60_000, description: "Whole CLI process deadline, default 15000 ms. Does not extend upstream's internal action/traversal deadlines." })),
52
53
  }, { additionalProperties: false }),
53
54
  async execute(_toolCallId, input, signal, _onUpdate, ctx) {
55
+ runtime ??= new DesktopRuntime(ctx.sessionManager.getSessionId());
54
56
  return runtime.execute(input, ctx.cwd, signal);
55
57
  },
56
58
  });
package/src/runtime.ts CHANGED
@@ -36,7 +36,7 @@ type Envelope = {
36
36
  };
37
37
  export type DesktopResult = {
38
38
  content: ({ type: "text"; text: string } | { type: "image"; data: string; mimeType: string })[];
39
- details: { command: string; fullOutputPath?: string; truncated?: boolean };
39
+ details: { command: string; fullOutputPath?: string; truncated?: boolean; cursorWarning?: string };
40
40
  isError?: boolean;
41
41
  };
42
42
  export type ProcessOptions = {
@@ -112,7 +112,8 @@ function required(value: unknown, field: string, allowEmpty = false): asserts va
112
112
  }
113
113
  }
114
114
 
115
- export function buildInvocation(input: DesktopInput): { args: string[]; stdin?: string } {
115
+ export function buildInvocation(input: DesktopInput, session: string): { args: string[]; stdin?: string } {
116
+ if (!/^pi-[A-Za-z0-9_-]+$/.test(session)) throw new Error("Expected a pi-<session-id> namespace");
116
117
  if (!COMMANDS.includes(input.command)) throw new Error("Unsupported desktop command");
117
118
  const allowed = new Set(["command", "timeoutMs", ...FIELDS[input.command]]);
118
119
  for (const [key, value] of Object.entries(input)) {
@@ -154,7 +155,7 @@ export function buildInvocation(input: DesktopInput): { args: string[]; stdin?:
154
155
  if (Buffer.byteLength(input.text) > limit) throw new Error(`text exceeds ${limit} bytes`);
155
156
  }
156
157
 
157
- const args = ["--session=pi", ...(input.headed ? ["--headed"] : []), input.command];
158
+ const args = [`--session=${session}`, ...(input.headed ? ["--headed"] : []), input.command];
158
159
  // '=' prevents a value such as '--headed' from being interpreted as a flag.
159
160
  if (input.app) args.push(`--app=${input.app}`);
160
161
  if (input.windowId) args.push(`--window-id=${input.windowId}`);
@@ -188,11 +189,15 @@ export class DesktopRuntime {
188
189
  private shutdown = new AbortController();
189
190
  private directory?: string;
190
191
  private closePromise?: Promise<void>;
192
+ private cursorOptions?: ProcessOptions;
193
+ private cursorWarning?: string;
194
+ private session: string;
191
195
 
192
196
  private binary: string;
193
197
  private run: RunProcess;
194
198
 
195
- constructor(binary = process.env.PI_AGENT_DESKTOP_BIN || "agent-desktop", run: RunProcess = runProcess) {
199
+ constructor(sessionId: string, binary = process.env.PI_AGENT_DESKTOP_BIN || "agent-desktop", run: RunProcess = runProcess) {
200
+ this.session = `pi-${sessionId}`;
196
201
  this.binary = binary;
197
202
  this.run = run;
198
203
  }
@@ -202,19 +207,36 @@ export class DesktopRuntime {
202
207
  const task = this.tail.then(async () => {
203
208
  try {
204
209
  combined.throwIfAborted();
205
- const invocation = buildInvocation(input);
210
+ const invocation = buildInvocation(input, this.session);
206
211
  this.directory ??= await mkdtemp(join(tmpdir(), "pi-agent-desktop-"));
207
212
  combined.throwIfAborted();
208
213
  const env: NodeJS.ProcessEnv = { ...process.env, AGENT_DESKTOP_HOME: join(this.directory, "state") };
209
214
  // Do not join a caller's snapshot namespace / subagent identity.
210
215
  delete env.AGENT_DESKTOP_SESSION;
211
216
  delete env.AGENT_DESKTOP_AGENT_ID;
217
+ if (!this.cursorOptions) {
218
+ // Presentation only: never opt into headed delivery to draw the overlay.
219
+ // Retain cleanup options even when enable times out after starting a renderer.
220
+ this.cursorOptions = { cwd, env, signal: new AbortController().signal, timeout: 5000, maxBuffer: 64 * 1024 };
221
+ try {
222
+ const cursor = await this.run(this.binary, [
223
+ `--session=${this.session}`, "cursor-overlay", "enable", `--label=${this.session}`,
224
+ ], { ...this.cursorOptions, signal: combined });
225
+ const envelope = JSON.parse(cursor.stdout) as Envelope;
226
+ if (cursor.code !== 0 || cursor.stopped || !envelope?.ok || envelope.command !== "cursor-overlay" || envelope.data?.rendered !== true) {
227
+ this.cursorWarning = "Cursor overlay unavailable; desktop delivery policy is unchanged. Reload to retry cursor setup.";
228
+ }
229
+ } catch {
230
+ this.cursorWarning = "Cursor overlay unavailable; desktop delivery policy is unchanged. Reload to retry cursor setup.";
231
+ }
232
+ }
233
+ combined.throwIfAborted();
212
234
  const result = await this.run(this.binary, invocation.args, {
213
235
  cwd, env, signal: combined, timeout: input.timeoutMs ?? 15_000,
214
236
  maxBuffer: input.command === "screenshot" ? 12 * 1024 * 1024 : 2 * 1024 * 1024,
215
237
  stdin: invocation.stdin,
216
238
  });
217
- if (result.missing) throw new Error("agent-desktop not found. Install agent-desktop@1.3.0 or set PI_AGENT_DESKTOP_BIN to an executable path; then retry status.");
239
+ if (result.missing) throw new Error("agent-desktop not found. Install agent-desktop@1.3.1 or set PI_AGENT_DESKTOP_BIN to an executable path; then retry status.");
218
240
  if (result.stopped || combined.aborted) {
219
241
  throw new Error("Desktop command cancelled, timed out, or exceeded its output limit. Delivery may be uncertain: observe the UI before repeating any action.");
220
242
  }
@@ -245,6 +267,10 @@ export class DesktopRuntime {
245
267
  }
246
268
  if (result.code !== 0 && envelope.ok) text += `\n[CLI exited with code ${result.code}; treat this result as a failure.]`;
247
269
  content.unshift({ type: "text", text });
270
+ if (this.cursorWarning) {
271
+ details.cursorWarning = this.cursorWarning;
272
+ content.push({ type: "text", text: `[${this.cursorWarning}]` });
273
+ }
248
274
  return { content, details, isError: !envelope.ok || result.code !== 0 };
249
275
  } catch (error) {
250
276
  // Never stringify subprocess exceptions: they can include command text.
@@ -262,7 +288,14 @@ export class DesktopRuntime {
262
288
  this.shutdown.abort();
263
289
  return this.closePromise ??= (async () => {
264
290
  await this.tail;
265
- if (this.directory) await rm(this.directory, { recursive: true, force: true });
291
+ try {
292
+ if (this.cursorOptions) {
293
+ // Shutdown has aborted tool work; cleanup needs its own bounded signal.
294
+ await this.run(this.binary, [`--session=${this.session}`, "cursor-overlay", "disable"], this.cursorOptions);
295
+ }
296
+ } finally {
297
+ if (this.directory) await rm(this.directory, { recursive: true, force: true });
298
+ }
266
299
  })();
267
300
  }
268
301
  }