pi-agent-browser-native 0.2.31 → 0.2.33

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (66) hide show
  1. package/CHANGELOG.md +35 -0
  2. package/README.md +64 -18
  3. package/docs/ARCHITECTURE.md +13 -10
  4. package/docs/COMMAND_REFERENCE.md +71 -16
  5. package/docs/ELECTRON.md +387 -0
  6. package/docs/RELEASE.md +34 -4
  7. package/docs/REQUIREMENTS.md +5 -3
  8. package/docs/SUPPORT_MATRIX.md +36 -21
  9. package/docs/TOOL_CONTRACT.md +198 -40
  10. package/extensions/agent-browser/index.ts +1585 -3486
  11. package/extensions/agent-browser/lib/electron/cleanup.ts +287 -0
  12. package/extensions/agent-browser/lib/electron/discovery.ts +717 -0
  13. package/extensions/agent-browser/lib/electron/launch.ts +553 -0
  14. package/extensions/agent-browser/lib/input-modes/electron.ts +170 -0
  15. package/extensions/agent-browser/lib/input-modes/job.ts +203 -0
  16. package/extensions/agent-browser/lib/input-modes/lookups.ts +447 -0
  17. package/extensions/agent-browser/lib/input-modes/params.ts +188 -0
  18. package/extensions/agent-browser/lib/input-modes/semantic-action.ts +107 -0
  19. package/extensions/agent-browser/lib/input-modes/shared.ts +46 -0
  20. package/extensions/agent-browser/lib/input-modes/types.ts +221 -0
  21. package/extensions/agent-browser/lib/input-modes.ts +41 -0
  22. package/extensions/agent-browser/lib/orchestration/browser-run/diagnostics.ts +696 -0
  23. package/extensions/agent-browser/lib/orchestration/browser-run/final-result.ts +450 -0
  24. package/extensions/agent-browser/lib/orchestration/browser-run/index.ts +46 -0
  25. package/extensions/agent-browser/lib/orchestration/browser-run/prepare.ts +711 -0
  26. package/extensions/agent-browser/lib/orchestration/browser-run/process-output.ts +386 -0
  27. package/extensions/agent-browser/lib/orchestration/browser-run/session-state.ts +868 -0
  28. package/extensions/agent-browser/lib/orchestration/browser-run/types.ts +476 -0
  29. package/extensions/agent-browser/lib/orchestration/browser-run.ts +1 -0
  30. package/extensions/agent-browser/lib/orchestration/input-plan.ts +338 -0
  31. package/extensions/agent-browser/lib/playbook.ts +15 -13
  32. package/extensions/agent-browser/lib/process.ts +106 -4
  33. package/extensions/agent-browser/lib/results/action-recommendations.ts +269 -0
  34. package/extensions/agent-browser/lib/results/artifact-manifest.ts +114 -0
  35. package/extensions/agent-browser/lib/results/artifact-state.ts +13 -0
  36. package/extensions/agent-browser/lib/results/categories.ts +106 -0
  37. package/extensions/agent-browser/lib/results/contracts.ts +220 -0
  38. package/extensions/agent-browser/lib/results/editable-ref-evidence.ts +72 -0
  39. package/extensions/agent-browser/lib/results/envelope.ts +2 -1
  40. package/extensions/agent-browser/lib/results/network.ts +64 -0
  41. package/extensions/agent-browser/lib/results/next-actions.ts +117 -0
  42. package/extensions/agent-browser/lib/results/presentation/artifacts.ts +506 -0
  43. package/extensions/agent-browser/lib/results/presentation/batch.ts +355 -0
  44. package/extensions/agent-browser/lib/results/presentation/common.ts +53 -0
  45. package/extensions/agent-browser/lib/results/presentation/content.ts +36 -0
  46. package/extensions/agent-browser/lib/results/presentation/diagnostics.ts +730 -0
  47. package/extensions/agent-browser/lib/results/presentation/errors.ts +125 -0
  48. package/extensions/agent-browser/lib/results/presentation/large-output.ts +182 -0
  49. package/extensions/agent-browser/lib/results/presentation/navigation.ts +216 -0
  50. package/extensions/agent-browser/lib/results/presentation/registry.ts +154 -0
  51. package/extensions/agent-browser/lib/results/presentation/skills.ts +143 -0
  52. package/extensions/agent-browser/lib/results/presentation.ts +87 -2369
  53. package/extensions/agent-browser/lib/results/recovery-actions.ts +139 -0
  54. package/extensions/agent-browser/lib/results/recovery-next-actions.ts +71 -0
  55. package/extensions/agent-browser/lib/results/selector-recovery.ts +312 -0
  56. package/extensions/agent-browser/lib/results/shared.ts +17 -701
  57. package/extensions/agent-browser/lib/results/snapshot-high-value-controls.ts +262 -0
  58. package/extensions/agent-browser/lib/results/snapshot-refs.ts +100 -0
  59. package/extensions/agent-browser/lib/results/snapshot-segments.ts +366 -0
  60. package/extensions/agent-browser/lib/results/snapshot-spill.ts +63 -0
  61. package/extensions/agent-browser/lib/results/snapshot.ts +37 -489
  62. package/extensions/agent-browser/lib/results/text.ts +40 -0
  63. package/extensions/agent-browser/lib/results.ts +16 -5
  64. package/extensions/agent-browser/lib/session-page-state.ts +486 -0
  65. package/extensions/agent-browser/lib/temp.ts +26 -0
  66. package/package.json +6 -4
@@ -0,0 +1,387 @@
1
+ # Electron desktop apps
2
+
3
+ Related docs:
4
+ - [`../README.md`](../README.md)
5
+ - [`../AGENTS.md`](../AGENTS.md) — maintainer verification (`npm run verify`, lifecycle), Pi `tmux` smoke expectations, and upstream rebaselining
6
+ - [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md) — full `electron` and `qa.attached` field contracts
7
+ - [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md) — workflow snippets in the broader native command surface
8
+ - [`ARCHITECTURE.md`](ARCHITECTURE.md) — wrapper design and the closed `RQ-0068` recipe-layer decision
9
+ - [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) — `RQ-0096` Electron support row and verification gates
10
+
11
+ ## Purpose
12
+
13
+ This guide is the entry point for using `pi-agent-browser-native` against desktop **Electron** applications. The wrapper exposes a top-level `electron` shorthand that owns the awkward discover → launch → attach → probe → cleanup sequence so agents do not hand-build `--remote-debugging-port` argv, poll `DevToolsActivePort`, and `kill` profile directories. After attach, the rest of the native `agent_browser` surface (`snapshot`, `find`, `click`, `fill`, `get`, `eval --stdin`, `batch`, `qa.attached`, and similar) works the same way it does against a web page.
14
+
15
+ This document is structured for users, not implementers. Field-level rules live in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#electron); this guide focuses on **when** and **how** to use them, and on the safety and ownership boundary the wrapper enforces.
16
+
17
+ ## Who this is for
18
+
19
+ - **Pi users** who want an agent to operate a local Electron app the same way it operates a web page.
20
+ - **Coding agents** that need a low-context lifecycle for desktop apps such as VS Code, Cursor, Obsidian, Slack, or any app built on Electron, without re-implementing the CDP attach dance every session.
21
+ - **Maintainers and reviewers** validating the wrapper's Electron behavior before release; verification evidence lives under `RQ-0096` in [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md).
22
+
23
+ It is **not** an upstream `agent-browser` reference and it does **not** replace the canonical [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#electron) for exact field semantics, validation rules, or failure categories.
24
+
25
+ ## Mental model
26
+
27
+ ```
28
+ electron.list → discover Electron apps (host-only; no upstream spawn)
29
+ electron.launch → launch a wrapper-owned isolated app, attach via CDP, hand off (snapshot|tabs|connect)
30
+ electron.status → liveness, debug-port, and target inspection (read-only)
31
+ electron.probe → compact one-call state read (title/url/focus/tabs/snapshot)
32
+ electron.cleanup → close managed session, stop the tracked process, remove the temp profile
33
+ qa.attached → smoke check against the currently attached session (no URL)
34
+ ```
35
+
36
+ Two ownership modes coexist:
37
+
38
+ 1. **Wrapper-owned launches** — `electron.launch` starts a brand-new app process with an **isolated temporary user-data-dir** and an **OS-chosen debug port**. The wrapper records a `launchId` for every such launch and `electron.cleanup` only operates on those `launchId`s.
39
+ 2. **Manually launched apps** — you start the Electron app yourself (for example with `open -a Slack --args --remote-debugging-port=9222 --remote-allow-origins='*'`), then attach with `{ "args": ["connect", "9222"], "sessionMode": "fresh" }`. The wrapper does not own that process; **you** are responsible for shutting it down and cleaning its profile.
40
+
41
+ Choosing between the two is a real decision, not a stylistic one. See [Wrapper-owned vs manually launched](#wrapper-owned-vs-manually-launched).
42
+
43
+ ## Quick start
44
+
45
+ Discover the app, launch with the default snapshot handoff, work with current refs, then clean up:
46
+
47
+ ```json
48
+ { "electron": { "action": "list", "query": "code" } }
49
+ { "electron": { "action": "launch", "appName": "Visual Studio Code", "handoff": "snapshot" } }
50
+ { "args": ["snapshot", "-i"] }
51
+ { "electron": { "action": "probe", "timeoutMs": 5000 } }
52
+ { "electron": { "action": "cleanup", "launchId": "electron-…" } }
53
+ ```
54
+
55
+ The launch result carries both a `launchId` (used by `status`/`probe`/`cleanup`) and an attached `sessionName` (used by browser-style `snapshot`/`tab`/`click`/`find` calls). Read both from `details.electron.launch` and `details.electron.identifiers`. With default implicit session reuse, the quick-start `args: ["snapshot", "-i"]` line uses that attached session without an extra `--session` argument; pass `--session` explicitly when you target a named upstream session instead.
56
+
57
+ For a quick "is the app actually showing what we expect?" smoke check after attach:
58
+
59
+ ```json
60
+ { "qa": { "attached": true, "expectedText": "Explorer", "screenshotPath": ".dogfood/electron.png" } }
61
+ ```
62
+
63
+ `qa.attached` runs against the **current managed session** without opening a URL, so it works for any attached app — wrapper-owned or manually launched.
64
+
65
+ ## Wrapper-owned vs manually launched
66
+
67
+ Pick the mode that matches the **state you need**.
68
+
69
+ | | `electron.launch` (wrapper-owned) | `args: ["connect", …]` (manual host launch) |
70
+ |---|---|---|
71
+ | Profile | Isolated temporary `userDataDir` | The app's normal profile (your real signed-in state) |
72
+ | Debug port | OS-chosen via `--remote-debugging-port=0` and `DevToolsActivePort` | Caller-supplied port (for example `9222`) |
73
+ | Signed-in state | **No** — first-run or empty profile | **Yes** — whatever is in the launched profile |
74
+ | Already-running app | Cannot attach to it | Required (or relaunch yourself with a debug port) |
75
+ | Lifecycle ownership | Wrapper owns shutdown and profile cleanup | **You** own shutdown and profile cleanup |
76
+ | When to use | Anything you can do against a fresh app: tooling, UX flows, scripted local QA, exploring panels, packaged debugging | Tasks that explicitly need the user's signed-in Slack/Obsidian/VS Code state |
77
+ | How to clean up | `electron.cleanup` with the returned `launchId` | Close the app yourself; do **not** call `electron.cleanup` |
78
+
79
+ ### Manual host-launch pattern
80
+
81
+ When the explicit goal is the user's signed-in local app state and the app is not already running:
82
+
83
+ ```bash
84
+ # macOS example
85
+ open -a Slack --args --remote-debugging-port=9222 --remote-allow-origins='*'
86
+ ```
87
+
88
+ Then attach and choose a ready target before using refs:
89
+
90
+ ```json
91
+ { "args": ["connect", "9222"], "sessionMode": "fresh" }
92
+ { "args": ["tab", "list"] }
93
+ { "args": ["tab", "t2"] }
94
+ { "args": ["snapshot", "-i"] }
95
+ { "qa": { "attached": true, "expectedText": "Channels" } }
96
+ ```
97
+
98
+ A successful `connect` means the CDP endpoint accepted the session; it does **not** prove the app has an active rendered page yet. Prefer `details.nextActions` when present: `list-connected-session-tabs` inspects the attached session targets. After that read-only list, select or confirm a stable `t<N>` target and run `snapshot -i` explicitly before trusting refs. If the first `snapshot -i` says `No active page`, follow `list-tabs-after-no-active-page`. If it returns no useful refs without that error, manually run `tab list`, select a stable `t<N>` id for the app surface, then retry a condition wait or `snapshot -i` on that selected target.
99
+
100
+ If the app is already running without a debug port, ask before relaunching it — relaunching may lose unsaved state and Electron's single-instance behavior will silently drop a second invocation's `--remote-debugging-port` flag.
101
+
102
+ ## Readiness and waits
103
+
104
+ Use this ladder for desktop-host readiness instead of blind sleep loops:
105
+
106
+ 1. Prefer a real condition when one exists: `wait --text`, `wait --url`, `wait --fn`, `wait --load <state>`, or `wait --download`.
107
+ 2. After raw `connect`, inspect targets with `tab list`, select the stable `tab t<N>` app surface, then use a condition wait or `snapshot -i` on that selected surface.
108
+ 3. After wrapper-owned `electron.launch`, use `electron.probe` or `electron.status` when launch health, debug-port liveness, or target mismatch matters.
109
+ 4. Use `qa.attached` when the readiness check can be expressed as expected text or selector plus diagnostics against the current managed session.
110
+ 5. Use fixed waits only as a last resort, and keep each fixed wait below the wrapper IPC budget. `wait 30000` is intentionally blocked; use `25000` ms or less per call when a fixed wait is unavoidable.
111
+ 6. Treat a fixed-wait payload such as `"waited":"timeout"` as elapsed time, not proof that the host finished. Verify with an observed condition, fresh `snapshot -i`, or screenshot before continuing.
112
+
113
+ This project is not adding a first-class host-idle primitive yet. Revisit that only if repeated desktop smokes show that condition waits, `qa.attached`, `electron.probe`, snapshots, and screenshots cannot cover the workflow.
114
+
115
+ ## Action reference
116
+
117
+ The exact field schemas, validation rules, and `details.*` payload shapes live in [`TOOL_CONTRACT.md#electron`](TOOL_CONTRACT.md#electron). This section is a usage-oriented overview.
118
+
119
+ ### `electron.list` — discover apps
120
+
121
+ Host-only scan; does not spawn upstream `agent-browser`. macOS (`/Applications/*.app`, `~/Applications/*.app`) and Linux (`.desktop` launchers under standard XDG, Flatpak, and Snap locations) are supported in v1. On Windows (and any non-macOS/non-Linux host), `list` returns `details.electron.platform: "unsupported"` with an empty `apps` array—use `executablePath` (or a host `appPath` that resolves to a verifiable Electron binary) for `launch` instead; `inspectElectronExecutablePath` in `extensions/agent-browser/lib/electron/discovery.ts` still gates Windows executables before spawn.
122
+
123
+ ```json
124
+ { "electron": { "action": "list", "query": "code", "maxResults": 25 } }
125
+ ```
126
+
127
+ Returns app metadata under `details.electron.apps`: `name`, optional `bundleId`/`desktopId`, `appPath`, `executablePath`, `platform`, and optional non-blocking `sensitivity` annotations. Apps flagged as likely sensitive (categories such as `notes`, `chat`, `mail`, `developer-workspace`, or `passwords-auth`) are printed with `[likely sensitive: …]`. These are **advisory hints**, not enforcement; see [Safety and ownership](#safety-and-ownership) for the policy boundary.
128
+
129
+ ### `electron.launch` — launch and attach
130
+
131
+ Pass **exactly one** target: `appPath`, `appName`, `bundleId`, or `executablePath`. The wrapper resolves the target, verifies Electron framework evidence, applies optional caller-owned `allow` / `deny` policy, creates an isolated temp `userDataDir`, launches with `--remote-debugging-port=0` plus safe defaults, reads `DevToolsActivePort`, then attaches through upstream `connect` as a fresh managed session.
132
+
133
+ ```json
134
+ {
135
+ "electron": {
136
+ "action": "launch",
137
+ "appName": "Visual Studio Code",
138
+ "handoff": "snapshot",
139
+ "targetType": "page",
140
+ "timeoutMs": 30000,
141
+ "appArgs": ["--disable-telemetry"]
142
+ }
143
+ }
144
+ ```
145
+
146
+ Handoff selection (`handoff` field):
147
+
148
+ | Value | Behavior | When to use |
149
+ |---|---|---|
150
+ | `"snapshot"` (default) | Attach, list targets, capture `snapshot -i` in one call | You need interactive refs immediately for clicks/fills |
151
+ | `"tabs"` | Attach and list targets only | Safer diagnostic start when you only need target discovery |
152
+ | `"connect"` | Attach and stop | You will run your own follow-up commands |
153
+
154
+ `targetType` defaults to `"page"`; use `"webview"` or `"any"` for apps whose useful UI is exposed as a webview target.
155
+
156
+ Optional `timeoutMs` on `electron.launch` bounds host-side CDP readiness (waiting for `DevToolsActivePort` and attach). When omitted, the default is **15 seconds** with a hard maximum of **120 seconds**, matching `ELECTRON_LAUNCH_DEFAULT_TIMEOUT_MS` and `ELECTRON_LAUNCH_MAX_TIMEOUT_MS` in `extensions/agent-browser/lib/electron/launch.ts`.
157
+
158
+ Wrapper-owned launches **always** use an isolated temp profile and an OS-chosen port. `--user-data-dir`, `--remote-debugging-port`, `--remote-debugging-address`, `--remote-debugging-pipe`, and bare `--` in `appArgs` are rejected. There is no caller-supplied port and no way to make `electron.launch` reuse the app's normal signed-in profile or attach to an already-running app — by design. Use the manual path described above when those are the actual requirements.
159
+
160
+ ### `electron.status` — liveness and targets
161
+
162
+ Read-only inspection of one or more tracked launches. Without `launchId` or `all`, it selects the single active wrapper launch when unambiguous.
163
+
164
+ ```json
165
+ { "electron": { "action": "status" } }
166
+ { "electron": { "action": "status", "launchId": "electron-…" } }
167
+ { "electron": { "action": "status", "all": true } }
168
+ ```
169
+
170
+ Reports `cleanupState`, debug-port and PID liveness, and bounded CDP target metadata under `details.electron.statuses`. Mismatch fields surface when the current managed session or tab no longer matches a live wrapper launch target — typically the cue to follow `reattach-electron-launch` before trusting old refs.
171
+
172
+ ### `electron.probe` — compact state read
173
+
174
+ `probe` collapses what would otherwise be separate `get title` / `get url` / focused-element `eval` / `tab list` / `snapshot -i` calls into one bounded result. Use it instead of chaining those reads when you just need a quick "where are we?" check.
175
+
176
+ ```json
177
+ { "electron": { "action": "probe" } }
178
+ { "electron": { "action": "probe", "launchId": "electron-…", "timeoutMs": 5000 } }
179
+ ```
180
+
181
+ Output appears under `details.electron.probe`: `title`, `url`, `focusedElement`, `activeTab`, `tabs`, compact `snapshot` metadata (`refCount`, `refIds`, optional text preview and omission counts), and `errors`. When `launchId` is given, the probe is tied to that tracked launch and will surface mismatch guidance if the wrapper sees a session or target drift; visible output also includes debug-port/pid liveness so a stale `about:blank` against a dead launch is unmistakable.
182
+
183
+ `timeoutMs` bounds each underlying read subprocess. Use it for dense desktop apps when the default budget is too short, or to fail fast when you suspect the app process is wedged.
184
+
185
+ ### `electron.cleanup` — wrapper-owned only
186
+
187
+ Closes the tracked managed session, stops only the wrapper-tracked process, verifies that the debug port no longer serves `/json/version`, and removes the wrapper-created `userDataDir`. Cleanup partial failures fail the tool result with `failureCategory: "cleanup-failed"` and the `retry-electron-cleanup` next action references the same `launchId` so retries are bounded.
188
+
189
+ ```json
190
+ { "electron": { "action": "cleanup", "launchId": "electron-…" } }
191
+ { "electron": { "action": "cleanup", "all": true } }
192
+ ```
193
+
194
+ `electron.cleanup` **never** targets:
195
+
196
+ - manually launched apps
197
+ - externally supplied debug ports
198
+ - arbitrary Electron processes the wrapper did not start
199
+ - explicit screenshots, downloads, PDFs, traces, HAR files, or recordings saved to caller-chosen paths
200
+
201
+ For manual launches, `close` only closes the browser/CDP session. Close the app yourself and clean its profile/temp files with normal host tools.
202
+
203
+ On Pi session shutdown, active wrapper-owned Electron launches are best-effort cleaned. Stale restored records (PID gone, port dead) are **reported** instead of guessed at or killed.
204
+
205
+ ### `timeoutMs` by action (quick reference)
206
+
207
+ `electron.list` does not take `timeoutMs` (host scan only). For every other action, `timeoutMs` applies to **different surfaces**; treat values as per-call budgets, not one global knob. Authoritative rules and env overrides live under **Validation and defaults** in [`TOOL_CONTRACT.md#electron`](TOOL_CONTRACT.md#electron).
208
+
209
+ | Action | What `timeoutMs` covers when set | Typical default when omitted |
210
+ | --- | --- | --- |
211
+ | `launch` | Host-side wait for `DevToolsActivePort` and CDP readiness | **15 s**, hard-capped at **120 s** (`normalizeTimeoutMs` in `extensions/agent-browser/lib/electron/launch.ts`) |
212
+ | `status` | Optional managed-session `get title` / `get url` reads used for mismatch diagnostics | Normal tool subprocess budget from `runAgentBrowserProcess` / `AGENT_BROWSER_DEFAULT_TIMEOUT`; localhost CDP HTTP probes keep a short fixed budget (`ELECTRON_STATUS_FETCH_TIMEOUT_MS` in `extensions/agent-browser/lib/electron/cleanup.ts`) |
213
+ | `cleanup` | One combined budget for managed-session `close`, tracked process exit, debug-port verification, and temp profile removal | `PI_AGENT_BROWSER_IMPLICIT_SESSION_CLOSE_TIMEOUT_MS` when set, else **5000 ms** (`getImplicitSessionCloseTimeoutMs` in `extensions/agent-browser/lib/runtime.ts`, passed through `cleanupTrackedElectronLaunches` in `extensions/agent-browser/index.ts`) |
214
+ | `probe` | **Each** upstream read in the probe chain (`get title`, `get url`, focused `eval --stdin`, `tab list`, `snapshot -i`) | Same default as other tool calls (typically **28 s** per subprocess unless `AGENT_BROWSER_DEFAULT_TIMEOUT` / `PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides `runAgentBrowserProcess` in `extensions/agent-browser/lib/process.ts`) |
215
+
216
+ ## `qa.attached` — current-session smoke check
217
+
218
+ `qa` has two forms: the URL form (`qa: { url, … }`) and the attached form (`qa: { attached: true, … }`). The attached form is the right tool for Electron smoke checks after either launch path because it does not open a URL and runs all checks against the current managed session.
219
+
220
+ ```json
221
+ {
222
+ "qa": {
223
+ "attached": true,
224
+ "expectedText": "Explorer",
225
+ "expectedSelector": "@e1",
226
+ "checkConsole": true,
227
+ "checkErrors": true,
228
+ "screenshotPath": ".dogfood/electron.png"
229
+ }
230
+ }
231
+ ```
232
+
233
+ `qa.attached` rejects `url` and is incompatible with `sessionMode: "fresh"` — attach first with `electron.launch` or raw `connect`, then run `qa.attached`. The full field rules and pass/fail classification live in [`TOOL_CONTRACT.md#qa`](TOOL_CONTRACT.md#qa).
234
+
235
+ In attached Electron sessions, broad selectors such as `body`, `html`, `main`, or `[role=application]` can read the entire app shell. When `get text <selector>` looks too broad, the wrapper may attach `details.electronGetTextScopeWarning` and a `snapshot-for-electron-text-scope` next action; prefer a fresh `snapshot -i`, a current `@ref`, or a narrower panel selector.
236
+
237
+ ## `sourceLookup` against packaged Electron apps
238
+
239
+ `sourceLookup` is an experiment for hinting at the source file/component behind a visible element. It is **opt-in** and **evidence-based**: it reports confidence and evidence rather than claiming a guaranteed mapping. The same experimental helper works against packaged Electron apps, but with two important boundaries:
240
+
241
+ 1. **Scope of the workspace scan.** `sourceLookup` walks the Pi session **cwd** (default `maxWorkspaceFiles: 2000`, hard cap 5000). It does **not** unpack `app.asar` or installed app resources. For packaged apps where the source lives inside `Contents/Resources/app.asar`, the workspace-search lane will commonly return no candidates.
242
+ 2. **React DevTools requirement.** `react inspect <id>` requires the session to have been launched with `--enable react-devtools` before first navigation. For Electron, the wrapper's `electron.launch` path does **not** inject `--enable react-devtools` into the Electron process; that flag belongs to upstream `agent-browser` Chromium launches. If the Electron app does not already expose a React DevTools backend, expect `react inspect` to fail; DOM-attribute and workspace-search candidates may still surface.
243
+
244
+ For wrapper-tracked packaged Electron sessions where `status` is `no-candidates`, the wrapper attaches `workspaceRoot` plus optional `electronContext` (`launchId?`, `appName?`, `appPath?`, `executablePath?`, `sessionName?`, `url?`) and limitations explaining the bundle/asar boundary, plus `snapshot-electron-session`, `probe-electron-launch`, and `list-electron-tabs` next actions so you can inspect the live app and decide whether to widen the workspace or pull source out-of-band before re-running the lookup.
245
+
246
+ ```json
247
+ { "sourceLookup": { "selector": "#save", "reactFiberId": "2", "componentName": "SaveButton" } }
248
+ ```
249
+
250
+ Treat `sourceLookup` output as a starting point for navigation, not a substitute for reading code. Full contract: [`TOOL_CONTRACT.md#sourcelookup`](TOOL_CONTRACT.md#sourcelookup).
251
+
252
+ ## Safety and ownership
253
+
254
+ Remote debugging exposes app content (DOM, network, JavaScript) to the attached browser tool. The wrapper ships **isolation defaults**; it does **not** classify any app as too-risky-to-launch.
255
+
256
+ ### What the wrapper always does
257
+
258
+ - Launches with `--user-data-dir=<wrapper-created-temp>` and `--remote-debugging-port=0`.
259
+ - Reads the OS-chosen port from `DevToolsActivePort`.
260
+ - Adds `--disable-extensions`, `--no-first-run`, and `--no-default-browser-check` alongside sanitized caller `appArgs`.
261
+ - Rejects `appArgs` that try to override lifecycle/debug flags.
262
+ - Refuses to launch non-Electron targets (correctness gate, not a security gate).
263
+ - Treats `electron.cleanup` as wrapper-owned only; never touches manually launched apps.
264
+
265
+ ### What the **caller** owns
266
+
267
+ - The decision to launch or attach to a sensitive app in the first place.
268
+ - Optional `allow` / `deny` policy lists when you want guardrails.
269
+ - Profile and process cleanup for manually launched apps.
270
+ - Host-file cleanup for any explicit screenshots, downloads, HARs, traces, or recordings saved to caller-chosen paths. `electron.cleanup` does not touch these.
271
+
272
+ ### Caller-owned policy: `allow` / `deny`
273
+
274
+ Both lists match `appName`, `bundleId`, `desktopId`, `appPath`, or `executablePath` by substring.
275
+
276
+ ```json
277
+ {
278
+ "electron": {
279
+ "action": "launch",
280
+ "appName": "Slack",
281
+ "allow": ["Slack"],
282
+ "deny": ["1Password", "Bitwarden"]
283
+ }
284
+ }
285
+ ```
286
+
287
+ Rules:
288
+
289
+ - If `allow` is set, the target must match at least one entry.
290
+ - If `deny` is set, a matching target is rejected.
291
+ - `deny` wins on conflict.
292
+ - With neither set, launch is permitted.
293
+
294
+ Policy mismatches fail with `failureCategory: "policy-blocked"` and `details.electron.failure.policy` names the matched list and entry.
295
+
296
+ ### Likely-sensitive annotations
297
+
298
+ `electron.list` may annotate common private-data apps (`notes`, `chat`, `mail`, `developer-workspace`, `passwords-auth`) with `sensitivity.level: "likely-sensitive"` and a visible `[likely sensitive: …]` marker. These are **advisory hints only**. They do not block `launch` and they do not replace caller `allow` / `deny`.
299
+
300
+ ## Failure categories and recovery
301
+
302
+ `details.failureCategory` values you should expect from Electron flows, with the recovery move:
303
+
304
+ | Category | When | Recovery |
305
+ |---|---|---|
306
+ | `validation-error` | Bad input (missing target, conflicting fields, non-Electron target) | Fix the request; the message names the problem |
307
+ | `policy-blocked` | Caller `allow` / `deny` rejected the launch | Adjust the policy or pick a different target |
308
+ | `timeout` | `DevToolsActivePort` never appeared in time | Inspect `details.electron.failure.diagnostics` (PID, profile path, port file state, elapsed/timeout); retry with a higher `timeoutMs` if the app legitimately needs more time |
309
+ | `upstream-error` | Launch/attach/spawn/CDP failure that does not fit a more specific bucket | Inspect `details.electron.failure.diagnostics`; the app may be missing dependencies or hitting a CDP race |
310
+ | `tab-drift` | A successful-looking command was followed by a dead process / debug port / unrecoverable `about:blank` | Use the appended `status-electron-launch` / `probe-electron-launch` next actions, then decide whether to relaunch |
311
+ | `cleanup-failed` | Cleanup only partially succeeded | Inspect `details.electron.cleanup.results[].steps` for remaining process/port/profile state; `retry-electron-cleanup` references the same `launchId` |
312
+ | `stale-ref` | `@e…` ref reused after a navigation/rerender | Take a fresh `snapshot -i` (or follow `refresh-electron-refs-after-rerender` when the wrapper appends it) |
313
+
314
+ Single-instance Electron behavior is a common cause of `timeout` and `upstream-error`. Many Electron apps enforce a single running instance and silently drop a second invocation's `--remote-debugging-port` flag. If the app is already running without a debug port, quit it first or use the manual host-launch path against the existing instance instead.
315
+
316
+ ## Troubleshooting
317
+
318
+ ### Launch hangs and then times out
319
+ - The app is enforcing single-instance; quit the running copy first, then retry.
320
+ - The app may have moved its Electron framework directory; pass `executablePath` explicitly.
321
+ - `timeoutMs` is too short for a heavy app; raise it (`launch.timeoutMs` is bounded but generous).
322
+ - Read `details.electron.failure.diagnostics`: presence/absence of `DevToolsActivePort`, port number, PID liveness, and elapsed time usually identify the issue.
323
+
324
+ ### `electron.list` returns nothing
325
+ - On Linux, the binary may be a custom rebrand without `chrome_*.pak` siblings, an AppImage without a `.desktop` entry, or a statically linked fork. Pass `executablePath` directly.
326
+ - On macOS, apps installed outside `/Applications` and `~/Applications` are not scanned in v1. Pass `appPath` or `executablePath` explicitly.
327
+ - Windows hosts report `platform: "unsupported"` from `electron.list`; always pass `executablePath` (or a resolvable `appPath`) for `launch`.
328
+
329
+ ### Attach succeeds but `snapshot -i` returns no refs
330
+ - Some Electron apps take a beat to render. The default `handoff: "snapshot"` already retries briefly; if it still reports no refs, run `tab list`, select the intended stable `t<N>` app tab, then run `snapshot -i` again.
331
+ - For raw `connect`, do the same target check before assuming the signed-in app is ready; the attach can succeed before an active page is available.
332
+ - For apps whose UI lives in a webview, switch `targetType` to `"webview"` or `"any"` so the wrapper attaches to the right CDP target.
333
+
334
+ ### "I clicked, but nothing happened"
335
+ - A successful upstream `click` means the action was dispatched, not that the app handled it. Re-snapshot, check `details.pageChangeSummary`, or use `qa.attached` to verify.
336
+ - Electron apps frequently rerender in place (no URL change). The wrapper may attach `refresh-electron-refs-after-rerender` to remind you to re-snapshot before reusing `@e…` refs.
337
+
338
+ ### `fill` looks fine but the field is empty
339
+ - Custom quick-input controls (VS Code's quick-pick, command palette, etc.) often need focus + keyboard typing rather than a direct `fill`. The wrapper attaches `details.fillVerification` when `get value` disagrees with the requested text; follow `inspect-after-fill-verification` and switch to focus + `keyboard type` before submitting.
340
+
341
+ ### `get text` returns the whole app
342
+ - Broad selectors (`body`, `html`, `main`, `[role=application]`) read the entire shell. Use a current `@ref` or a narrower panel selector. The wrapper attaches `details.electronGetTextScopeWarning` and a `snapshot-for-electron-text-scope` next action when it detects this pattern.
343
+
344
+ ### `sourceLookup` says `no-candidates` for a packaged app
345
+ - Expected when the app's source lives inside `app.asar`. The wrapper does not unpack bundles. Use `electron.probe` / `snapshot-electron-session` / `list-electron-tabs` next actions to inspect the live UI, or pull source separately into the Pi session cwd before re-running the lookup.
346
+
347
+ ### Mismatch between `status` and the active session
348
+ - `electron.status` may report a live wrapper launch while the managed session has drifted to `about:blank`. Follow `reattach-electron-launch`, then refresh refs before reusing old `@e…` handles. For non-wrapper tab drift where `details.nextActions` names `select-intended-tab-after-drift`, use that stable `t<N>` action plus `snapshot-after-tab-recovery` before continuing.
349
+
350
+ ## Cleanup checklist
351
+
352
+ Before ending the task:
353
+
354
+ - Call `electron.cleanup` (or `electron.cleanup` with `all: true`) for every wrapper-owned `launchId` you started. The result reports per-step state for `managed-session`, `process`, `debug-port`, and `user-data-dir`.
355
+ - Confirm `details.electron.cleanup.summary` does not list remaining resources.
356
+ - For **manually launched** apps, close the app yourself and clean any profile or temp files you created. `electron.cleanup` will not (and should not) touch them.
357
+ - Remove any explicit screenshots, recordings, downloads, PDFs, traces, or HAR files you saved to caller-chosen paths. Artifact cleanup is host-owned; the wrapper only reports them under `details.artifacts` and `details.artifactCleanup`.
358
+
359
+ If `cleanup` returns `failureCategory: "cleanup-failed"`, inspect `details.electron.cleanup.results[].steps` and use `retry-electron-cleanup` for the same `launchId`. Do not invent new cleanup commands for processes the wrapper did not start.
360
+
361
+ ## Verification and benchmarks
362
+
363
+ Electron support is gated by the same release evidence as the rest of the wrapper:
364
+
365
+ - `RQ-0096` in [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) records the contract, runtime, test, and verification coverage.
366
+ - `electron-lifecycle` and `electron-probe` scenarios in `scripts/agent-browser-efficiency-benchmark.mjs` track the token-efficiency claim deterministically (no real browser, no real launches).
367
+ - Fake-upstream coverage for Electron schema/probe/mismatch/post-command-health/fill-verification/broad-text/discovery-sensitivity lives in `test/agent-browser.extension-validation.test.ts`.
368
+ - Real-app validation is a manual `tmux` smoke pass per the maintainer notes in `AGENTS.md`; the 2026-05-21 dogfood result is recorded at the end of [`docs/plans/electron-extension-2026-05-20.md`](plans/electron-extension-2026-05-20.md).
369
+
370
+ Run the local gate the same way as the rest of the project:
371
+
372
+ ```bash
373
+ npm run verify
374
+ ```
375
+
376
+ The token-efficiency claim has its own opt-in run:
377
+
378
+ ```bash
379
+ npm run benchmark:agent-browser
380
+ ```
381
+
382
+ ## Where to go next
383
+
384
+ - For exact field semantics, schemas, and `details.*` payloads: [`TOOL_CONTRACT.md#electron`](TOOL_CONTRACT.md#electron) and [`TOOL_CONTRACT.md#qa`](TOOL_CONTRACT.md#qa).
385
+ - For workflow examples woven into the broader command surface: [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#electron-desktop-apps).
386
+ - For the closed `RQ-0068` recipe-layer decision that bounds why Electron support is a typed shorthand and not a generic recipe runtime: [`ARCHITECTURE.md`](ARCHITECTURE.md#no-reusable-recipe-layer-yet).
387
+ - For the full release-readiness audit and the `RQ-0096` evidence row: [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md).
package/docs/RELEASE.md CHANGED
@@ -5,11 +5,12 @@ Related docs:
5
5
  - [`REQUIREMENTS.md`](REQUIREMENTS.md)
6
6
  - [`ARCHITECTURE.md`](ARCHITECTURE.md)
7
7
  - [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md)
8
+ - [`ELECTRON.md`](ELECTRON.md)
8
9
  - [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md)
9
10
  - Bounded `agent_browser` outcome metadata on `details` (`resultCategory`, `successCategory`, `failureCategory`, optional `nextActions`, optional `pageChangeSummary` with per-step summaries on `batch`): contract in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details); maintainer checklists under “Tool result categories” and “Page-change summaries” in [`../AGENTS.md`](../AGENTS.md)
10
11
  - Post-success `get text` selector visibility (`RQ-0074`): optional `details.selectorTextVisibility` / `selectorTextVisibilityAll`, visible warnings, and `inspect-visible-text-candidates*` next actions after read-only visibility probes—[`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md), [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details), and [`../AGENTS.md`](../AGENTS.md) maintainer checklist
11
12
  - Managed-session outcomes (`RQ-0077`): after extension-managed implicit or fresh `--session` injection reaches process execution, `details.managedSessionOutcome` records the transition (`created` / `replaced` / `unchanged` / `closed` on success; `preserved` / `abandoned` when a plan fails before a new session becomes current). Failing `sessionMode: "fresh"` calls also append model-visible `Managed session outcome: …`—[`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details), [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md), [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md), and [`../AGENTS.md`](../AGENTS.md) maintainer checklist
12
- - Stateful context commands (`cookies`, `storage`, `auth`, `dialog`, `frame`, `state`) and aggregate `batch` results: model-facing `details.data` is summarized or redacted per [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details); aggregate `batch` replaces top-level `details.data` with a compact per-step matrix (`success`, argv-redacted `command`, redacted `result` or scrubbed `error`) while full per-step payloads, artifacts, and categories remain on `batchSteps[]`—operational notes in [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#use-stateful-browser-context-commands-safely), assembly in `extensions/agent-browser/lib/results/presentation.ts`
13
+ - Stateful context commands (`cookies`, `storage`, `auth`, `dialog`, `frame`, `state`) and aggregate `batch` results: model-facing `details.data` is summarized or redacted per [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details); aggregate `batch` replaces top-level `details.data` with a compact per-step matrix (`success`, argv-redacted `command`, redacted `result` or scrubbed `error`) while full per-step payloads, artifacts, and categories remain on `batchSteps[]`—operational notes in [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#use-stateful-browser-context-commands-safely), assembly in `extensions/agent-browser/lib/results/presentation/batch.ts`
13
14
 
14
15
  ## Purpose
15
16
 
@@ -36,7 +37,9 @@ npm run verify -- release
36
37
 
37
38
  `prepublishOnly` intentionally does **not** run `npm run verify -- lifecycle`, `npm run verify -- real-upstream`, or `npm run verify -- benchmark`; those are separate `npm run verify` modes in [`scripts/project.mjs`](../scripts/project.mjs). Treat the bullets below as the full pre-publish contract even though only the `release` slice is automated at publish time.
38
39
 
39
- Every release also requires interactive `tmux`-driven Pi dogfood with the native `agent_browser` tool against real sites. For extension-focused release smokes, use `pi --no-extensions --no-skills -e .` from the checkout before publish so auto-loaded dogfood/QA skills cannot replace the bounded smoke workflow; run separate skill-enabled dogfood only when validating skill routing or report-generation behavior. Drive prompts with `tmux send-keys`, exercise at least one simple static site and one real documentation/product site, include the higher-level `qa` or `job`/`batch` surfaces when they changed, close every opened browser session, remove screenshots/temp artifacts, and record the outcome in the release notes or support-matrix evidence. Automated localhost and fake-upstream gates do not replace this human-readable live-site transcript evidence. For dense-dashboard stress coverage, use the [public Grafana stress checklist](#public-grafana-stress-checklist) below; it is a maintainer workflow, not bundled product skill or recipe runtime.
40
+ Every release also requires interactive `tmux`-driven Pi dogfood with the native `agent_browser` tool against real sites. For extension-focused release smokes, use `pi --no-extensions --no-skills -e .` from the checkout before publish so auto-loaded dogfood/QA skills cannot replace the bounded smoke workflow; run separate skill-enabled dogfood only when validating skill routing or report-generation behavior. Drive prompts with `tmux send-keys`, exercise at least one simple static site and one real documentation/product site, include the higher-level `qa` or `job`/`batch` surfaces when they changed, close every opened browser session, remove screenshots/temp artifacts, and record the outcome in the release notes or support-matrix evidence. Automated localhost and fake-upstream gates do not replace this human-readable live-site transcript evidence. When `electron.*` surfaces, attached-session diagnostics, or `qa.attached` changed, add a local Electron pass: `electron.list` → `electron.launch` (expect isolated profile behavior) → `snapshot -i` or `electron.probe` / `qa.attached` → `electron.cleanup` with the returned `launchId`, verifying status/mismatch guidance if you simulate a dead renderer or stale refs. For dense-dashboard stress coverage, use the [public Grafana stress checklist](#public-grafana-stress-checklist) below; it is a maintainer workflow, not bundled product skill or recipe runtime.
41
+
42
+ When reviewing saved session JSONL after a failed smoke or a `qa` preset that reclassified an upstream-successful batch, expect `agent_browser` tool rows to carry `isError: true` whenever `details.resultCategory` is `failure`. For normal prose output, model-visible text should end with a `Pi tool isError: true` category line; for caller-requested `--json` output, the hook preserves parseable JSON and only patches `isError`. The extension applies that patch on the `tool_result` path so Pi’s transcript matches the wrapper contract ([`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details)). Preserve a normal Pi session directory for those checks; avoiding `--no-session` keeps this evidence intact ([`AGENTS.md`](../AGENTS.md) preferred validation workflow).
40
43
 
41
44
  The configured-source lifecycle regression harness is required before release because it launches an interactive `pi` process under `tmux` and validates `/reload` plus restart/`/resume` behavior:
42
45
 
@@ -181,7 +184,34 @@ Run the automated harness for deterministic configured-source lifecycle regressi
181
184
  npm run verify -- lifecycle
182
185
  ```
183
186
 
184
- The harness creates an isolated `PI_CODING_AGENT_DIR`, writes settings with exactly one temporary configured package source, runs plain `pi` in `tmux`, puts a deterministic fake `agent-browser` first on `PATH`, and drives `/reload`, full restart, and `/resume`. It asserts same-page managed-session continuity, persisted `details.fullOutputPath` reachability after resume, and updated extension-code pickup through a temporary sentinel command. On failure it retains transcripts/session artifacts; on success it performs best-effort cleanup. It does not replace occasional real-browser manual smoke testing.
187
+ The harness creates an isolated `PI_CODING_AGENT_DIR`, writes settings with exactly one temporary configured package source, runs plain `pi` in `tmux` with default model **`zai/glm-5.1`**, puts a deterministic fake `agent-browser` first on `PATH`, and drives `/reload`, full restart, and `/resume`. Per-step tmux waits default to **180000 ms** (three minutes) in [`scripts/verify-lifecycle.mjs`](../scripts/verify-lifecycle.mjs) (`DEFAULT_TIMEOUT_MS`); override with `--timeout-ms <ms>` when slower models or cold starts need more headroom. Override the model when needed:
188
+
189
+ ```bash
190
+ npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal
191
+ ```
192
+
193
+ Combine flags in one invocation when both apply (order after `lifecycle` is flexible as long as each value-taking flag is immediately followed by its value):
194
+
195
+ ```bash
196
+ npm run verify -- lifecycle --model openai-codex/gpt-5.5:minimal --timeout-ms 600000
197
+ ```
198
+
199
+ It asserts same-page managed-session continuity, persisted `details.fullOutputPath` reachability after resume, and updated extension-code pickup through a temporary sentinel command. On failure it retains transcripts/session artifacts; on success it performs best-effort cleanup. It does not replace occasional real-browser manual smoke testing.
200
+
201
+ **Lifecycle triage:** a timeout on sentinel `v2` after `/reload` often means Pi rejected reload while the TUI still showed `Working…` (`Wait for the current response to finish before reloading`), even when the session JSONL already has a final assistant message. Re-run with `--keep-artifacts --verbose`, inspect the retained pane capture, and confirm the configured model follows tool prompts reliably. Slower models may need a higher `--timeout-ms` than the **180000 ms** default.
202
+
203
+ ### Environment and automation pitfalls
204
+
205
+ These show up often in cloud dev boxes and scripted smokes; they are maintainer notes, not product defects.
206
+
207
+ | Topic | What to watch for | Mitigation |
208
+ | --- | --- | --- |
209
+ | **Pi CLI vs repo devDependencies** | Global `pi` older than the `@earendil-works/pi-coding-agent` range in `package.json` can change TUI behavior, `/reload`, and tool routing during lifecycle or checkout smokes. | Align `pi` with the repo’s pinned coding-agent release before release gates (`pi update` or install the matching version). |
210
+ | **npm lockfile (`packageManager`)** | `package.json` pins **npm@11**. npm 10 may only strip optional `libc` metadata on `@esbuild/*` platform entries in `package-lock.json` (no dependency version change). | Prefer `npx -y npm@11.14.0 install` when refreshing the lockfile; do not commit npm-10-only lockfile churn. |
211
+ | **`pi -p` / print mode** | Non-interactive `pi -p` may hang or emit no stdout for long real-browser smokes without a TTY. | Use **tmux**-driven interactive `pi` for release evidence and checkout smokes; reserve `-p` for short, non-browser checks. |
212
+ | **Real-browser cleanup** | `real-upstream`, Sauce Demo, and live-site runs can leave defunct Chrome/`agent-browser` children if a session aborts mid-flow. | Close via `agent_browser` / `agent-browser` `close`, kill stray tmux sessions, and remove temp screenshots/HARs under `/tmp` or your chosen artifact dirs. |
213
+ | **Automated prompt driving** | Grepping tmux pane text for words that also appear in the **user** prompt (`PASS`, `FAIL`, `checkout overview`, `Smoke result:`) can false-complete before the agent finishes. | Wait for pane idle (no `Working…`), `agent_browser close` / `Artifact lifecycle`, or JSONL tool results—not instruction phrases copied from the prompt. |
214
+ | **Lifecycle verify flags** | `npm run verify -- lifecycle --model` or `--timeout-ms` without the next argv token fails fast with a usage error—the `project.mjs` facade validates passthrough the same way as `scripts/verify-lifecycle.mjs`. | Always pair flags with values (`--model openai-codex/gpt-5.5:minimal`, `--timeout-ms 600000`) or omit `--model` / `--timeout-ms` to keep the harness defaults (`zai/glm-5.1`, **180000 ms** per-step waits). |
185
215
 
186
216
  Manual validation remains useful for release confidence and installed-package checks:
187
217
 
@@ -192,7 +222,7 @@ Manual validation remains useful for release confidence and installed-package ch
192
222
 
193
223
  ### Real upstream contract validation
194
224
 
195
- The default `npm test` and `npm run verify` paths use fast deterministic tests and fake binaries. When a change touches upstream command planning, result presentation, managed-session behavior, or the canonical capability baseline, also run the opt-in real-upstream contract suite:
225
+ The default `npm test` and `npm run verify` paths use fast deterministic tests and fake binaries. For a focused single-file rerun, use `npx tsx --test test/<file>.test.ts`; `npm test -- test/<file>.test.ts` still runs the package script's full glob. When a change touches upstream command planning, result presentation, managed-session behavior, or the canonical capability baseline, also run the opt-in real-upstream contract suite:
196
226
 
197
227
  ```bash
198
228
  npm run verify -- real-upstream
@@ -4,6 +4,7 @@ Related docs:
4
4
  - [`../README.md`](../README.md)
5
5
  - [`ARCHITECTURE.md`](ARCHITECTURE.md)
6
6
  - [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md)
7
+ - [`ELECTRON.md`](ELECTRON.md)
7
8
  - [`RELEASE.md`](RELEASE.md)
8
9
  - [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md)
9
10
 
@@ -63,7 +64,7 @@ Define the product requirements and constraints for `pi-agent-browser-native`.
63
64
 
64
65
  ### Native `agent_browser` inputs
65
66
 
66
- - Each tool invocation must supply **exactly one** of: `args` (full upstream argv after the binary name), top-level `semanticAction` (a small intent object compiled into existing upstream `find` argv for locator actions or upstream `select <selector> <value...>` argv for native dropdown selection), `job`, `qa`, `sourceLookup`, or `networkSourceLookup`. Supplying multiple modes or none is rejected before launch (`extensions/agent-browser/index.ts`, `test/agent-browser.extension-validation.test.ts`).
67
+ - Each tool invocation must supply **exactly one** of: `args` (full upstream argv after the binary name), top-level `semanticAction` (a small intent object compiled into existing upstream `find` argv for locator actions or upstream `select <selector> <value...>` argv for native dropdown selection), `job`, `qa`, `sourceLookup`, `networkSourceLookup`, or `electron` (bounded desktop lifecycle: host `list`, wrapper-owned isolated `launch` with CDP attach, `status`, compact `probe`, and `cleanup`; mutually exclusive with caller `stdin`). Supplying multiple modes or none is rejected before launch (`extensions/agent-browser/index.ts`, `test/agent-browser.extension-validation.test.ts`). Contract and field rules: [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#electron); operator workflow: [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#electron-desktop-apps).
67
68
  - `semanticAction` is not a nested shape inside `batch` stdin; batch steps remain upstream argv string arrays, including `find` steps expressed as token lists.
68
69
  - Supported actions, locators, exclusivity rules, when `details.compiledSemanticAction` appears, and bounded `try-*-candidate` follow-ups on `selector-not-found` (specific action/locator pairs only; see contract) are specified in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#semanticaction), with workflow examples in [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md).
69
70
 
@@ -84,7 +85,7 @@ Define the product requirements and constraints for `pi-agent-browser-native`.
84
85
  - The primary confidence path is a real `pi` session driven in `tmux`.
85
86
  - For quick local checkout smoke validation, launch `pi --no-extensions -e .` from the repository root so only the checkout copy loads; do not rely on Pi settings or `/reload` semantics in this isolated mode.
86
87
  - For hot-reload validation, configure exactly one active source for this extension in Pi settings and launch plain `pi`; validate `/reload` there because it exercises auto-discovered/configured resources.
87
- - Maintain a tmux-driven configured-source lifecycle harness (`npm run verify -- lifecycle`; required before release per `docs/RELEASE.md`) that isolates Pi settings, uses exactly one configured source, exercises `/reload`, full restart, and `/resume`, and asserts managed-session continuity plus persisted artifact survival. It is its own `npm run verify` mode rather than part of the default `npm run verify` sequence, but operators still run it before every publish. Keep `docs/RELEASE.md` accurate about the harness behavior, cleanup, transcript retention, and limitations.
88
+ - Maintain a tmux-driven configured-source lifecycle harness (`npm run verify -- lifecycle`; required before release per `docs/RELEASE.md`) that isolates Pi settings, uses exactly one configured source, exercises `/reload`, full restart, and `/resume`, and asserts managed-session continuity plus persisted artifact survival. It is its own `npm run verify` mode rather than part of the default `npm run verify` sequence, but operators still run it before every publish. The harness defaults Pi to model `zai/glm-5.1` (`scripts/verify-lifecycle.mjs`); pass `--model <id>` after `lifecycle` when a different model is required. Keep `docs/RELEASE.md` accurate about the harness behavior, cleanup, transcript retention, and limitations.
88
89
  - Validate a full `pi` restart with `/resume` when changes touch managed-session continuity, reload behavior, or persisted artifact paths.
89
90
  - Prefer full `pi` restart over `/reload` when validating extension changes beyond a quick reload smoke check.
90
91
  - Use `/resume` when needed after restart.
@@ -103,13 +104,14 @@ The design should comfortably support workflows such as:
103
104
  - headless authenticated `chat.com` / ChatGPT / OpenAI browsing without forcing `--headed` or `--auto-connect`
104
105
  - upstream profile/debug workflows without adding a local profile-cloning layer in this package
105
106
  - provider-backed or iOS device launches where upstream owns credentials, env, and setup; the wrapper forwards argv and a curated provider-related environment without emulating those backends
107
+ - desktop Electron targets using top-level `electron` for discover → isolated launch → attach → probe/cleanup, or raw `args: ["connect", …]` when the operator launches the real app with a debug port for signed-in state (see [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#electron) and [`COMMAND_REFERENCE.md`](COMMAND_REFERENCE.md#electron-desktop-apps))
106
108
 
107
109
  ## Implications for the implementation
108
110
 
109
111
  - Package-manifest behavior matters more than repo-local development wiring.
110
112
  - The extension should use official `pi` hooks and package resources where possible.
111
113
  - The wrapper should stay thin, with upstream `agent-browser` remaining the source of truth for command semantics.
112
- - Successful and failed tool outcomes should surface bounded machine-readable fields on Pi-facing `details` (`resultCategory`, `successCategory`, `failureCategory`, optional structured `nextActions`, optional `pageChangeSummary` with per-step summaries on `batch`, optional `artifactVerification` with the same shape on successful `batchSteps[]` rows) so agents can branch without parsing prose; stateful commands (`auth`, `cookies`, `storage`, `dialog`, `frame`, `state`) plus other structured diagnostics (for example `network`, `diff`, `trace`, `stream`, `dashboard`, `chat`) and `batch` should redact secret-bearing payloads in model-facing `details.data`, including the compact per-step `batch` roll-up on the parent result (full per-step payloads live on `batchSteps[]`). The contract lives in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details), enums and classifier precedence live in `extensions/agent-browser/lib/results/shared.ts`, and presentation-time summaries, redaction, network request follow-ups, and artifact verification rollups are assembled in `extensions/agent-browser/lib/results/presentation.ts` (`buildPageChangeSummary`, `PAGE_CHANGE_SUMMARY_COMMANDS`, `redactPresentationData`, `buildArtifactVerificationSummary`, `buildBatchPresentation`).
114
+ - Successful and failed tool outcomes should surface bounded machine-readable fields on Pi-facing `details` (`resultCategory`, `successCategory`, `failureCategory`, optional structured `nextActions`, optional `pageChangeSummary` with per-step summaries on `batch`, optional `artifactVerification` with the same shape on successful `batchSteps[]` rows) so agents can branch without parsing prose; stateful commands (`auth`, `cookies`, `storage`, `dialog`, `frame`, `state`) plus other structured diagnostics (for example `network`, `diff`, `trace`, `stream`, `dashboard`, `chat`) and `batch` should redact secret-bearing payloads in model-facing `details.data`, including the compact per-step `batch` roll-up on the parent result (full per-step payloads live on `batchSteps[]`). The contract lives in [`TOOL_CONTRACT.md`](TOOL_CONTRACT.md#details), enums and classifier precedence live in `extensions/agent-browser/lib/results/categories.ts` and `contracts.ts` (also re-exported from `shared.ts`), and presentation-time summaries, redaction, network request follow-ups, and artifact verification rollups are assembled in `extensions/agent-browser/lib/results/presentation.ts` (`buildPageChangeSummary`, `PAGE_CHANGE_SUMMARY_COMMANDS`, `redactPresentationData`, `buildArtifactVerificationSummary`, `buildBatchPresentation`).
113
115
  - User-facing docs belong in `README.md` and the canonical published files under `docs/`.
114
116
  - Agent workflow and deeper testing procedures can stay in `AGENTS.md`, but published docs must not depend on that file being present.
115
117
  - When upstream `agent-browser` changes, refresh the local command reference, prompt guidance, and other extension-side docs so agents still have a repo-readable equivalent of the blocked direct-binary help path.