pi-agent-browser-native 0.6.17 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/README.md +55 -167
- package/dist/extensions/agent-browser/index.js +312 -223
- package/dist/extensions/agent-browser/lib/browser-transcript.js +33 -0
- package/dist/extensions/agent-browser/lib/execution-cwd.js +4 -2
- package/dist/extensions/agent-browser/lib/input-modes/job.js +2 -213
- package/dist/extensions/agent-browser/lib/input-modes/params.js +146 -142
- package/dist/extensions/agent-browser/lib/input-modes/script.js +64 -92
- package/dist/extensions/agent-browser/lib/managed-session-policy-lock.js +170 -24
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/diagnostics.js +13 -1
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result.js +26 -13
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/index.js +21 -1
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/snapshot-filter.js +4 -2
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare.js +1 -3
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output.js +35 -20
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/recording-recovery.js +3 -1
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/screenshot-observation.js +79 -0
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/index.js +3 -4
- package/dist/extensions/agent-browser/lib/orchestration/input-plan.js +19 -52
- package/dist/extensions/agent-browser/lib/orchestration/native-session-defaults.js +3 -3
- package/dist/extensions/agent-browser/lib/orchestration/output-file.js +3 -3
- package/dist/extensions/agent-browser/lib/orchestration/script-mode.js +91 -278
- package/dist/extensions/agent-browser/lib/pi-tool-rendering.js +9 -9
- package/dist/extensions/agent-browser/lib/playbook.js +46 -25
- package/dist/extensions/agent-browser/lib/process-identity.js +22 -6
- package/dist/extensions/agent-browser/lib/process.js +16 -13
- package/dist/extensions/agent-browser/lib/results/action-recommendations.js +2 -2
- package/dist/extensions/agent-browser/lib/results/next-actions.js +1 -2
- package/dist/extensions/agent-browser/lib/results/presentation/artifacts.js +25 -4
- package/dist/extensions/agent-browser/lib/results/presentation/batch.js +6 -2
- package/dist/extensions/agent-browser/lib/results/presentation/common.js +2 -0
- package/dist/extensions/agent-browser/lib/results/presentation/content.js +39 -0
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostics.js +3 -3
- package/dist/extensions/agent-browser/lib/results/presentation/large-output.js +41 -1
- package/dist/extensions/agent-browser/lib/results/presentation.js +16 -11
- package/dist/extensions/agent-browser/lib/runtime.js +18 -6
- package/dist/extensions/agent-browser/lib/session-page-state.js +3 -4
- package/dist/extensions/agent-browser/lib/tool-surface.js +138 -0
- package/dist/extensions/agent-browser/script-worker.js +4 -1
- package/docs/ARCHITECTURE.md +46 -62
- package/docs/COMMAND_REFERENCE.md +77 -136
- package/docs/ELECTRON.md +51 -49
- package/docs/RELEASE.md +35 -21
- package/docs/REQUIREMENTS.md +18 -14
- package/docs/SUPPORT_MATRIX.md +33 -15
- package/docs/TOOL_CONTRACT.md +216 -405
- package/docs/platform-smoke.md +2 -2
- package/package.json +1 -1
- package/scripts/doctor.mjs +2 -2
package/docs/TOOL_CONTRACT.md
CHANGED
|
@@ -8,12 +8,11 @@ Related docs:
|
|
|
8
8
|
- [`ELECTRON.md`](ELECTRON.md)
|
|
9
9
|
- [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md)
|
|
10
10
|
|
|
11
|
-
##
|
|
11
|
+
## Browser tools
|
|
12
12
|
|
|
13
|
-
|
|
13
|
+
Always-on tools are `agent_browser` (native argv), `agent_browser_code` (fresh JavaScript over a persistent browser), and `agent_browser_tools` (advanced capability discovery/activation). Optional credential-backed `agent_browser_web_search` remains independent.
|
|
14
14
|
|
|
15
|
-
|
|
16
|
-
- `agent_browser_web_search` when an Exa or Brave Search credential source is configured or resolvable from available config and `webSearch.enabled` is not false
|
|
15
|
+
Pi 0.87.0 or newer is required; official and fork qualification is independent. Five specialized tools are registered but initially inactive: `agent_browser_action`, `agent_browser_qa`, `agent_browser_electron`, `agent_browser_source`, and `agent_browser_network_source`. Explicit CLI allowlists, filtered SDK catalogs, and restored Pi tool history are respected. Omit CLI `--tools` for lazy activation, or include every desired advanced tool in its allowlist; the loader reports excluded tools as unavailable rather than bypassing host selection. Pi 0.87 does not distinguish an SDK explicitly selecting the complete browser catalog from its default catalog; SDK hosts choosing activation for that complete catalog can call `session.setActiveToolsByName(...)` after `bindExtensions()`.
|
|
17
16
|
|
|
18
17
|
## Why this tool shape
|
|
19
18
|
|
|
@@ -46,15 +45,15 @@ agentBrowserExtension(pi, {
|
|
|
46
45
|
|
|
47
46
|
`beforeExecute` is an optional host callback, not a tool input or package config field. It receives the original outer Pi tool-call ID and the current `ExtensionContext`, with `ctx.signal` set to the dispatched call's abort signal. Hosts must honor that signal when waiting so Stop remains responsive.
|
|
48
47
|
|
|
49
|
-
The extension awaits it after input resolution succeeds and before each
|
|
48
|
+
The extension awaits it after input resolution succeeds and before each ordinary dispatch, including each accepted `browser(...)` call inside code. The code wrapper itself does not call it; inner calls retain the original outer ID and use their own cancellation signals. The existing inner-call queue remains serial even for `Promise.all`, so a completed inner call's files are available to the next callback. Internal helper probes and cleanup do not call it, nor does each row of a native `batch` get a separate callback. Native setup and helper probes do not constitute host callback invocations.
|
|
50
49
|
|
|
51
|
-
Supplying the callback registers `agent_browser` with Pi's native `executionMode: "sequential"`. Pi then finishes earlier sibling tools before entering the callback and dispatching the browser call. Rejection prevents that dispatch: direct calls become Pi tool errors, while
|
|
50
|
+
Supplying the callback registers `agent_browser` with Pi's native `executionMode: "sequential"`. Pi then finishes earlier sibling tools before entering the callback and dispatching the browser call. Rejection prevents that dispatch: direct calls become Pi tool errors, while code inner calls receive a failed observation. The host owns saving/retry policy; the extension adds no checkpoint store, retry loop, or deadline. Code deadlines and child reaping still apply; finishing code does not close its selected browser. Omitting the callback preserves ordinary scheduling and behavior; the separate web-search tool is unchanged.
|
|
52
51
|
|
|
53
52
|
## Optional companion web search
|
|
54
53
|
|
|
55
54
|
`agent_browser_web_search` is a separate custom tool, not an `agent_browser` input mode. It is available when the extension can see at least one configured/resolvable Exa or Brave credential source from `~/.pi/config/pi-agent-browser-native/config.json`, `.pi/config/pi-agent-browser-native/config.json`, `PI_AGENT_BROWSER_CONFIG`, or the `EXA_API_KEY` / `BRAVE_API_KEY` environment fallbacks, and runtime execution still checks that the final available merged config has not set `webSearch.enabled` to `false`. Config layers merge global → project → `PI_AGENT_BROWSER_CONFIG` override; under Pi 0.84.0+, globally installed and CLI-loaded copies read `.pi/config/...` when Pi trust allows that project layer, and they skip the project layer when Pi reports the project is untrusted or when launched with `--no-approve`. Disable scope is explicit: a global disable is a normal user default, a project disable applies to one repo, and an override file with `webSearch.enabled: false` is the highest-priority hard disable for that run. Credential sources may be plaintext, `$ENV_VAR` / `${ENV_VAR}` interpolation, escaped literals, or command sources such as `"!op read 'op://Private/Exa/API Key'"` from any loaded config layer; they make the tool available without exposing the value in status text, and command values resolve when the tool executes. Browser profile/executable config uses the same paths and emits prompt guidance from the highest-priority loaded layer, including project config when that layer is loaded.
|
|
56
55
|
|
|
57
|
-
Prefer it for live/current external web facts, current docs/news, and candidate URLs. Prefer it over browser-driving public search-engine forms such as Google: headless
|
|
56
|
+
Prefer it for live/current external web facts, current docs/news, and candidate URLs. Prefer it over browser-driving public search-engine forms such as Google: headless typing flows may be redirected to anti-bot or CAPTCHA pages, and agents should use search API results, then `agent_browser` on a target URL, instead of attempting CAPTCHA bypass. Use `agent_browser` when the task needs browser interaction, screenshots, authenticated/profile content, page inspection, or DOM work. The search tool is namespaced to avoid colliding with generic `web_search`, chooses Exa or Brave automatically from available credentials, defaults to Exa when both are available (unless `webSearch.preferredProvider` is set), and must not expose resolved API keys in content, details, errors, status output, docs examples, logs, or PR artifacts.
|
|
58
57
|
|
|
59
58
|
Config shape:
|
|
60
59
|
|
|
@@ -132,19 +131,21 @@ For Exa, `details.searchType` is the effective requested type even when the prov
|
|
|
132
131
|
|
|
133
132
|
## Input mode chooser
|
|
134
133
|
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
| When you need | Use | Notes |
|
|
134
|
+
| Need | Tool | Input |
|
|
138
135
|
| --- | --- | --- |
|
|
139
|
-
| One
|
|
140
|
-
|
|
|
141
|
-
|
|
|
142
|
-
|
|
|
143
|
-
|
|
|
144
|
-
|
|
|
145
|
-
|
|
|
136
|
+
| One native command | `agent_browser` | `{ args, stdin?, outputPath?, timeoutMs?, sessionMode? }` |
|
|
137
|
+
| Fixed sequence | `agent_browser` | `args: ["batch", "--bail"]` and native JSON-array `stdin` |
|
|
138
|
+
| Loops, branches, aggregation | `agent_browser_code` | `{ code, session?, namespace?, timeoutMs?, outputPath? }` |
|
|
139
|
+
| Discover/enable advanced tools | `agent_browser_tools` | `{ enable?: ["action", "qa", "electron", "source", "network"] }` |
|
|
140
|
+
| Stable locator or native select | `agent_browser_action` | Flat action/locator fields |
|
|
141
|
+
| Diagnostic verdict | `agent_browser_qa` | Flat URL or attached-check fields |
|
|
142
|
+
| Desktop app lifecycle | `agent_browser_electron` | Flat action/target fields |
|
|
143
|
+
| UI/source candidates | `agent_browser_source` | Flat selector/fiber/component fields |
|
|
144
|
+
| Failed-request/source candidates | `agent_browser_network_source` | Flat request/filter fields |
|
|
145
|
+
|
|
146
|
+
Run `agent_browser_tools {}` for inventory; `enable` adds selected capabilities without removing unrelated active tools. Pi's native selected-tool history handles reload/resume rather than a package-owned activation registry. If an exact recovery action names an inactive advanced tool, enable its capability before using that payload.
|
|
146
147
|
|
|
147
|
-
|
|
148
|
+
Use exact labels from the latest snapshot. Return to the model when a fresh observation needs judgment. Neither native batch nor code is a named recipe registry.
|
|
148
149
|
|
|
149
150
|
## Snapshot and getter batching
|
|
150
151
|
|
|
@@ -162,6 +163,8 @@ For link and button text, use the **exact** visible label from the latest `snaps
|
|
|
162
163
|
|
|
163
164
|
`record start` / `restart` accept `--cursor`, `--contact-sheet`, and `--contact-sheet-threshold <n>`. Reported `contactSheetPath` values become image artifacts: pending during capture, checked on disk after stop, and attached when verified. The contact-sheet destination shares the active video's reservation and transcript lifecycle; same-call/batch artifact and `outputPath` aliases are rejected before dispatch. A successful direct `record restart` retains the prior take's known contact-sheet path from the existing reservation, checks it on disk, and labels it unverified because native restart omits terminal sheet evidence. Use `record stop` then a new start when verified completion of both artifacts is required. Video receipt handling remains separate. Native `--input-mode <mode>` is a sticky, live-changeable session setting, not a launch flag; per-action pointer options pass through. See the [command reference](COMMAND_REFERENCE.md#upstream-0381-rebaseline).
|
|
164
165
|
|
|
166
|
+
<a id="wrapper-json"></a>
|
|
167
|
+
|
|
165
168
|
## Wrapper `--json`
|
|
166
169
|
|
|
167
170
|
When explicitly requested, visible JSON remains parseable for early preparation results too: snapshot/network filters, scroll results, and preparation failures use the same JSON envelope as ordinary execution. Diagnostic metadata remains in `details`; `outputPath` notices do not append prose to JSON. Help/version stays native text.
|
|
@@ -172,7 +175,7 @@ The extension always plans normal browser commands with `--json` prepended in `e
|
|
|
172
175
|
|
|
173
176
|
Upstream 0.36.0 exposes page-registered tools through ordinary `args`: `webmcp list [tool] [--frame <frame-id>]`, `webmcp invoke <tool> [--params <json|@file>] [--frame <frame-id>] [--detach] [--timeout <ms>]`, `webmcp result <id>`, and `webmcp cancel <id>`. Locally managed Chrome enables the experimental CDP feature by default. `--no-webmcp` / `AGENT_BROWSER_NO_WEBMCP` / upstream config `noWebmcp` disables it; attached browsers, providers, Lightpanda, Safari/iOS, and older Chrome builds may return upstream `webmcp_unsupported`.
|
|
174
177
|
|
|
175
|
-
Native 0.37 can include `data.webmcp` on successful navigation. When `available` is true and `toolCount` is a positive integer, the page summary shows the native availability hint and recommends `webmcp list`. Raw metadata remains in `details.data`; absent, unavailable, zero or invalid counts add no hint. The wrapper does not run a discovery probe. Native 0.38 also emits bounded catalog updates on normal responses when page tools change. Their names/descriptions and frame/origin metadata are shown as untrusted page data, including an empty catalog after removal; full schemas still require `webmcp list`. Because native updates are emitted once, helper reads also contribute to the current call's `details.webMcpCatalog`. A helper-only update is shown in prose without changing the main command's `details.data`; caller-requested JSON stays parseable. Collection is call-local, including concurrent sessions and
|
|
178
|
+
Native 0.37 can include `data.webmcp` on successful navigation. When `available` is true and `toolCount` is a positive integer, the page summary shows the native availability hint and recommends `webmcp list`. Raw metadata remains in `details.data`; absent, unavailable, zero or invalid counts add no hint. The wrapper does not run a discovery probe. Native 0.38 also emits bounded catalog updates on normal responses when page tools change. Their names/descriptions and frame/origin metadata are shown as untrusted page data, including an empty catalog after removal; full schemas still require `webmcp list`. Because native updates are emitted once, helper reads also contribute to the current call's `details.webMcpCatalog`. A helper-only update is shown in prose without changing the main command's `details.data`; caller-requested JSON stays parseable. Collection is call-local, including concurrent sessions and code inner calls, not a persistent catalog cache.
|
|
176
179
|
|
|
177
180
|
The wrapper keeps this as thin CLI pass-through. `webmcp list` is read-only. `invoke`, `result`, and `cancel` may run page code that mutates, rerenders, or navigates, so the wrapper rechecks the live target, emits the normal `pageChangeSummary` and `inspect-after-mutation` follow-up when applicable, and stores `refSnapshotInvalidation.reason: "page-transition"`; old page-scoped refs remain blocked until a fresh `snapshot -i`. A direct or batched call whose result is still `pending`, or a failed `result` / `cancel` attempt made while that target is unknown, does not treat the immediate URL probe or a same-batch snapshot as stable: `details.sessionTabTargetUnknown` stays true until a successful settlement, `get url`, or explicit navigation verifies the page. Its `details.nextActions` replaces the blocked snapshot suggestion with `verify-page-target-after-pending-webmcp` (`get url`); the action warns that URL inspection does not settle the detached page tool. Inside one `batch --bail`, put `get url` after a completed WebMCP mutation and before `snapshot -i`; a snapshot directly against the unknown target remains blocked. Detached invocation ids and page-returned data remain in `details.data`. When top-level `timeoutMs` is omitted, `webmcp invoke` and `webmcp result` extend the wrapper subprocess watchdog to the upstream `--timeout` value plus a small grace window, including effective raw-argument batch rows (which take precedence over stdin exactly as upstream does).
|
|
178
181
|
|
|
@@ -186,7 +189,7 @@ Upstream 0.35.2 adds `dashboard start --allowed-origins <origins>` and `AGENT_BR
|
|
|
186
189
|
|
|
187
190
|
Local Chrome uses stock `--args --no-startup-window` at bootstrap and composes it consistently with caller-configured Chrome arguments (CLI overrides environment, which overrides native config). Plain active-session follow-ups do not resend bootstrap settings. A pre-existing custom-argument browser may restart once because stock Chrome launch arguments form part of native session identity; there is no migration or launch-settings journal. Headless remains the default. CDP/auto-connect, providers, Electron, and Lightpanda do not receive this Chrome default. The wrapper does not delete tabs or edit profiles.
|
|
188
191
|
|
|
189
|
-
URL-less `open` is executed as native `get url`, including effective raw/stdin batch rows and
|
|
192
|
+
URL-less `open` is executed as native `get url`, including effective raw/stdin batch rows and code calls. This lazily opens a missing browser without navigating an existing page or issuing a second incomplete launch. `details.args` retains the request; `effectiveArgs`, `command`, native URL data and lifecycle show what ran. No launch receipt is synthesized. Explicit URLs, required-URL errors for `goto`/`navigate`, and top-level help/version argv keep their native meanings. Ignored batch stdin remains unchanged.
|
|
190
193
|
|
|
191
194
|
- Upstream 0.35.0 and newer require separate `args` entries for global flag values. The wrapper rejects `--flag=value` global tokens before normal command dispatch, including trailing tokens; `--restore=<key>` is the explicit upstream-supported exception. Plain help/version inspection preserves exact caller argv, matching upstream. These exceptions are top-level only: global flags for `batch` belong before `batch`, and row-local equals forms fail validation before dispatch.
|
|
192
195
|
- A bare `--no-sandbox` in the command slot, or an option position after `open` / `goto` / `navigate`, fails validation before dispatch with Chromium launch-argument guidance. The former is an unknown upstream command; navigation ignores the latter. Pass the switch as a separate `--args` value and use `sessionMode: "fresh"` for launch changes. For native batches, `--args` belongs on the outer call before `batch`, not in a row. Other commands keep their native literal text, script, select, path and option-value handling; top-level help/version inspection remains unchanged.
|
|
@@ -202,37 +205,38 @@ URL-less `open` is executed as native `get url`, including effective raw/stdin b
|
|
|
202
205
|
|
|
203
206
|
<!-- agent-browser-playbook:start shared-guidelines -->
|
|
204
207
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
205
|
-
-
|
|
208
|
+
- Choose agent_browser for one native command, native batch --bail with JSON-array stdin for a known fixed sequence, or agent_browser_code for loops, branches, and aggregation. Return to the model when a fresh observation needs judgment. Neither tool is a named recipe registry.
|
|
209
|
+
- agent_browser_code takes { code, session?, namespace?, timeoutMs?, outputPath? }. JavaScript globals are fresh for every call; the selected browser persists and is shared with agent_browser. Use await browser({ args, stdin?, timeoutMs? }), inspect result.success, and emit selected JSON instead of whole envelopes. Use emitImage(result.imageObservations[0]) only when image inspection is needed; image handles belong to the current code call. No imports or host filesystem/network/process APIs are exposed.
|
|
206
210
|
- Standard workflow: open the page, snapshot -i, interact using current @refs from that snapshot, and re-snapshot after navigation, scrolling, rerendering, or other major DOM changes because refs are page-scoped; the wrapper fails mutation-prone stale/recycled refs before upstream can silently target a different current-page element. On dense pages, use wrapper-side snapshot -i --search <text> or snapshot -i --filter role=<role> to render matching refs while preserving the full ref map in details.refSnapshot, add snapshot --viewport when scroll position or above/below-fold context matters, and add snapshot --diff when a quick before/after ref-map delta would prevent reading a full spill file.
|
|
207
211
|
- For ordinary forms from one snapshot, batch multiple fill @refs before the submit/click step to avoid serial tool calls; if a fill may autosubmit, navigate, or rerender later fields, split the flow and refresh refs first.
|
|
208
212
|
- Do not use browser automation to drive public search-engine forms such as Google for discovery; headless jobs that type a query and press Enter can be redirected to anti-bot or CAPTCHA pages. Prefer agent_browser_web_search for live discovery, then agent_browser on a target URL. Do not attempt CAPTCHA bypass.
|
|
209
213
|
- Snapshot choice: prefer snapshot -i for routine clicks/fills (interactive @refs, main-content-first). Use snapshot --compact when you need a denser same-page tree without full spill; use full snapshot (no -i) only when you need the complete accessibility tree. Re-snapshot after navigation or major DOM changes. When snapshot -i compacts because the tree is oversized, scan visible output for Omitted high-value controls and optional details.data.highValueControlRefIds before opening the spill file: those list bounded searchboxes, textboxes, comboboxes, buttons, named action links, tabs, checkboxes, radios, options, and menuitems that did not fit the key/other ref previews.
|
|
210
214
|
- When a visible text or accessible-name target should survive ref churn, prefer find locators such as role, text, label, placeholder, alt, title, or testid with the intended action instead of guessing a CSS selector.
|
|
211
|
-
- For desktop or host-controlled rich inputs, if
|
|
215
|
+
- For desktop or host-controlled rich inputs, if agent_browser_action fill misses, refresh refs and prefer a current editable @ref from details.richInputRecovery or the latest snapshot; focus or click that ref, then use keyboard type for framework-controlled editors that require real key events. keyboard inserttext is paste-like and can change a DOM value without updating application state, so use it only when later application-state evidence proves the edit was accepted. Do not auto-submit with Enter or a submit button unless the user flow explicitly calls for it.
|
|
212
216
|
- Do not assume Playwright selector dialects such as text=Close or button:has-text('Close') are supported wrapper syntax unless current upstream agent-browser behavior has been verified.
|
|
213
217
|
- For authenticated or user-specific content explicitly requested by the user, such as feeds, inboxes, account pages, or private dashboards, use a real profile only when the user/config asks for it or profiles have been inspected; do not assume --profile Default exists on every machine. Do not use a real profile for public pages just because they are dashboards. Treat visible page content from real profiles as model-visible transcript data. On macOS, copied Chrome profiles may omit encrypted cookies, so profile selection alone is not proof of authentication; verify the target page and use a user-approved headed login once when needed. Use --auto-connect only if profile-based reuse is unavailable or the task is specifically about attaching to a running debug-enabled browser. If profile/user-data-dir resolution fails, stop retrying opens, run profiles and/or doctor through agent_browser, then report what the user needs to configure.
|
|
214
218
|
- Ordinary bare calls share one native browser within your root Pi parent/subagent group; unrelated roots browse independently. Coordinate multi-call navigation inside your group with existing subagent/intercom tools. Explicit native session defaults still win. Pi quit does not close group browsers; never close an unrelated browser.
|
|
215
219
|
- When using launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --ca-cert, --no-ca-cert, --enable, --executable-path, --webgpu, --no-webmcp, --init-script, --idle-timeout, --args, --user-agent, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), put them on the first command for that session. If you intentionally use an explicit --session, keep using that same explicit session for follow-ups.
|
|
216
|
-
-
|
|
220
|
+
- Cooperating Pi processes serialize each complete browser operation and code cell by native socket context, namespace, and session, including target/ref helpers and state updates. Different identities remain concurrent; namespace-wide close waits for matching operations. All participating Pi instances must load this release; humans, third-party clients, and older extensions are outside that coordination. It is not a transaction or rollback. For raw batches whose later content depends on navigation, use exact batch --bail or split the calls.
|
|
217
221
|
- After a successful `connect`, `--cdp`, or enabled `--auto-connect` call, verify with get url and keep using the resulting session without repeating the attach flag. The wrapper remembers that attachment across active-branch reload/resume and live-checks the URL before later page reads/interactions because an attached browser can drift externally; caller config, file access, launch arguments, and environment pass through unchanged. A successful close clears the marker. When several named sessions share one Chrome, pass --pin-tab once (AGENT_BROWSER_PIN_TAB) so a closed bound tab fails as tab_gone instead of acting on a neighbor; recover with tab new or tab list. --no-pin-tab turns the sticky pin off. tab list includes each tab's CDP targetId, accepted as a tab ref.
|
|
218
222
|
- If you already used the implicit session and now need launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --ca-cert, --no-ca-cert, --enable, --executable-path, --webgpu, --no-webmcp, --init-script, --idle-timeout, --args, --user-agent, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), retry with top-level sessionMode set to fresh or pass an explicit --session for the new launch; never pass --session-mode inside args. After a successful unnamed fresh launch, later auto calls follow that new session.
|
|
219
223
|
- For WebGPU pages, use args ["--webgpu", "open", "<url>"] on a fresh local browser launch; use doctor --webgpu (or --headed on Linux/Windows capture paths) to prove rendering before trusting a non-black screenshot. WebGPU cannot be combined with --cdp, --auto-connect, or provider launches unless --webgpu false overrides an enabled config/environment default.
|
|
220
224
|
- For experimental WebMCP page tools, use webmcp list, then webmcp invoke <tool> with --params and optional --frame/--detach/--timeout; use webmcp result or cancel for detached calls. Locally managed Chrome enables WebMCP by default; a positive navigation hint means the page has tools to list. --no-webmcp is launch-scoped and requires a fresh session; invoke/result/cancel can mutate or navigate, so refresh snapshot refs afterward.
|
|
221
225
|
- For --allowed-domains, use a fresh local Chrome context. Upstream rejects CDP/auto-connect, profiles, restore/state replay, direct-page providers, iOS/Safari, and startup/profile Chrome args because they cannot guarantee containment; Chromium also disables RTCPeerConnection while the allowlist is active.
|
|
222
|
-
- For React introspection, launch the page with --enable react-devtools before first navigation, then use react tree, react inspect <fiberId>,
|
|
226
|
+
- For React introspection, launch the page with --enable react-devtools before first navigation, then use react tree, react inspect <fiberId>, agent_browser_source candidates for local UI source hints, react renders start/stop, or react suspense; agent_browser_source is experimental and reports confidence/evidence instead of guaranteed DOM-to-file mappings. For failed fetches and APIs, agent_browser_network_source (experimental) correlates failed network requests with initiator metadata and bounded workspace URL literals—candidates only, not definitive blame. Use vitals [url] for Core Web Vitals and hydration timing, and pushstate <url> for client-side SPA navigation.
|
|
223
227
|
- For first-navigation setup, use open without a URL plus network route --resource-type <csv>, cookies set --curl <file>, or --init-script/--enable before navigate/opening the target page.
|
|
224
228
|
- For stateful browser context work, use auth save --password-stdin with the tool stdin field for credentials, auth list/show/delete/remove for local auth-profile maintenance, auth login when you need the browser to fill a saved profile, and state save/load/list/show/rename/clear/clean for upstream saved-state lifecycle. State paths, restore identifiers, wrapper-prefixed sessions, and all upstream list rows remain available; credential values inside cookie/storage/auth payloads are still redacted from presentation.
|
|
225
229
|
- Upstream restore sessions periodically autosave cookies and localStorage while the browser stays open, including page-driven background changes; AGENT_BROWSER_AUTOSAVE_INTERVAL_MS controls the interval (30000 by default; 0 disables periodic saves but keeps save-on-close), while the never value for --restore-save disables automatic saves for that restore session. For wrapper-owned headed launches, the wrapper defaults the interval to 0 because upstream 0.33.2 collects multi-origin storage through visible temporary tabs, then records and reapplies the effective launch-time value to helpers and follow-ups so daemon configuration remains stable. Native close still saves, but direct window close can lose newer state because upstream exempts headed browsers from idle shutdown; set AGENT_BROWSER_AUTOSAVE_INTERVAL_MS before launch when periodic preservation matters, because changing it on a running wrapper-owned headed session requires close plus a fresh launch.
|
|
226
230
|
- For batch chains that touch cookies, storage, auth, or other secret-bearing commands, use details.batchSteps for per-step artifacts, categories, spill paths, and full structured errors; top-level details.data on batch is only a compact redacted step matrix (success, argv-redacted command, redacted result or scrubbed error text) built from the same presentation rules as standalone calls.
|
|
227
|
-
- For non-core families, pass current upstream commands through the native tool directly: network requests, network route <url>, network har start/stop [path], diff snapshot, diff screenshot --baseline <file>, diff url <u1> <u2>, trace start, trace stop [path], profiler start, profiler stop [path], record start <path>, record stop, console/errors [--clear], highlight <selector>, inspect, clipboard read, clipboard write <text>, clipboard copy/paste, stream enable/disable/status, dashboard start/stop, device list for iOS simulator inventory, and chat <message>. For compact network requests output, prefer details.nextActions for request detail, route-mock diagnostics, actionable failed-request
|
|
228
|
-
- For Electron desktop apps,
|
|
231
|
+
- For non-core families, pass current upstream commands through the native tool directly: network requests, network route <url>, network har start/stop [path], diff snapshot, diff screenshot --baseline <file>, diff url <u1> <u2>, trace start, trace stop [path], profiler start, profiler stop [path], record start <path>, record stop, console/errors [--clear], highlight <selector>, inspect, clipboard read, clipboard write <text>, clipboard copy/paste, stream enable/disable/status, dashboard start/stop, device list for iOS simulator inventory, and chat <message>. For compact network requests output, prefer details.nextActions for request detail, route-mock diagnostics, actionable failed-request agent_browser_network_source, filtering, clearing the aggregate buffer before repro, or HAR capture follow-ups instead of guessing request-id syntax. Artifact-producing commands report details.artifacts and verification state; long-running starts such as stream, dashboard, trace/profiler, and record should be paired with the matching stop/disable command when the task is done; stream enable already-enabled outcomes are treated as idempotent success with status/disable follow-ups.
|
|
232
|
+
- For Electron desktop apps, enable agent_browser_electron through agent_browser_tools for wrapper-owned discovery, isolated launch, status, compact probe, and cleanup: list first, treat likely-sensitive annotations as hints rather than enforcement, launch with the default snapshot handoff unless handoff: "tabs" is the safer diagnostic starting point, use agent_browser_electron probe or snapshot -i/agent_browser_qa attached:true for current-session state, and always cleanup the returned launchId when done. agent_browser_electron launch uses an isolated temporary profile; it does not reuse the app's normal signed-in profile or attach to an already-running authenticated app. For signed-in local app state, host-launch the normal app with --remote-debugging-port when appropriate, then use raw args connect <port|url>; after connect, run get url to verify the active target before page-content reads, inspect tab list, select the stable tab id such as tab t2, verify it again with get url, then run a condition wait or snapshot -i before using refs. close commands (`close`, `quit`, or `exit`) only close the browser/CDP session; leave manually launched app shutdown, profile cleanup, and explicit artifacts to the host owner.
|
|
229
233
|
- For provider or specialized app workflows, load version-matched upstream guidance with skills get agentcore|electron|slack|dogfood|vercel-sandbox|derive-client through the native tool; add --full when you need references/templates, and use skills get --all only for broad skill audits. Use derive-client when recording HAR traffic to generate a standalone API client; prefer network har start (text bodies by default) or network har start --content all|none before multi-step capture. For accessibility audits use a11y or a11y --tags wcag2a,wcag2aa (CDP browsers only). Hosted sandbox workflows should use upstream @agent-browser/sandbox helpers outside this wrapper. Provider launches such as -p ios, --provider browserbase/kernel/browseruse/browserless/agentcore, and iOS --device are upstream-owned setup paths; use sessionMode fresh when switching providers and expect external credentials or local Appium/Xcode setup to be required.
|
|
230
234
|
- For dialogs and frames, use dialog status/accept/dismiss and frame <selector|main> through native args; dialog commands and eval snippets that look like alert/confirm/prompt/dialog triggers are shorter-bounded than normal browser calls, and timed-out dialog-like interactions may add inspect-dialog-after-timeout, dismiss-dialog-after-timeout, or recover-fresh-session-after-dialog-timeout nextActions. When --confirm-actions produces a pending confirmation, use details.nextActions or exact confirm <id> / deny <id> calls instead of inventing ids.
|
|
231
|
-
- If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab <tab-id-or-label> / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load <state>, wait --download, or
|
|
235
|
+
- If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab <tab-id-or-label> / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load <state>, wait --download, or agent_browser_qa attached:true; for disappearance checks, use wait --fn predicates instead of stale upstream-help examples like wait <selector> --state hidden. Use agent_browser_electron probe/status for wrapper-owned launch health or target mismatch. Fixed waits are a last resort: their duration is positional (wait <ms>, not wait --time <ms>). Use explicit --timeout or top-level timeoutMs for legitimately slow waits, and treat a successful payload like "waited":"timeout" as elapsed time only—verify completion with an observed condition, fresh snapshot, or screenshot.
|
|
232
236
|
- For feed, timeline, or inbox reading tasks, focus on the main timeline/list region and read the first item there rather than unrelated composer or sidebar content.
|
|
233
237
|
- For read-only browsing tasks, use read <url> for documentation or other unstructured text without requiring a Chrome page, or read with no URL for rendered active-tab DOM. Prefer the current snapshot, structured ref labels, getters, or scoped eval --stdin when you need interactive structure or targeted page state. Only click into media viewers, detail routes, or new pages when the current view does not contain the needed information.
|
|
234
238
|
- For downloads, prefer download <selector> <path> when an element click should save a file; native download owns the click and download, including loopback links, generated Blob exports, and redirects. Do not rely on click alone when you need the downloaded file on disk.
|
|
235
|
-
- On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll <selector> <dir> [px|percent]. For native selects, use select <selector> <value...> (or
|
|
239
|
+
- On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll <selector> <dir> [px|percent]. For native selects, use select <selector> <value...> (or agent_browser_action select) instead of clicking option refs; for custom comboboxes, a click/agent_browser_action may only focus the field, so re-snapshot and use keyboard type <text> for focused input, press ArrowDown or press Enter, or visible option refs. Raw type requires both <selector> and <text>.
|
|
236
240
|
- When using eval --stdin, scope checks and actions to the target element or route whenever possible instead of relying on broad page-wide text heuristics.
|
|
237
241
|
- When using eval --stdin for extraction, pass the JavaScript through the native tool stdin field, not as an extra args token after --stdin, and return the value you want instead of relying on console.log as the primary result channel. Prefer plain expressions like ({ title: document.title }) or explicitly invoked functions like (() => ({ title: document.title }))(); use outputPath when the eval/get/snapshot data should be saved as a durable local file, but never reuse a screenshot, download, recording, or other browser artifact destination as outputPath. If a function-shaped snippet returns {}, details.evalStdinHint may warn that the function was serialized instead of called. Local file pages and caller-selected output paths are supported when upstream allows them. If get text on a broad CSS selector surfaces details.selectorTextVisibility or selectorTextVisibilityAll, prefer a visible @ref, a more specific selector, or the inspect-visible-text-candidates nextAction over hidden tab content.
|
|
238
242
|
- When details.pageChangeSummary is present, use changeType and summary as a compact signal for navigation, DOM mutation, confirmations, or artifacts; when nextActionIds is set, match those ids to entries in details.nextActions (or per-step nextActions inside batch) for concrete follow-up payloads instead of inferring from prose alone. Click-dispatch probes require native get attr readback of a temporary marker on the candidate target; unproven ref or XPath identity leaves the native click unverified, not failed. If details.clickDispatch reports a click-dispatch miss, refresh/inspect/retry the real click first; for static local fixtures only, an explicit eval --stdin programmatic .click() can exercise app handlers, but treat it as an untrusted scripted workaround and never use it to bypass stop-before-submit/order/purchase boundaries. If an upstream click failure says the target is covered by another element at the target's click point, use the inspect-overlay-state nextAction to refresh refs and inspect the blocker before deciding whether to retry; do not blindly repeat the blocked click. If a no-navigation click surfaces details.overlayBlockers, inspect the fresh snapshot evidence before using a close/dismiss candidate nextAction; ordinary page chrome without dialog/alertdialog evidence should not trigger this diagnostic.
|
|
@@ -246,35 +250,20 @@ URL-less `open` is executed as native `get url`, including effective raw/stdin b
|
|
|
246
250
|
|
|
247
251
|
## Parameters
|
|
248
252
|
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
```json
|
|
252
|
-
{ "script": "const title = await browser({ args: ['get', 'title'] }); if (!title.ok) throw new Error(title.error); emit(title.data.title ?? title.data.result);" }
|
|
253
|
-
```
|
|
254
|
-
|
|
255
|
-
```json
|
|
256
|
-
{ "args": ["open", "https://example.com"], "stdin": "optional raw stdin content", "outputPath": "logs/result.json", "timeoutMs": 35000, "sessionMode": "auto" }
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
```json
|
|
260
|
-
{ "semanticAction": { "action": "click", "locator": "role", "value": "button", "name": "Export" }, "sessionMode": "auto" }
|
|
261
|
-
{ "semanticAction": { "action": "fill", "selector": "@e1", "text": "prompt text" } }
|
|
262
|
-
{ "semanticAction": { "action": "select", "selector": "#flavor", "value": "chocolate" } }
|
|
263
|
-
```
|
|
253
|
+
`agent_browser` accepts only `args`, optional `stdin`, `outputPath`, `timeoutMs`, and `sessionMode`. Advanced tools take flat fields documented below. Unknown public fields, including the former `script` and `job`, are rejected.
|
|
264
254
|
|
|
265
255
|
```json
|
|
266
|
-
{ "
|
|
267
|
-
{ "electron": { "action": "launch", "appName": "Visual Studio Code", "handoff": "snapshot" } }
|
|
256
|
+
{ "args": ["open", "https://example.com"], "sessionMode": "auto" }
|
|
268
257
|
```
|
|
269
258
|
|
|
270
259
|
### `args`
|
|
271
260
|
|
|
272
261
|
- type: `string[]`
|
|
273
|
-
- required
|
|
262
|
+
- required, non-empty
|
|
274
263
|
- exact CLI args passed after `agent-browser`; this is the 1:1 upstream CLI coverage path for the targeted `agent-browser` version
|
|
275
264
|
- no shell operators
|
|
276
265
|
- do not include the binary name
|
|
277
|
-
-
|
|
266
|
+
- omit `--json` for prose; include it for parseable model-visible JSON (the wrapper injects upstream JSON internally either way)
|
|
278
267
|
- first-call recipe: `open` → `snapshot -i` → `click` / `fill` with current `@eN` refs from that snapshot → `snapshot -i` again after navigation or DOM changes
|
|
279
268
|
|
|
280
269
|
Examples:
|
|
@@ -290,390 +279,148 @@ Examples:
|
|
|
290
279
|
|
|
291
280
|
### Execution directory
|
|
292
281
|
|
|
293
|
-
Relative file operands, `outputPath`, and source/network-source scans use one `pi-change-working-dir` owner reply captured before browser policy and queues;
|
|
294
|
-
|
|
295
|
-
Browser/session/config ownership stays separate. Ordinary calls retain the native project or managed launch root, while an unnamed fresh launch and explicit native config resolve against the execution directory. `details.managedSessionCwd`, when present, records the managed launch root for transcript replay. Pi package config/trust and artifact-store identity remain unchanged. A deleted launch directory returns `validation-error` before dispatch; restore it or explicitly choose fresh/config from the new directory. See [working-directory changes](../README.md#working-directory-changes).
|
|
296
|
-
|
|
297
|
-
### `script`
|
|
298
|
-
|
|
299
|
-
- type: string, maximum 65,536 characters at the public schema and 65,536 UTF-8 bytes at execution
|
|
300
|
-
- optional; mutually exclusive with `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, `electron`, `sessionMode`, and top-level `stdin`
|
|
301
|
-
- intended only for one-shot loops, conditional page branches, and multi-page aggregation where several top-level browser calls and host artifact reads would otherwise be required; use ordinary `args`, `job`, or `qa` for linear workflows
|
|
302
|
-
- executes as an async JavaScript body in a separate Node child with a 64 MiB V8 heap limit, Node permission mode, VM string/WebAssembly code generation disabled, an empty environment, no imports or dynamic imports, and no exposed host objects/functions; common host capabilities such as `process`, `require`, filesystem, network, and timers are absent, while context-native `eval` / `Function` code generation is disabled
|
|
303
|
-
- exposes only `browser(params)` and `emit(value)` as null-prototype task-specific functions. The ordinary context-native JavaScript intrinsics remain available for local computation.
|
|
304
|
-
- `browser({ args, stdin?, timeoutMs? })` runs each inner call through the same full native-tool executor used by a top-level `args` call. It returns a JSON clone shaped as `{ ok, data, details?, error?, failureCategory?, nextActions?, resultCategory, successCategory?, summary, text }`; callers must check `ok`. Wrapper-verified parse-valid compact spills may be rehydrated into `data` before the clone is returned, and ordinary structured/prose redaction still applies. `summary` and `text` are bounded before they enter IPC or `scriptSteps`; if the complete serialized envelope still exceeds the per-message limit, that inner call resolves to a bounded `{ ok: false, failureCategory: "upstream-error" }` envelope instead of terminating the sandbox protocol. Browser-bearing `nextActions` are exposed only when their params pass script policy after the wrapper removes the current isolated `--namespace "" --session piab-script-*` prefix; unsupported top-level modes and local commands are omitted, while artifact-path metadata may remain.
|
|
305
|
-
- `emit(value)` accepts JSON-compatible data. One emission becomes `data` directly; multiple emissions become a `data` array in emission order. With no emission, the async body’s return value becomes `data`; `data` is omitted when the body returns nothing. When any emission exists, the body return value is ignored.
|
|
306
|
-
- browser calls are serialized even when source uses `Promise.all`; the maximum is 25 attempted calls. Source and final serialized output are each capped at 64 KiB; cumulative child/parent IPC is bounded. Final data is redacted, serialized without pretty-print amplification, and byte-checked again before presentation; excessive depth or post-redaction growth becomes `failureCategory: "validation-error"` with data omitted.
|
|
307
|
-
- default top-level `timeoutMs` is 120,000 ms for script; the hard ceiling is 300,000 ms. Every inner timeout is clamped to the remaining outer deadline. Abort, timeout, Pi branch change, quit, reload, and child failure cascade to the active inner call, wait for isolated-session cleanup, then terminate and reap the sandbox child with a bounded SIGTERM/SIGKILL sequence.
|
|
308
|
-
- before the first accepted inner browser call, the wrapper appends a model-invisible Pi custom-entry lease containing only a strict wrapper-generated `piab-script-<uuid>` session name, exact close argv, `launchAttempted: true`, and cleanup state. Script mode therefore requires a persisted Pi session and fails validation under `--no-session`.
|
|
309
|
-
- every invocation uses a unique wrapper-owned session in the empty canonical namespace, with managed restore disabled. It never reads, replaces, or updates the extension-managed implicit conversation session. Inner identity/lifecycle/attachment/local commands, nested `batch`, nested top-level modes, `--session`, `--namespace`, `--config`, profile/state/restore/provider/CDP/raw-args/init/extension launch controls, and local/sessionless commands are rejected before upstream spawn. Every script-owned helper and cleanup subprocess also clears ambient `AGENT_BROWSER_*` and standard proxy variables and uses an empty private temporary native config, bypassing HOME/project profile defaults, before wrapper-owned namespace, timeout, and compatibility values are applied.
|
|
310
|
-
- the wrapper closes the isolated session in `finally`, including after error, timeout, abort, or branch change. `session_tree` and `session_shutdown` abort active scripts and await their normal cleanup before branch restoration or shutdown cleanup continues. On session start or branch change, exact non-closed leases from the active branch are then retried; forged names or close argv are ignored.
|
|
311
|
-
- details include `scriptRun.callCount`, disjoint `successfulCallCount`, ordinary dispatched `failedCallCount`, `preDispatchRejectedCallCount`, and emission/timeout/abort fields plus bounded `scriptSteps` category/summary rows. Any pre-dispatch policy or validation rejection fails the top-level tool result even when source handles the returned `{ ok: false }` envelope. When a browser launch was attempted, `scriptSession: { sessionName, cleanup: "closed", closeCommandArgs, launchAttempted: true }` and the compact prose both confirm successful cleanup. Uncaught user-source exceptions use `failureCategory: "script-error"`. Cleanup failure overrides every script outcome with `failureCategory: "cleanup-failed"`, `scriptSession.cleanup: "failed"`, a redacted error, and `nextActions[0].id: "close-script-session-after-cleanup-failure"` carrying the exact close args.
|
|
312
|
-
- this is not a reusable recipe layer: there is no script name, registry, persistence, cross-call state, host module import, or workflow versioning surface. Pi approval applies to the one visible top-level tool input, which may issue up to 25 inner calls. The collapsed Pi tool row renders a bounded terminal-safe source preview with line breaks marked as `↵`; expand the row to inspect the full terminal-safe script before approving it. Rendering normalizes JavaScript CR, U+2028, and U+2029 line terminators to visible newlines and replaces stripped terminal/directional/zero-width controls with a visible marker so sanitization cannot join a line comment to executable source or silently hide removed characters.
|
|
313
|
-
|
|
314
|
-
Example with a conditional branch and aggregation:
|
|
315
|
-
|
|
316
|
-
```json
|
|
317
|
-
{
|
|
318
|
-
"script": "const out = []; for (const url of ['https://example.com/a', 'https://example.com/b']) { const opened = await browser({ args: ['open', url] }); if (!opened.ok) throw new Error(opened.error); const probe = await browser({ args: ['eval', '--stdin'], stdin: \"({ blocked: Boolean(document.querySelector('[role=dialog]')), title: document.title })\" }); if (!probe.ok) throw new Error(probe.error); if (probe.data.result.blocked) { const closed = await browser({ args: ['click', '[role=dialog] button'] }); if (!closed.ok) throw new Error(closed.error); } out.push({ url, title: probe.data.result.title }); } emit(out);"
|
|
319
|
-
}
|
|
320
|
-
```
|
|
321
|
-
|
|
322
|
-
### `semanticAction`
|
|
323
|
-
|
|
324
|
-
- type: object
|
|
325
|
-
- optional; mutually exclusive with `script`, `args`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, and `electron` (omit all of them when using this field)
|
|
326
|
-
- top-level tool input only: `batch` stdin remains upstream argv arrays; express find steps inside batch as string arrays such as `["find","role","button","click","--name","Export"]`, not nested `semanticAction` objects
|
|
327
|
-
- thin intent schema compiled by this wrapper into existing upstream commands; locator actions compile to `find`, direct selector/ref `click` / `check` / `fill` compile to the matching upstream command, and native dropdown selection compiles to `select <selector> <value...>`; behavior and locator/selector semantics stay upstream-owned
|
|
328
|
-
- supported actions: `click`, `fill`, `check`, `select`
|
|
329
|
-
- supported locators for `click` / `fill` / `check`: `role`, `text`, `label`, `placeholder`, `alt`, `title`, `testid`
|
|
330
|
-
- optional `selector` is accepted for direct `click`, `check`, and `fill` targets (including current `@refs`); do not combine it with `locator`, `value`, `role`, or `name`. For `fill`, `text` is still required.
|
|
331
|
-
- `semanticAction` does not expose `uncheck` because upstream `find` actions are only `click, fill, check, hover, text`; use raw `args: ["uncheck", <selector-or-ref>]` after a stable selector or current snapshot ref
|
|
332
|
-
- for locator actions, `value` is the locator argument (for example ARIA role token `"button"`, label text, or visible substring), must be a non-empty string after trim; for `locator: "role"`, callers may provide `role` instead of redundant `value`
|
|
333
|
-
- `fill` requires non-empty `text` (compiled as the trailing value argument to `find`); `values` is only available for `select` in the tool schema, matching runtime validation
|
|
334
|
-
- `select` accepts either a non-empty direct `selector` plus `value` (one option value) or `values` (one or more option values), or an active-session semantic locator that the wrapper resolves through a fresh `snapshot -i` to exactly one current visible native dropdown ref. For `locator: "role"`, set `role` to `combobox` or `listbox`, set `name` to the accessible name, and provide option `value` / `values`. For `locator: "label"`, `value` is the accessible label text and `values` contains the option value(s). Locator-based select fails before action when no active browser exists or the current snapshot is missing/ambiguous; no fuzzy target is selected.
|
|
335
|
-
- optional `name` is only valid with `locator: "role"` and compiles to `--name <name>` after the action (and after `text` for `fill` when present)
|
|
336
|
-
- optional `role` is accepted only when `locator` is `role`; it may replace `value`, and must equal `value` if both are set
|
|
337
|
-
- optional `session` is an upstream session name; when set, compilation prepends `--session <session>` before the compiled `find`, direct selector/ref command, or `select` command so the shorthand targets that named browser context instead of the managed default; this is independent of top-level `sessionMode`, which only injects or rotates the extension-managed implicit session when the planned argv does not already start with `--session` (see `buildExecutionPlan` in `extensions/agent-browser/lib/runtime.ts`). On successful unified results, `details.sessionName` matches that name and `usedImplicitSession` is `false` because the call named upstream directly rather than consuming the extension-managed implicit session slot.
|
|
338
|
-
|
|
339
|
-
Compilation (then `--json` and session handling apply like any other call):
|
|
340
|
-
|
|
341
|
-
| Fields | Compiled `args` (conceptually) |
|
|
342
|
-
| --- | --- |
|
|
343
|
-
| `click` or `check` + non-`role` locator | `["find", <locator>, <value>, <action>]` |
|
|
344
|
-
| `click` / `check` / `fill` + `role` or `value` + optional `name` | `["find","role",<role-or-value>,<action>]` plus `["--name",<name>]` when `name` is set; active-session fill may pre-resolve to `fill @ref <text>` when one exact editable current ref matches |
|
|
345
|
-
| `fill` | `["find",<locator>,<value>,"fill",<text>]` plus optional `["--name",<name>]` after `text` when `locator` is `role` and `name` is set |
|
|
346
|
-
| `click` / `check` / `fill` + `selector` | `[<action>,<selector>]` (plus `<text>` for `fill`) |
|
|
347
|
-
| `select` + `selector` + `value` / `values` | `["select",<selector>,<value...>]` |
|
|
348
|
-
| `select` + `locator: "role"` + `role: "combobox"` or `"listbox"` + `name` + `value` / `values` | resolves one exact current visible ref, then executes `["select","@ref",<value...>]` |
|
|
349
|
-
| `select` + `locator: "label"` + label `value` + option `values` | resolves one exact current visible combobox/listbox ref, then executes `["select","@ref",<value...>]` |
|
|
350
|
-
| any supported action + `session` | prepends `["--session",<session>]` before the compiled argv |
|
|
351
|
-
|
|
352
|
-
When `semanticAction` compiles successfully, `details.compiledSemanticAction` echoes `{ action, locator, args }` for `find` actions, `{ action, selector, args }` for direct selector/ref click/check/fill actions, or `{ action: "select", selector?, locator?, values, args }` for `select`, with `args` redacted the same way as other invocation details. Expect it on the initial wrapper validation return (when that path still builds the early `details` object) and on the unified result after `agent-browser` runs. It is omitted when the call used `args` only, when compilation never produced argv, and on some in-`execute` error returns that attach a slimmer `details` shape before the unified merge (for example certain session-plan, stdin-contract, tab-pinning, or missing-binary guard paths); compare `extensions/agent-browser/index.ts` where `compiledSemanticAction` is assigned. For active sessions, role/name `click`, `check`, guarded `fill`, and locator-based `select` semantic actions may be resolved through one fresh `snapshot -i` to a current visible `@ref` before execution; for a caller-owned explicit `semanticAction.session`, the wrapper first verifies the live URL, and the same process-local per-session critical section covers that probe, snapshot, and main action; fill only resolves when one exact editable `combobox`, `searchbox`, or `textbox` ref matches. This avoids hidden duplicate matches stealing an upstream `find` action. In that case `details.compiledSemanticAction` keeps the original semantic target fields; locator-based `select` reports the resolved `select @ref` in its `args`, while other resolved actions keep their original compiled `find` args and expose the executed ref action through `details.effectiveArgs`.
|
|
282
|
+
Relative file operands, `outputPath`, and source/network-source scans use one `pi-change-working-dir` owner reply captured before browser policy and queues; code calls inherit it. The integration uses the synchronous `pi-change-working-dir:resolve-execution-cwd` event with `{ sessionManager, result?: { cwd, error? } }`. Owner errors or invalid replies fail validation. No owner means native `ctx.cwd`; a known older owner identified through public tool/command package provenance requires an update and Pi restart. No private directory journal is read.
|
|
353
283
|
|
|
354
|
-
|
|
284
|
+
Browser/session/config ownership stays separate. Ordinary calls retain the native project or managed launch root, while an unnamed fresh launch and explicit native config resolve against the execution directory. `details.managedSessionCwd`, when present, records the managed launch root for transcript replay, including ordered code transitions. Pi package config/trust and artifact-store identity remain unchanged. A deleted launch directory returns `validation-error` before dispatch; restore it or explicitly choose fresh/config from the new directory. See [working-directory changes](../README.md#working-directory-changes).
|
|
355
285
|
|
|
356
|
-
|
|
286
|
+
### `agent_browser_code`
|
|
357
287
|
|
|
358
|
-
|
|
288
|
+
Strict JSON input: `{ code, session?, namespace?, timeoutMs?, outputPath? }`. `code` is a nonempty async JavaScript body, capped at 65,536 characters by schema and 65,536 UTF-8 bytes at execution. No `args`, top-level `stdin`, or `sessionMode` fields are accepted here.
|
|
359
289
|
|
|
360
|
-
|
|
290
|
+
- `session` selects a nonempty native session name. Omission follows the same native configured/root/managed default as direct calls; the last explicit direct call does not change that default slot.
|
|
291
|
+
- `namespace` selects a native namespace; `""` explicitly selects the default namespace. The parent resolves one fixed socket/namespace/session identity before executing the cell.
|
|
292
|
+
- `timeoutMs` is a positive integer: 120,000 ms default, 300,000 ms maximum. Admission, local queue waits, the shared lock, and inner work use the cell deadline; each inner timeout is clamped to the remaining budget.
|
|
293
|
+
- `outputPath` saves result data through the ordinary artifact-aware writer and must not alias browser artifacts.
|
|
361
294
|
|
|
362
|
-
|
|
295
|
+
The browser persists between code and direct calls. JavaScript variables do not. Native batch, auth, profile, state, connect, local commands, and close remain available through `browser()`, using ordinary parent-side config/environment and ownership rules. Inner commands cannot select another session/namespace or run `close --all`; use `agent_browser` outside the cell for namespace-wide cleanup or a fresh launch. Closing inside a cell closes only its fixed browser identity. A later call may relaunch it according to normal native rules. If it closes a wrapper-managed fresh browser and reopens that fixed name, the reopen is an explicit caller-owned session; the automatic managed slot has already rotated. Close that reopened name explicitly when finished.
|
|
363
296
|
|
|
364
|
-
|
|
365
|
-
{ "semanticAction": { "action": "click", "locator": "role", "value": "button", "name": "Export" } }
|
|
366
|
-
{ "semanticAction": { "action": "click", "locator": "role", "role": "button", "name": "Continue without Signing In" } }
|
|
367
|
-
{ "semanticAction": { "action": "click", "locator": "text", "value": "Close" } }
|
|
368
|
-
{ "semanticAction": { "action": "fill", "locator": "label", "value": "Email", "text": "user@example.com" } }
|
|
369
|
-
{ "semanticAction": { "action": "fill", "selector": "@e1", "text": "prompt text" } }
|
|
370
|
-
{ "semanticAction": { "action": "click", "selector": "#submit" } }
|
|
371
|
-
{ "semanticAction": { "action": "select", "selector": "#flavor", "value": "chocolate" } }
|
|
372
|
-
{ "semanticAction": { "action": "select", "selector": "#multi", "values": ["dark", "compact"] } }
|
|
373
|
-
{ "semanticAction": { "action": "select", "locator": "role", "role": "combobox", "name": "Flavor", "value": "chocolate" } }
|
|
374
|
-
{ "semanticAction": { "action": "select", "locator": "label", "value": "Flavor", "values": ["chocolate"] } }
|
|
375
|
-
{ "semanticAction": { "action": "check", "locator": "label", "value": "Remember me" } }
|
|
376
|
-
{ "semanticAction": { "action": "click", "locator": "text", "value": "Close", "session": "named-browser" } }
|
|
377
|
-
```
|
|
297
|
+
#### Code globals and output
|
|
378
298
|
|
|
379
|
-
|
|
380
|
-
|
|
381
|
-
-
|
|
382
|
-
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `qa`, `sourceLookup`, `networkSourceLookup`, and `electron`
|
|
383
|
-
- top-level tool input only; do not nest `job` inside `batch` stdin
|
|
384
|
-
- constrained orchestration only: every step compiles to existing upstream `batch` argv and the compiled plan is echoed as `details.compiledJob`
|
|
385
|
-
- optional `failFast` boolean; defaults to `true`, compiling to upstream `batch --bail` so later mutating job steps do not run after an earlier required step fails. Set `failFast: false` only when you explicitly want upstream batch's continue-after-error behavior and every later step remains safe if an earlier navigation fails; otherwise the wrapper may require fail-fast behavior or split calls.
|
|
386
|
-
- there is no separate reusable named “browser recipe” extension surface above `job`, `qa`, and raw `batch` yet; the closed `RQ-0068` decision, evidence bar, and revisit criteria are in [`ARCHITECTURE.md`](ARCHITECTURE.md#no-reusable-recipe-layer-yet) and [`SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md)
|
|
387
|
-
- supported steps (`open.loadState` can emit an extra readiness row immediately after the `open`; otherwise each row becomes one upstream `batch` step except paced `type`, which expands to existing focus/keyboard/wait/press rows; `click` / `fill` accept either `selector` or semantic locator fields, while `type` / `select` pass `selector` through as the same argv token shape standalone upstream commands would use, including `@refs`):
|
|
388
|
-
- `open` with `url`; optional `loadState` (`domcontentloaded`, `load`, or `networkidle`) inserts `wait --load <state>` immediately after the open when the next step needs page-readiness evidence
|
|
389
|
-
- `click` with `selector` or semantic locator fields
|
|
390
|
-
- `fill` with `selector` and `text`, or semantic locator fields plus `text`
|
|
391
|
-
- `type` with `text`; optional `selector` focuses the target first, optional `delayMs` emits per-character `keyboard type` plus `wait` rows, and optional `press` emits a final `press <key>` row such as `Enter`. Delayed typing is capped at 200 characters per step. Model-visible batch prose compacts generated per-character rows; `details.batchSteps` and `details.compiledJob.steps` retain the full bounded row list.
|
|
392
|
-
- `select` with `selector` plus either `value` or `values` (one or more option values; compiled as `select <selector> <value...>`)
|
|
393
|
-
- `wait` with positive integer `milliseconds`
|
|
394
|
-
- `assertText` with `text` only, without selector or locator fields (compiled as passive `wait --text <text>`)
|
|
395
|
-
- `assertUrl` with exact URL or `*` / `**` glob-style `url` pattern (compiled as `wait --url <url-or-pattern>`)
|
|
396
|
-
- `waitForDownload` with `path` (compiled as `wait --download <path>`)
|
|
397
|
-
- `snapshot` (compiled as `snapshot -i`; useful between mutation-prone steps before reusing current refs)
|
|
398
|
-
- `screenshot` with `path`
|
|
399
|
-
|
|
400
|
-
Clicks can stale subsequent `@refs`; split the job and take a fresh snapshot before using those refs.
|
|
401
|
-
|
|
402
|
-
**Navigation assertions are explicit only.** `job` never treats a successful `click` (or a `select` / submit-style interaction that may navigate) as proof that the expected next page loaded. Top-level `click` may still surface optional `details.navigationSummary` or `pageChangeSummary` hints for operators, but compiled `job` / `batch` steps do **not** auto-insert `assertUrl` or `assertText` after clicks—there is no deterministic expected URL source without caller intent. Use `open.loadState` to wait for initial page readiness after an `open`; after any later navigation-prone step (link/submit clicks, checkout or form flows, tab-sensitive UI), add an explicit `assertUrl` with the exact destination URL or a `*` / `**` glob-style pattern that does not already match the starting page, `assertText` for new on-page copy, or both, **before** screenshots or steps that assume the new page state. A broad URL pattern that already matches the current route is a precondition, not proof of navigation.
|
|
403
|
-
|
|
404
|
-
Example (static landing page):
|
|
299
|
+
- `await browser({ args, stdin?, timeoutMs? })` returns a JSON-cloned [canonical observation](#canonical-observation): check `success` before consuming `data`. Native batch is permitted, but nested native batch rows remain unsupported by the ordinary executor.
|
|
300
|
+
- `emit(value)` selects JSON output. One emission becomes `data`; multiple emissions become an ordered array. If there are no emissions, the async body's return value is used; an absent return omits data.
|
|
301
|
+
- `emitImage(observation.imageObservations[0])` selects a verified real image handle from this cell. Paths, arbitrary objects, and handles from earlier cells are not image inputs. Selection rechecks file size/mtime and ordinary inline-image eligibility. Duplicate selection is coalesced. At most eight selected images and 20 MiB total are allowed, subject to the configurable inline bound (5 MiB per image by default via `PI_AGENT_BROWSER_INLINE_IMAGE_MAX_BYTES`). Selected images are real Pi image content blocks, not JSON strings.
|
|
405
302
|
|
|
406
303
|
```json
|
|
407
304
|
{
|
|
408
|
-
"
|
|
409
|
-
"steps": [
|
|
410
|
-
{ "action": "open", "url": "https://example.com" },
|
|
411
|
-
{ "action": "assertText", "text": "Example Domain" },
|
|
412
|
-
{ "action": "screenshot", "path": ".dogfood/example.png" }
|
|
413
|
-
]
|
|
414
|
-
}
|
|
305
|
+
"code": "const page = await browser({ args: ['get', 'title'] }); if (!page.success) throw new Error(page.error); emit({ title: page.data.title ?? page.data.result });"
|
|
415
306
|
}
|
|
416
307
|
```
|
|
417
308
|
|
|
418
|
-
Example (open → fill → click → assert destination → screenshot):
|
|
419
|
-
|
|
420
309
|
```json
|
|
421
310
|
{
|
|
422
|
-
"
|
|
423
|
-
"steps": [
|
|
424
|
-
{ "action": "open", "url": "https://shop.example/checkout" },
|
|
425
|
-
{ "action": "fill", "selector": "#email", "text": "user@example.com" },
|
|
426
|
-
{ "action": "click", "selector": "#continue" },
|
|
427
|
-
{ "action": "assertUrl", "url": "**/shipping" },
|
|
428
|
-
{ "action": "assertText", "text": "Shipping address" },
|
|
429
|
-
{ "action": "screenshot", "path": ".dogfood/shipping.png" }
|
|
430
|
-
]
|
|
431
|
-
}
|
|
311
|
+
"code": "const shot = await browser({ args: ['screenshot', '/tmp/page.png'] }); if (!shot.success) throw new Error(shot.error); emit(shot.artifactVerification); if (shot.imageObservations?.length) emitImage(shot.imageObservations[0]);"
|
|
432
312
|
}
|
|
433
313
|
```
|
|
434
314
|
|
|
435
|
-
|
|
315
|
+
Inner calls are serialized, including calls issued with `Promise.all`. The maximum is 25 attempted calls, 64 KiB emitted JSON, 1 MiB per IPC message, and 8 MiB cumulative IPC. Inner execution keeps full redacted data without full model rendering/spilling/image encoding; only selected outer output is rendered. An observation that cannot fit the bridge limit fails rather than silently discarding recovery or fabricating data.
|
|
436
316
|
|
|
437
|
-
|
|
438
|
-
{
|
|
439
|
-
"args": ["batch", "--bail"],
|
|
440
|
-
"stdin": "[[\"open\",\"https://shop.example/checkout\"],[\"fill\",\"#email\",\"user@example.com\"],[\"click\",\"#continue\"],[\"wait\",\"--url\",\"**/shipping\"],[\"wait\",\"--text\",\"Shipping address\"],[\"screenshot\",\".dogfood/shipping.png\"]]"
|
|
441
|
-
}
|
|
442
|
-
```
|
|
317
|
+
Source runs in a fresh permissioned Node child with an empty child environment, 64 MiB V8 heap ceiling, and a VM context whose string/WebAssembly code generation is disabled. Task globals have null prototypes and JSON-only IPC; no parent promise/object/function is exposed. There are no host filesystem, network, process, import, or timer APIs. This language boundary is separate from native browser capabilities: parent-side browser subprocesses still receive ordinary auth/config/environment. `node:vm` alone is not a security boundary, and this contract is not a universal sandbox certification.
|
|
443
318
|
|
|
444
|
-
|
|
319
|
+
#### Failure and persistence
|
|
445
320
|
|
|
446
|
-
`
|
|
321
|
+
`codeRun` reports `callCount`, `emitCount`, `failedCallCount`, `rejectedCallCount`, and timeout/abort flags. `codeSteps` retains bounded audit rows. Handled browser failures may coexist with successful source completion, but remain model-visible in `failures` with exact recovery evidence and in the failure count. A successful code result therefore does not assert that every inner command succeeded. Invalid inner input shapes make the outer result fail even if source handles their observations. Uncaught source exceptions use `script-error`; deadline and cancellation use `timeout` and `aborted`; output/bridge validation retains its specific failure.
|
|
447
322
|
|
|
448
|
-
|
|
323
|
+
Pi persistence is required: `--no-session` fails before code execution. Before each accepted browser dispatch, the parent appends an ordered `agent-browser-transition` intent; completion appends reducer-consumed state from the same executor as direct calls. This includes target/ref invalidation, attachment, close/relaunch, and existing ownership evidence. Browser state uses existing replay reducers; there is no persistent JavaScript registry. An interrupted intent leaves the target unknown until inspection. If completion persistence fails after a mutation, the failure says the command may already have run; do not retry blindly.
|
|
449
324
|
|
|
450
|
-
|
|
325
|
+
Abort, timeout, branch change, reload, and shutdown stop/reap the child and settle the active inner call before releasing coordination. They do not roll back browser actions or automatically close the selected browser. Existing ownership still controls later shutdown cleanup. The only retained pre-0.7 script machinery reads exact old isolated-session cleanup leases during upgrade; it does not accept old public script execution.
|
|
451
326
|
|
|
452
|
-
### `
|
|
327
|
+
### `agent_browser_action`
|
|
453
328
|
|
|
454
|
-
|
|
455
|
-
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `sourceLookup`, `networkSourceLookup`, and `electron`
|
|
456
|
-
- lightweight preset built on the same batch compiler path as `job`, using `batch --bail` so missing readiness/text/selector assertions stop before slower diagnostics can burn the wrapper watchdog
|
|
457
|
-
- URL form: clears enabled network/console diagnostic buffers first and snapshots any page-error residue after `errors --clear` (native clear can leave residue), then opens `url`, waits with `wait --load <state>` using the resolved `loadState`, adds a bounded 150 ms diagnostic settle when console or page-error checks are enabled so immediate post-load callbacks can report, optionally asserts `expectedText` (string or string array, compiled to bounded visible-text `wait --fn … --timeout 5000` predicates after load) and/or `expectedSelector` (each may be omitted for a load-plus-diagnostics-only smoke), then runs enabled diagnostics: `network requests`, `console`, and `errors` only if preceding batch steps pass. Successful reset-step rows are labeled as reset output. Final page-error rows that match a nonempty post-clear baseline make the check unverified and fail QA; identical rows do not prove that no new error occurred. Multiset subtraction still identifies definitely new rows, which count as page errors. Failed reset commands still fail the batch.
|
|
458
|
-
- URL QA uses its explicitly requested navigation without requiring the previous tab; diagnostic clears remain before `open`. This does not reopen remembered pages for ordinary reads or `qa.attached`.
|
|
459
|
-
- attached form: `qa: { attached: true, expectedText?, expectedSelector?, screenshotPath?, checkNetwork?, checkConsole?, checkErrors?, loadState? }` runs the same waits (including the bounded diagnostic settle when needed), optional assertions, diagnostics, and screenshot against the current attached managed session without opening a URL. It rejects `url` and cannot be used with `sessionMode: "fresh"`; attach first with `electron.launch` or raw `args: ["connect", "<port-or-url>"]`, then run `qa.attached`. Before spawning the diagnostic batch, the wrapper preflights the attached session: `get url` must succeed and return a non-empty page URL. Missing URLs and read failures fail fast with `failureCategory: "validation-error"`, `details.validationError`, and recovery `nextActions` such as `list-tabs-before-qa-attached` and `snapshot-before-qa-attached` instead of running the full QA batch. `file:`, custom-scheme, and other attached targets are accepted. Attached QA does **not** run `network requests --clear`, `console --clear`, or `errors --clear`; `details.compiledQaPreset.checks.diagnosticsResetAtStart` is `false`. Visible text warns that existing diagnostic buffers were preserved only when `checkNetwork`, `checkConsole`, or `checkErrors` is enabled, and those diagnostics may include events from before the QA check.
|
|
460
|
-
- `loadState` is optional and must be `domcontentloaded`, `load`, or `networkidle`; it defaults to `domcontentloaded` so analytics-heavy or long-polling pages do not hang routine QA. Use `networkidle` only when the site is expected to go fully quiet.
|
|
461
|
-
- `checkNetwork`, `checkConsole`, and `checkErrors` default to `true` for URL-opening QA; for `qa.attached` they default to `false` because preserved upstream buffers may predate the current check. Set a field to `true` on `qa.attached` to opt into preserved-buffer diagnostics. Explicit `checkErrors: false` skips error clearing, baseline collection, and the final error read; an otherwise healthy limited check can pass.
|
|
462
|
-
- optional `screenshotPath` adds an evidence screenshot step
|
|
463
|
-
- reports `details.compiledQaPreset` with the compiled batch plan and resolved `loadState`, plus `details.qaPreset` with `{ passed, failedChecks, warnings, summary }`
|
|
464
|
-
- on success with no failed checks, model-visible prose collapses to a compact pass summary (current page URL/title when known, checks run, optional screenshot path plus artifact verification, pointer to `details.qaPreset` and `details.batchSteps` for the full matrix). Failed QA uses a compact failure summary with reasons in `failedChecks`; the full diagnostic matrix remains in `details.batchSteps`.
|
|
465
|
-
- fails the native tool result with `failureCategory: "qa-failure"` when diagnostics report definitely new page errors, an unverified page-error check, console error messages, actionable failed network requests, any batch step failure, or missing expected text after the requested load state. Expected-text QA checks use bounded visible-element predicates plus fail-fast batch behavior instead of reading all body text, so dense pages can pass on visible headings/copy and missing text reports `expected text not found: …` rather than burning the full wrapper watchdog. Benign classification (implementation: `classifyNetworkRequestFailure` → `isBenignAssetFailure` in `extensions/agent-browser/lib/results/network.ts`) applies only when the row is already treated as failed (`status >= 400`, `failed: true`, or a string `error`—see `isFailedNetworkRequest`), the URL path’s last segment matches the icon basename heuristic (`favicon` plus `.ico`/`.png`/`.svg`, or `apple-touch-icon` plus `.png`, each allowing an optional `[-.\w]*` stem suffix before the extension), **and** at least one of `status === 404`, `failed === true`, or `typeof request.error === "string"` holds (so a **status-only** failure such as `500` on that path with neither `failed` nor a string `error` stays actionable). It also requires the upstream `resourceType` / `mimeType` (whichever is present) to be absent or look image-like: `image`, `img`, `other`, or a value starting with `image/`. Those rows are counted in `qaPreset.warnings` (for example `N benign network request failure(s) ignored`) and omitted from the actionable failed-network tally; every other failed request stays actionable.
|
|
329
|
+
Enable `action` through `agent_browser_tools`, then call the tool with flat fields. Optional `outputPath` and `timeoutMs` apply to the ordinary executor; this tool has no `sessionMode` field.
|
|
466
330
|
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
```json
|
|
470
|
-
{ "qa": { "url": "https://example.com", "expectedText": "Example Domain", "screenshotPath": ".dogfood/qa-example.png" } }
|
|
471
|
-
```
|
|
472
|
-
|
|
473
|
-
A nonempty post-clear baseline is not reliable event identity. A full native FIFO buffer can replace old errors with identical new errors while both reads remain identical. Matched final rows therefore add “page-error check could not be verified” to `failedChecks`, set `qaPreset.passed: false`, and take the existing `qa-failure` / Pi `isError: true` path. This is an evidence failure, not proof that the application emitted those matched rows during QA; even a genuinely clean page can be non-pass when residue cannot be cleared. The wrapper does not reset the browser, add event-correlation heuristics, or repair native clearing.
|
|
474
|
-
|
|
475
|
-
| Error evidence with `checkErrors` enabled | QA outcome when other checks pass |
|
|
331
|
+
| Field | Contract |
|
|
476
332
|
| --- | --- |
|
|
477
|
-
|
|
|
478
|
-
|
|
|
479
|
-
|
|
|
480
|
-
|
|
|
481
|
-
|
|
|
482
|
-
|
|
483
|
-
|
|
333
|
+
| `action` | Required: `click`, `check`, `fill`, or `select` |
|
|
334
|
+
| `selector` | Direct CSS/selector/current-ref target; cannot combine with locator fields for click/check/fill |
|
|
335
|
+
| `locator` | `role`, `text`, `label`, `placeholder`, `alt`, `title`, or `testid`; select supports only role/label |
|
|
336
|
+
| `value` | Locator value or one select option; for label select, this is the label |
|
|
337
|
+
| `values` | One or more select options; required for label select; rejected on other actions |
|
|
338
|
+
| `text` | Fill text; required for fill, rejected on select |
|
|
339
|
+
| `role`, `name` | Accessible role and optional name; role may replace locator value and must agree if both are given |
|
|
340
|
+
| `session` | Explicit native session; preserved in compiled commands and recovery |
|
|
484
341
|
|
|
485
|
-
|
|
342
|
+
Locator click/check/fill compiles to native `find`; direct targets compile to their ordinary command. Select uses native `select <selector> <value...>`. Accessible select resolves exactly one current visible combobox/listbox through a fresh snapshot; missing or ambiguous targets fail before action. Use `role: "combobox"` or `"listbox"` plus `name`, or label plus option `values`. No native `find ... select` capability is assumed.
|
|
486
343
|
|
|
487
|
-
|
|
344
|
+
```json
|
|
345
|
+
{ "action": "fill", "locator": "label", "value": "Email", "text": "user@example.com" }
|
|
346
|
+
{ "action": "select", "locator": "role", "role": "combobox", "name": "Flavor", "value": "chocolate" }
|
|
347
|
+
{ "action": "select", "locator": "label", "value": "Flavor", "values": ["chocolate"] }
|
|
348
|
+
```
|
|
488
349
|
|
|
489
|
-
-
|
|
490
|
-
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, and `networkSourceLookup`
|
|
491
|
-
- top-level wrapper shorthand for Electron desktop apps; do not nest it inside `batch` stdin
|
|
492
|
-
- `stdin` is rejected with `electron`; host-only actions manage their own local work and `launch` manages its own upstream `connect`
|
|
493
|
-
- supported actions: `list`, `launch`, `status`, `cleanup`, and `probe`
|
|
350
|
+
`details.compiledSemanticAction` retains the compiler's internal name and redacted input/argv; `effectiveArgs` shows actual current-ref resolution. Same-session URL verification, snapshot resolution, and action share the execution lock. Stale locator failures can offer a same-target `find` retry after snapshot refresh; stale direct/select refs get refresh guidance only. Selector misses may add exact visible ref candidates or bounded button/link name candidates. Fill recovery omits submitted text and never submits: focus/click a current editable ref, use native `keyboard type` if needed, and verify application state. `uncheck` remains a direct native command, not a shorthand action.
|
|
494
351
|
|
|
495
|
-
|
|
352
|
+
### `agent_browser_qa`
|
|
496
353
|
|
|
497
|
-
|
|
498
|
-
| --- | --- | --- |
|
|
499
|
-
| `list` | `query?`, `maxResults?` | Scans supported platform app locations for Electron evidence and returns bounded app metadata in `details.electron.apps`. Likely-sensitive app annotations are advisory metadata only. Does not spawn upstream `agent-browser`; list output also warns that later wrapper launches are isolated and will not read existing signed-in desktop state. |
|
|
500
|
-
| `launch` | exactly one of `appPath`, `appName`, `bundleId`, or `executablePath`; optional `appArgs`, `handoff`, `targetType`, `timeoutMs`, `allow`, `deny` | Resolves and verifies an Electron target, launches it with a wrapper-owned isolated profile and OS-chosen CDP port, attaches through upstream `connect` using `sessionMode: "fresh"`, and records `details.electron.launch`. It does not reuse the app's normal signed-in profile or attach to an already-running authenticated app; when signed-in local app state is the goal, use a host debug-port launch plus raw `connect` instead. Pi cancellation before spawn returns `failureCategory: "aborted"` without opening the app; cancellation during readiness polling or post-attach handoff closes the managed session, stops the process, and removes its isolated profile. If any wrapper validation, policy, or live-URL handoff check fails after launch, the wrapper immediately cleans the new process/profile and retains a partial record only when cleanup could not finish. |
|
|
501
|
-
| `status` | optional `launchId` or `all`, optional `timeoutMs` | Inspects wrapper-tracked launches, debug-port liveness, and current CDP targets without mutating the app. With neither `launchId` nor `all`, selects the single active wrapper launch when unambiguous. Runtime-owned launches remain visible by `launchId` after a Pi `session_tree` branch switch. Current branch-visible launches survive `/reload`; off-branch owned launches are cleaned on reload. |
|
|
502
|
-
| `cleanup` | optional `launchId` or `all`, optional `timeoutMs` | Closes the tracked upstream session when present, stops only the wrapper-tracked process, verifies debug-port shutdown, removes the wrapper-created `userDataDir`, and marks records cleaned or partial. Cleanup is serialized with managed-session and artifact-lifecycle browser work, reuses the launch record's canonical namespace/session identity, and a successful managed-session close step clears live/restore and recording-reservation state even when host process/profile cleanup remains partial. |
|
|
503
|
-
| `probe` | optional `launchId`, optional `timeoutMs` | Runs bounded current-session or launch-scoped state reads (`get url`, then `get title`, focused-element `eval --stdin`, `tab list`, compact `snapshot -i`) and reports `details.electron.probe`; current-managed probes also persist top-level `details.namespace`, `sessionTabTarget`, and `refSnapshot` so reload/branch replay keeps the same namespaced page identity. Without `launchId`, requires an active attached managed session; with `launchId`, it resolves the tracked launch session, including runtime-owned off-branch launch records, and can report mismatch guidance. If every read fails, the tool fails as `upstream-error`; the live URL is rechecked before any title/content helper. |
|
|
504
|
-
|
|
505
|
-
Validation and defaults:
|
|
506
|
-
|
|
507
|
-
- `launch` requires exactly one target field. `list` accepts only `query` and `maxResults`, with no configurable timeout (neither top-level `timeoutMs` nor nested `electron.timeoutMs`); `probe` accepts only `launchId` and `timeoutMs` beyond `action`; `status` / `cleanup` accept only `launchId`, `all`, and `timeoutMs`.
|
|
508
|
-
- Host install discovery (`electron.list` and resolving `appName` / `bundleId` through `discoverElectronApps` in `extensions/agent-browser/lib/electron/discovery.ts`) runs on **macOS** and **Linux** only. On **Windows** (and any other platform), `list` returns `platform: "unsupported"` with an empty `apps` array, and `launch` cannot resolve purely name-based targets without a prior scan—use `executablePath` or a host `appPath` that resolves to a verifiable Electron binary (`inspectElectronExecutablePath` still gates Windows executables).
|
|
509
|
-
- `launch.handoff` defaults to `"snapshot"`; supported values are `"connect"`, `"tabs"`, and `"snapshot"`. `"connect"` stops after attach, `"tabs"` verifies the live URL before a session-scoped `tab list` and is the safer diagnostic starting point when you do not want to capture refs/content yet, and `"snapshot"` verifies the live URL before `tab list` plus `snapshot -i` so current refs are immediately available. If the first Electron snapshot returns no refs while the app is still settling, the wrapper retries briefly before reporting no refs and tells agents to run `snapshot -i` once more before treating the UI as unusable.
|
|
510
|
-
- `launch.targetType` defaults to `"page"`; supported values are `"page"`, `"webview"`, and `"any"`. When a matching CDP target exposes a WebSocket URL, launch connects to that target; otherwise it falls back to the browser port.
|
|
511
|
-
- `appArgs` are passed to the Electron app, but wrapper-owned lifecycle/debug flags are rejected (`--user-data-dir`, `--remote-debugging-port`, `--remote-debugging-address`, `--remote-debugging-pipe`, and `--`).
|
|
512
|
-
- `allow` and `deny` are optional caller-owned policy lists. Entries match app name, bundle id, desktop id, app path, or executable path by substring. If `allow` is set, the target must match it; `deny` wins on conflict. With neither list, launch is permitted.
|
|
513
|
-
- `electron.status` / `electron.cleanup` accept optional `all` only as the boolean literal `true` to include every active wrapper-tracked launch (including dead, failed, or partial records, but excluding cleaned records); `all` and `launchId` cannot both be set. Status and cleanup use the same runtime wrapper-tracked scope: current branch-visible records plus still-owned off-branch records. Default no-argument status/cleanup is intentionally ambiguous when more than one active launch is in that merged scope; pass `launchId` or `all: true`.
|
|
514
|
-
- `electron.launch.timeoutMs` sets the host CDP readiness polling budget: **15000 ms** by default, capped at **120000 ms** (`normalizeTimeoutMs` in `extensions/agent-browser/lib/electron/launch.ts`). Its clock starts after target discovery and policy checks, before creating the isolated profile. Discovery has no configurable deadline; upstream attach and handoff use separate subprocess budgets.
|
|
515
|
-
- `status.timeoutMs` applies to each managed-session `get url` / `get title` subprocess for mismatch diagnostics, including `get cdp-url` when verifying a restored connection. `probe.timeoutMs` applies to each upstream read (optional `get cdp-url` verification, then `get url`, `get title`, `eval --stdin`, `tab list`, `snapshot -i`). Their default wrapper budget is **35000 ms**, overridden by `PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` (`getAgentBrowserProcessTimeoutMs` in `extensions/agent-browser/lib/process.ts`). Localhost CDP HTTP probes use a fixed **1000 ms** each (`ELECTRON_CDP_FETCH_TIMEOUT_MS` in `extensions/agent-browser/lib/electron/cdp.ts`). Profile-path inspection has no configurable timeout.
|
|
516
|
-
- `cleanup.timeoutMs` is applied separately to the managed-session `close` subprocess and the initial host process-exit wait, not one combined deadline. It defaults to `PI_AGENT_BROWSER_IMPLICIT_SESSION_CLOSE_TIMEOUT_MS` or **35000 ms** (`getImplicitSessionCloseTimeoutMs` in `extensions/agent-browser/lib/runtime.ts`). Restored-PID verification and the later force-kill wait each have separate **1000 ms** limits; debug-port checks use the fixed CDP fetch budget, and profile removal has no configurable deadline.
|
|
517
|
-
- Non-Electron targets are rejected as a correctness failure; the wrapper does not blindly launch arbitrary executables as Electron.
|
|
518
|
-
|
|
519
|
-
Safety defaults and ownership:
|
|
520
|
-
|
|
521
|
-
- `launch` always uses a new isolated wrapper-created `userDataDir` and `--remote-debugging-port=0`, then reads `DevToolsActivePort`; callers cannot choose a fixed debug port, cannot reuse the app's normal signed-in profile, and cannot use this path to attach to an already-running authenticated app.
|
|
522
|
-
- The wrapper also passes `--disable-extensions`, `--no-first-run`, and `--no-default-browser-check` alongside sanitized caller `appArgs`.
|
|
523
|
-
- Remote debugging exposes app contents to the attached browser tool. The wrapper gives isolation defaults and optional `allow` / `deny`; the user still owns the decision to launch or attach to a sensitive desktop app.
|
|
524
|
-
- `electron.list` may annotate apps as likely sensitive (`sensitivity.level: "likely-sensitive"`, categories such as `notes`, `chat`, `mail`, `developer-workspace`, or `passwords-auth`) and print `[likely sensitive: …]`. These annotations are non-blocking hints, not enforcement; caller-owned `allow` / `deny` policy still controls launch decisions.
|
|
525
|
-
- Cleanup is wrapper-owned **only** for records created by `electron.launch`. `electron.cleanup` never targets manually launched apps, externally supplied debug ports, or arbitrary Electron processes. Explicit screenshots/downloads/HARs/traces remain host-file cleanup, not Electron cleanup. If `electron.cleanup` closes the upstream managed session but process/profile cleanup remains partial, later shutdown cleanup does not close that managed session a second time; retry cleanup focuses on the remaining host resources.
|
|
526
|
-
- On Pi `quit`, active wrapper-owned Electron launches are best-effort cleaned. On `/reload`, current branch-visible active Electron launches are preserved for reload continuity, including their isolated `userDataDir` profile directories, while off-branch owned launches are cleaned before process-local ownership is cleared. First reuse after reload/resume verifies the recorded namespace/session, live PID/profile presence, and saved browser WebSocket endpoint; native `get cdp-url` must match that browser or one of its current targets. Ordinary browser calls, status reads, and probes share the existing locked check, without reconnecting or resetting refs. The `get cdp-url` read uses the caller's `timeoutMs` and cancellation; localhost CDP fetch budgets are unchanged. Failed checks do not grant reuse or relax the generic restore-disabled-session policy. If cleanup is partial and deliberately skips or fails `user-data-dir` removal because the process or debug port is still live, generic temp cleanup preserves that profile path across reload, quit, later temp sweeps, process-exit cleanup, and stale temp-root pruning after restart instead of deleting it underneath the remaining host resource. Stale restored records are reported instead of guessed/killed when the wrapper lacks a live child process.
|
|
527
|
-
|
|
528
|
-
Details fields:
|
|
354
|
+
Enable `qa`. Flat input is either `{ url, ...checks }` or `{ attached: true, ...checks }`; these forms are exclusive. Checks are optional `expectedText` (string or string array), `expectedSelector`, `screenshotPath`, `checkNetwork`, `checkConsole`, `checkErrors`, and `loadState` (`domcontentloaded` default, `load`, or `networkidle`). `outputPath`, `timeoutMs`, and `sessionMode` are supported; attached QA only accepts `sessionMode: "auto"`.
|
|
529
355
|
|
|
530
356
|
```json
|
|
531
|
-
{
|
|
532
|
-
|
|
533
|
-
"action": "launch",
|
|
534
|
-
"appName": "My App",
|
|
535
|
-
"handoff": "snapshot",
|
|
536
|
-
"targetType": "page",
|
|
537
|
-
"appArgs": ["--safe-mode"]
|
|
538
|
-
},
|
|
539
|
-
"electron": {
|
|
540
|
-
"action": "launch",
|
|
541
|
-
"status": "succeeded",
|
|
542
|
-
"launch": {
|
|
543
|
-
"version": 1,
|
|
544
|
-
"launchId": "electron-…",
|
|
545
|
-
"launchedByWrapper": true,
|
|
546
|
-
"appName": "My App",
|
|
547
|
-
"bundleId": "com.example.my-app",
|
|
548
|
-
"appPath": "/Applications/My App.app",
|
|
549
|
-
"executablePath": "/Applications/My App.app/Contents/MacOS/My App",
|
|
550
|
-
"userDataDir": "/tmp/pi-agent-browser-electron-…",
|
|
551
|
-
"port": 12345,
|
|
552
|
-
"pid": 1234,
|
|
553
|
-
"sessionName": "pi-…",
|
|
554
|
-
"webSocketDebuggerUrl": "ws://127.0.0.1:12345/devtools/browser/…",
|
|
555
|
-
"createdAtMs": 1770000000000,
|
|
556
|
-
"cleanupState": "active"
|
|
557
|
-
},
|
|
558
|
-
"targets": [{ "type": "page", "title": "My App", "url": "app://index" }],
|
|
559
|
-
"handoff": { "handoff": "snapshot" },
|
|
560
|
-
"profileIsolation": {
|
|
561
|
-
"isolatedLaunch": true,
|
|
562
|
-
"reusesExistingSignedInProfile": false,
|
|
563
|
-
"attachesToAlreadyRunningApp": false,
|
|
564
|
-
"hostDebugLaunchExample": "macOS: open -a <App Name> --args --remote-debugging-port=9222 --remote-allow-origins='*'; then agent_browser connect 9222 with sessionMode=fresh"
|
|
565
|
-
}
|
|
566
|
-
}
|
|
567
|
-
}
|
|
357
|
+
{ "url": "https://example.com", "expectedText": "Example Domain", "screenshotPath": "/tmp/qa.png" }
|
|
358
|
+
{ "attached": true, "expectedText": "Explorer", "checkErrors": true }
|
|
568
359
|
```
|
|
569
360
|
|
|
570
|
-
|
|
361
|
+
QA compiles a fixed `batch --bail` diagnostic plan. URL QA clears enabled network/console buffers, captures page-error residue after native clear, opens the requested URL, waits for readiness, allows a bounded 150 ms diagnostic settle when needed, then performs visible-text/selector assertions, enabled diagnostic reads, and optional screenshot. Expected text uses bounded visible-element predicates with a 5-second native timeout. Missing assertions stop before later diagnostics. URL QA does not require the previous page target.
|
|
571
362
|
|
|
572
|
-
|
|
573
|
-
- `launch`: `{ action: "launch", status, launch, targets?, version?, handoff?, cleanup?, identifiers?, profileIsolation }`. `profileIsolation` states that wrapper launches use a new temporary profile, do not reuse existing signed-in app state, and do not attach to already-running authenticated apps; it also includes host debug-launch guidance for the separate normal-app attach path. `identifiers` repeats the launch-scoped `launchId` and attached `sessionName` so agents distinguish Electron lifecycle actions from browser session/tab actions. `launch.cleanupState` is one of `"active"`, `"cleaned"`, `"dead"`, `"failed"`, or `"partial"`. Failed launches expose `details.electron.failure.diagnostics` when available, including `pid` / `pidAlive`, wrapper `userDataDir`, elapsed/timeout timing, `DevToolsActivePort` file state, discovered port, and whether CDP `/json/version` was reached. `outputCaptured` reports whether stdout/stderr capture was configured; readable streams include `stdoutTail` / `stderrTail` (empty strings for empty output) and `stdoutTruncated` / `stderrTruncated`. Each tail reads at most the last 4096 source bytes before UTF-8 decoding and credential redaction; the same tails appear in visible error text. Optional `stdoutError` / `stderrError` report read/close failures without hiding the primary failure. Capture files are mode-0600 `stdout.log` / `stderr.log` inside the isolated profile and follow its lifecycle; file growth is not capped by the tail-read limit. Failed-startup termination errors preserve the profile/logs through generic cleanup and host exit. Persistence failures join `failure.cleanupError` while in-memory protection remains.
|
|
574
|
-
- `status`: `{ action: "status", status: "succeeded", launches, statuses, targets, identifiers?, identifierList?, managedSession?, managedSessions?, sessionMismatch?, sessionMismatches? }`, where each status includes the tracked `launchId`, `cleanupState`, independently measured port/pid liveness, bounded CDP targets, and fresh `userDataDirState`: `"present"`, `"absent"`, or `"unknown"`. Native `lstat` success means present (including dangling symlinks); only ENOENT means absent, and other filesystem errors mean unknown. This measures the tracked profile path, not all app residue, and is not stored in `ElectronLaunchRecord`. Explicit-ID status labels cleaned records as historical; default and `all: true` exclude them. Mismatch fields explain when the current managed session or tab does not match a live wrapper launch target.
|
|
575
|
-
- `cleanup`: `{ action: "cleanup", status: "succeeded" | "partial", cleanup: { partial, records, results } }`. Partial cleanup is a failed tool result with `failureCategory: "cleanup-failed"` and retry next actions. Cleanup steps may include `managed-session`, `process`, `debug-port`, and `user-data-dir`; managed-session close failures are reported while host-owned process/profile cleanup still runs.
|
|
576
|
-
- `probe`: `{ action: "probe", status: "succeeded" | "partial", probe, probeContext, identifiers?, sessionMismatch?, statusTargets?, launchStatus? }`. `probeContext` records whether the probe inspected the current managed session or a specific `launchId`. `probe` includes bounded `title`, `url`, `focusedElement`, `activeTab`, `tabs`, compact `snapshot` metadata (`refCount`, `refIds`, optional text preview and omission counts), `errors?`, and `summary`. When launch status is known, `launchStatus.userDataDirState` carries the same fresh profile-path measurement as `status`, and visible probe output includes debug-port/pid liveness so `about:blank` plus a dead wrapper launch is unmistakable. It also updates the normal session target/ref tracking when a snapshot is collected.
|
|
363
|
+
Attached QA checks the current browser without navigation or buffer clears. It first requires a live nonempty URL, including file/custom app URLs. Failed target verification returns exact tab/URL recovery instead of running the batch. Diagnostic reads default off because preserved buffers may predate this check; URL diagnostics default on. Explicit `checkErrors: false` skips the error-baseline check.
|
|
577
364
|
|
|
578
|
-
|
|
365
|
+
The verdict is `qaPreset: { passed, failedChecks, warnings, summary }`. Console errors, definitely new page errors, ambiguous matching post-clear residue, actionable failed requests, failed batch steps, or failed visible assertions cause `qa-failure`. Matching a nonempty error baseline is an **unverified check**, not proof of a new application error. A clean final buffer can pass. Benign low-impact icon misses may be warnings when the failed row and resource metadata satisfy `classifyNetworkRequestFailure`; ordinary document/script/API failures remain actionable. Native diagnostic-read success alone never constitutes a QA pass.
|
|
579
366
|
|
|
580
|
-
|
|
581
|
-
- `cleanup-failed` is used when `electron.cleanup` only partially cleans tracked resources; inspect `details.electron.cleanup.results[].steps` for remaining process, port, or profile cleanup state.
|
|
582
|
-
- Launch timeout maps to `timeout`; non-Electron targets and input issues map to `validation-error`; launch/attach/spawn/CDP failures map to `upstream-error` unless a more specific category applies. If a successful-looking Electron mutation is followed by a dead process/debug port or an unrecoverable `about:blank` target, the wrapper upgrades the result to `failureCategory: "tab-drift"` and sets `details.electronPostCommandHealth` with the launch status (including fresh `userDataDirState`) and recovery action ids.
|
|
583
|
-
- Successful Electron `fill <selector> <text>` commands may run a read-only `get value <selector>` verification, and successful fills against refs whose latest snapshot metadata proves a contenteditable target may run `get text <ref>` verification. If the value/text still differs, the wrapper keeps the tool successful but adds `details.fillVerification`, visible guidance to use snapshot/focus/keyboard typing for custom quick-input or rich editor controls, and `inspect-after-fill-verification` / `verify-filled-value` next actions.
|
|
584
|
-
- Successful active launches/status/probe results may include exact `details.nextActions` with ids `status-electron-launch`, `probe-electron-launch`, `cleanup-electron-launch`, `list-electron-tabs`, and `snapshot-electron-session`. Electron status/probe mismatch diagnostics may also include `reattach-electron-launch` before fresh tab/snapshot inspection. Electron post-command health failures include status/probe/cleanup actions for the same `launchId`; Electron `@e…` mutations may add `refresh-electron-refs-after-rerender` because desktop apps often rerender without URL changes. Electron cleanup partial failures may include `status-electron-launch` and `retry-electron-cleanup`.
|
|
367
|
+
`details.compiledQaPreset` retains the generated plan and checks, including `diagnosticsResetAtStart`; `compiledJob` is internal QA batch-plan metadata, not a public job DSL. Use native batch/code for custom checks.
|
|
585
368
|
|
|
586
|
-
|
|
369
|
+
### `agent_browser_electron`
|
|
587
370
|
|
|
588
|
-
|
|
589
|
-
|
|
590
|
-
|
|
591
|
-
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
371
|
+
Enable `electron`. Inputs are flat `{ action, ...fields, outputPath? }`; caller `stdin` and `sessionMode` are not accepted. The [Electron guide](ELECTRON.md) covers workflows and ownership.
|
|
372
|
+
|
|
373
|
+
| Action | Fields | Behavior |
|
|
374
|
+
| --- | --- | --- |
|
|
375
|
+
| `list` | `query?`, `maxResults?` | Host-only bounded discovery; default 50, cap 200. No timeout field. macOS/Linux scanning only; Windows returns unsupported discovery. |
|
|
376
|
+
| `launch` | Exactly one of `appPath`, `appName`, `bundleId`, `executablePath`; `appArgs?`, `handoff?`, `targetType?`, `timeoutMs?`, `allow?`, `deny?` | Verify Electron evidence, create a temporary isolated profile/OS-chosen debug port, attach through native connect, hand off, and record ownership. |
|
|
377
|
+
| `status` | `launchId?` or `all: true`, `timeoutMs?` | Inspect tracked PID/port/profile and targets; explicit cleaned IDs remain inspectable as historical records. |
|
|
378
|
+
| `probe` | `launchId?`, `timeoutMs?` | Verify URL then read title/focus/tabs/snapshot with owned session settings. Without an ID, inspect the current attached managed session. |
|
|
379
|
+
| `cleanup` | `launchId?` or `all: true`, `timeoutMs?` | Close the tracked native session, stop only the owned app, verify port shutdown, remove its temporary profile. Partial cleanup fails with exact retry guidance. |
|
|
597
380
|
|
|
598
381
|
```json
|
|
599
|
-
{
|
|
600
|
-
|
|
601
|
-
"id": "snapshot-electron-session",
|
|
602
|
-
"reason": "Refresh interactive refs for the attached Electron session.",
|
|
603
|
-
"safety": "Use current Electron refs only after a fresh snapshot for this session.",
|
|
604
|
-
"params": { "args": ["--session", "pi-…", "snapshot", "-i"] }
|
|
605
|
-
}
|
|
382
|
+
{ "action": "launch", "appName": "Visual Studio Code", "handoff": "snapshot" }
|
|
383
|
+
{ "action": "cleanup", "launchId": "electron-…" }
|
|
606
384
|
```
|
|
607
385
|
|
|
608
|
-
|
|
386
|
+
`handoff` defaults to `snapshot`; `tabs` verifies/discovers targets without an interactive snapshot, and `connect` stops after attach. `targetType` is `page` by default, or `webview`/`any`. Launch cannot reuse the normal signed-in app profile or an existing app; use host launch plus direct native `connect` for that. Windows launch requires a verifiable path rather than scan-based name resolution; this is not Windows qualification evidence.
|
|
609
387
|
|
|
610
|
-
|
|
611
|
-
{ "electron": { "action": "list", "query": "code" } }
|
|
612
|
-
{ "electron": { "action": "launch", "appName": "Visual Studio Code", "handoff": "snapshot" } }
|
|
613
|
-
{ "electron": { "action": "probe" } }
|
|
614
|
-
{ "qa": { "attached": true, "expectedText": "Explorer" } }
|
|
615
|
-
{ "electron": { "action": "cleanup", "launchId": "electron-…" } }
|
|
616
|
-
```
|
|
388
|
+
`appArgs` cannot override user-data-dir/debug lifecycle flags or include bare `--`. Caller `allow`/`deny` substring policies match app names/IDs/paths, with deny winning. Likely-sensitive discovery annotations are advisory. Non-Electron executables fail validation.
|
|
617
389
|
|
|
618
|
-
|
|
390
|
+
Timeouts are per action: launch readiness defaults to 15 seconds, capped at 120 seconds after discovery; status/probe use the normal 35-second subprocess budget per read (or configured override); cleanup applies its default 35-second managed-close budget separately to close and initial process-exit wait. Local CDP HTTP probes use one second. There is no combined end-to-end teardown deadline. Cancellation before spawn opens nothing; cancellation during launch cleans the new process/profile.
|
|
619
391
|
|
|
620
|
-
|
|
621
|
-
{ "args": ["connect", "9222"], "sessionMode": "fresh" }
|
|
622
|
-
```
|
|
392
|
+
`details.electron` retains action-specific `apps`, `launch`, `statuses`, `targets`, `probe`, `cleanup`, and failure diagnostics; `compiledElectron` retains the action plan. `launchId` identifies host lifecycle; `sessionName` identifies browser commands. `userDataDirState` measures the tracked path as present, absent (ENOENT), or unknown; it does not audit all app residue. Failed startup output includes redacted last-4096-byte stdout/stderr tails; private profile-local files follow profile ownership, not a lifetime byte cap. Failed cleanup preserves live profiles. Reload preserves current branch-visible launches and cleans off-branch owned launches; quit cleans owned launches. Restored attachments must match live endpoint/PID/profile evidence before reuse.
|
|
623
393
|
|
|
624
|
-
|
|
625
|
-
|
|
626
|
-
- type: object with at least one of `selector`, `reactFiberId`, or `componentName`
|
|
627
|
-
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `qa`, `networkSourceLookup`, and `electron`
|
|
628
|
-
- **EXPERIMENTAL — candidates only:** opt-in helper for local app debugging; it reports candidate source locations with confidence and evidence instead of claiming a guaranteed DOM-to-file mapping. Do not treat output as authoritative file ownership or edit targets without verification.
|
|
629
|
-
- compiles to existing upstream `batch` commands only:
|
|
630
|
-
- `selector` adds `is visible <selector>` and, unless `includeDomHints: false`, adds `get html <selector>` for source-like DOM attributes (`data-source-file`, `data-file`, `data-component-file`, `data-source`, plus optional `data-source-line` / `data-line` and `data-source-column` / `data-column`) and for `.ts`/`.tsx`/`.js`/`.jsx` paths embedded in HTML text
|
|
631
|
-
- `reactFiberId` runs `react inspect <id>`; this requires the page to have been launched with `--enable react-devtools` before first navigation and for the app build to expose source information
|
|
632
|
-
- `componentName` runs `react tree` and performs a bounded local workspace scan under the Pi tool session **cwd** for matching component declarations in `.ts`, `.tsx`, `.js`, and `.jsx` files (skipping directories such as `.git`, `node_modules`, `dist`, `build`, `coverage`, `.next`, `out`, `tmp`, and `temp`); the walk stops after `maxWorkspaceFiles` files (default 2000, hard cap 5000) and records at most ten `workspace-search` candidates
|
|
633
|
-
- optional `includeDomHints: false` skips the selector HTML read
|
|
634
|
-
- optional `maxWorkspaceFiles` bounds the local component-name scan; default is 2000 source files and the hard maximum is 5000
|
|
635
|
-
- reports `details.compiledSourceLookup` with the generated batch plan and `details.sourceLookup` with `{ status, candidates, limitations, summary, workspaceRoot?, electronContext? }`
|
|
636
|
-
- each `candidates[]` entry includes `source` (`react-inspect`, `dom-attribute`, or `workspace-search`), `confidence` (`high`, `medium`, or `low`), `evidence` (string reasons), and optional `file`, `line`, `column`, and `componentName`
|
|
637
|
-
- `details.sourceLookup.status` is one of `candidates-found`, `no-candidates`, or `unsupported`; `unsupported` applies only when **no** candidates were collected **and** at least one compiled `react` batch step failed (for example React DevTools not enabled, no renderer, or inspect errors). If DOM or workspace evidence still produced candidates, `status` stays `candidates-found` even when a `react` step failed
|
|
638
|
-
- when analysis produces a `summary`, the wrapper prepends it to the primary visible text block (or inserts a leading text block) for quick scanning; unlike `qa`, it never flips the unified tool outcome to failed solely because diagnostics look noisy or because `status` is `no-candidates` / metadata was missing—failed upstream batch steps still surface as normal tool errors
|
|
639
|
-
- for wrapper-tracked packaged Electron sessions where `status` is `no-candidates`, the wrapper may add `workspaceRoot` and `electronContext` (`launchId?`, `appName?`, `appPath?`, `executablePath?`, `sessionName?`, `url?`) plus limitations explaining that the local workspace scan only covered the Pi tool cwd and did not unpack installed app resources or `app.asar`; it may also append `snapshot-electron-session`, `probe-electron-launch`, and `list-electron-tabs` next actions for live app inspection
|
|
394
|
+
Post-command process/target loss fails as `tab-drift`; fill verification and broad-shell-text diagnostics remain available. Recovery actions use `agent_browser_electron` with flat params for status/probe/cleanup and `agent_browser` for native tab/URL/ref operations. If inactive, enable `electron` first. Manually launched apps and explicit saved artifacts remain host-owned.
|
|
640
395
|
|
|
641
|
-
|
|
396
|
+
### `agent_browser_source`
|
|
397
|
+
|
|
398
|
+
Enable `source`. Flat fields: at least one of `selector`, `reactFiberId`, or `componentName`; optional `includeDomHints` (default true), `maxWorkspaceFiles` (default 2,000, maximum 5,000), `outputPath`, `timeoutMs`, and `sessionMode`.
|
|
642
399
|
|
|
643
400
|
```json
|
|
644
|
-
{ "
|
|
401
|
+
{ "selector": "#save", "componentName": "SaveButton" }
|
|
645
402
|
```
|
|
646
403
|
|
|
647
|
-
|
|
404
|
+
Selector inspection runs native `is visible` and optional `get html`; React inspection/tree uses native tooling and requires an exposed DevTools renderer. Component-name search scans bounded local `.ts/.tsx/.js/.jsx` files while skipping build/dependency directories, returning at most ten workspace matches. It does not unpack installed Electron bundles or `app.asar`.
|
|
648
405
|
|
|
649
|
-
|
|
406
|
+
`sourceLookup` remains the internal result field: `{ status, candidates, limitations, summary, workspaceRoot?, electronContext? }`. Candidates carry source, confidence, evidence, and optional file/line/column/component. `unsupported` means no candidates and a failed React step; DOM/workspace candidates keep `candidates-found` even if React fails. `no-candidates` alone does not fail a successful batch, but failed native steps still do. Packaged Electron results explain workspace limits and may offer app inspection actions. These are candidates, not authoritative ownership or edit instructions.
|
|
650
407
|
|
|
651
|
-
|
|
652
|
-
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, and `electron`
|
|
653
|
-
- **EXPERIMENTAL — candidates only:** failed-request source-hint helper; it reports failed network requests and candidate source hints with evidence instead of assigning blame or proving root cause
|
|
654
|
-
- compiles to existing upstream `batch` commands only: `network request <requestId>` when provided plus `network requests` with `--filter <filter-or-url>` when a filter or URL is provided (if both are set, `filter` wins; when only `url` is set, it becomes the `--filter` argument); optional `namespace` / `session` prepends `--namespace <name>` / `--session <name>` before that generated `batch`; exact `namespace: ""` selects the default namespace and remains explicit so an ambient namespace cannot redirect the lookup
|
|
655
|
-
- detects failed requests from `status >= 400`, `failed: true`, or an `error` field
|
|
656
|
-
- candidate sources come from source-like initiator/stack metadata in upstream network results and bounded local workspace search for URL/path literals under the captured execution cwd
|
|
657
|
-
- optional `maxWorkspaceFiles` defaults to 2000 and cannot exceed 5000; workspace-search candidates are capped at ten
|
|
658
|
-
- reports `details.compiledNetworkSourceLookup` with the generated batch plan and `details.networkSourceLookup` with `{ status, failedRequests, candidates, limitations, summary }`
|
|
659
|
-
- `details.networkSourceLookup.status` is one of `failed-requests-found`, `no-failed-requests`, or `no-candidates`
|
|
660
|
-
- `details.networkSourceLookup.failedRequests[]` lists correlated failed requests (optional `requestId`, `url`, HTTP `status`, `method`, `error`) after the same failure heuristics as analysis; those request URLs are diagnostic evidence only and do not replace the session’s active page target or invalidate the latest app-page `refSnapshot`
|
|
661
|
-
- each `candidates[]` entry uses `source` `initiator` (parsed from upstream initiator/stack/source/trace fields on matching requests) or `workspace-search` (string match of URL/path needles in local source files), plus `confidence`, `evidence`, optional `file`/`line`, and optional `requestUrl`; URLs and query parameters in these surfaces are redacted for model-facing output
|
|
408
|
+
### `agent_browser_network_source`
|
|
662
409
|
|
|
663
|
-
|
|
410
|
+
Enable `network`. Flat fields: at least one of `requestId`, `filter`, or `url`; optional `session`, `namespace`, `maxWorkspaceFiles` (2,000 default, 5,000 cap), `outputPath`, `timeoutMs`, and `sessionMode`. Explicit `namespace: ""` overrides an ambient namespace.
|
|
664
411
|
|
|
665
412
|
```json
|
|
666
|
-
{ "
|
|
413
|
+
{ "requestId": "req-1", "url": "/api/fail" }
|
|
667
414
|
```
|
|
668
415
|
|
|
669
|
-
|
|
416
|
+
A request ID runs native `network request`; filter/URL adds `network requests --filter`, with explicit filter winning. Failed HTTP/failed/error rows are correlated with initiator metadata and a bounded URL-literal search under the captured execution directory (up to eight needles and ten workspace matches). `networkSourceLookup` retains `{ status, failedRequests, candidates, limitations, summary }`; statuses are `failed-requests-found`, `no-failed-requests`, or `no-candidates`. URLs and credentials are redacted. Request URLs are diagnostic evidence and never replace the active page target. No candidates alone does not fail a successful batch or assign blame. HAR and full native diagnostics remain available through `args`.
|
|
670
417
|
|
|
671
418
|
### `stdin`
|
|
672
419
|
|
|
673
420
|
- type: `string`
|
|
674
421
|
- optional
|
|
675
|
-
- raw stdin for `eval --stdin`, `batch`, and `auth save --password-stdin
|
|
676
|
-
-
|
|
422
|
+
- raw stdin for `eval --stdin`, `batch`, and `auth save --password-stdin`
|
|
423
|
+
- advanced tools own their generated batch input and do not accept caller `stdin`; code puts stdin inside `browser({ stdin })`
|
|
677
424
|
- rejected before launch for any other command/stdin combination, including commands such as `click`, `snapshot`, or `open`
|
|
678
425
|
|
|
679
426
|
Examples:
|
|
@@ -711,10 +458,10 @@ Example:
|
|
|
711
458
|
### `timeoutMs`
|
|
712
459
|
|
|
713
460
|
- type: positive integer milliseconds
|
|
714
|
-
- optional per-call wrapper subprocess watchdog for
|
|
461
|
+
- optional per-call wrapper subprocess watchdog for direct/action/QA/source tools; code uses its whole-cell deadline and Electron uses the flat action-specific timeout documented above
|
|
715
462
|
- managed-session daemon-policy inspection has its own fixed budget of up to 35 seconds before that process and is intentionally not shortened by `timeoutMs`, so a busy valid daemon does not become an unsafe false negative
|
|
716
|
-
- automatic managed-session cleanup uses `PI_AGENT_BROWSER_IMPLICIT_SESSION_CLOSE_TIMEOUT_MS` (default 35 seconds) separately from the original call or
|
|
717
|
-
- use for long opens, large snapshots, paced
|
|
463
|
+
- automatic managed-session cleanup uses `PI_AGENT_BROWSER_IMPLICIT_SESSION_CLOSE_TIMEOUT_MS` (default 35 seconds) separately from the original call or code deadline. The default leaves room for native Chrome shutdown, including its five-second exit grace. Expiry reports the cleanup budget, phase (policy coordination, daemon inspection, or native close), and time spent in that phase—not caller cancellation. Closure remains unconfirmed and cleanup ownership is retained for recovery. The ordinary subprocess watchdog still applies: when extending cleanup beyond it, raise `PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` as well.
|
|
464
|
+
- use for long opens, large snapshots, paced native batch typing, or captures that legitimately need more than the default watchdog
|
|
718
465
|
- explicit long `wait` steps are forwarded to upstream; top-level `timeoutMs` only controls the wrapper subprocess watchdog and should be at least the wait duration plus a small grace window when supplied manually
|
|
719
466
|
- when the watchdog fires, `details.timeoutMs`, `details.timedOut`, and possibly `details.timeoutPartialProgress` explain what was recovered. Explicit URL reads and proven browser-independent read confirmations do not run timeout page probes. A `session info` timeout also leaves browser/page/ref state untouched and offers only `retry-session-info` for the same session/namespace, without claiming liveness. Uncertain recording stops instead use the bounded receipt recovery described below. If the page target is unknown, standalone snapshot suggestions are removed and one session-scoped `verify-page-target-after-timeout` fail-fast batch (`get url`, then `snapshot -i`) appears in visible failure text and `details.nextActions`, so the returned recovery is executable under the same page-target guard.
|
|
720
467
|
|
|
@@ -726,14 +473,14 @@ Example:
|
|
|
726
473
|
|
|
727
474
|
### `sessionMode`
|
|
728
475
|
|
|
729
|
-
A native `session` default from user/project JSON or `AGENT_BROWSER_SESSION` selects a caller-owned browser for
|
|
476
|
+
A native `session` default from user/project JSON or `AGENT_BROWSER_SESSION` selects a caller-owned browser for direct, code, action, QA, and source tools. Per-call flags still win. Like literal `--session`, this selection takes precedence over `fresh`; it reports `usedImplicitSession: false` and is not closed when Pi quits. Without a selected native session, attachment, or explicit fresh launch, auto calls select a stable root-Pi named session and root-specific native restore key. `PI_SUBAGENT_ROOT_SESSION_ID` carries the parent identity to descendants; ordinary new/fork/clone roots differ, while resume/cwd changes keep identity. Root sessions report `usedImplicitSession: false` and survive parent/child quit. Global/override package profile names with `policy: "always"` and executable defaults bootstrap only inactive automatic roots. Active roots preserve native launch settings/current restore key, including matching root-name follow-ups; unrelated explicit browsers are unchanged. Profile paths and project-only profile config remain advisory. Native restore covers cookies and web storage, not newly created IndexedDB or memory-only authentication. See the README for communal bootstrap and persistent-profile fallback. Active older/fresh managed sessions retain the lifecycle below. `agent_browser_qa` with `attached: true` can inspect this shared current session using the same live target checks. See [shared browser defaults](../README.md#shared-browser-defaults).
|
|
730
477
|
|
|
731
478
|
- type: `"auto" | "fresh"`
|
|
732
479
|
- optional
|
|
733
480
|
- default: `"auto"`
|
|
734
481
|
|
|
735
482
|
Behavior:
|
|
736
|
-
- if `args` already include `--session` (including argv compiled from
|
|
483
|
+
- if `args` already include `--session` (including argv compiled from `agent_browser_action.session`), upstream session choice wins
|
|
737
484
|
- `"auto"` prepends the current extension-managed active session when appropriate
|
|
738
485
|
- after resume, a confirmed inactive wrapper-owned daemon with automatic managed restore enabled reopens its complete recorded URL, including its fragment, before the first current-page operation, including `get url` and `reload`. Non-page calls such as `tab list` or explicit HTTP `read <url>` can start a daemon without consuming that pending reopen, even across branch/reload replay. The wrapper verifies the observed tab and invalidates old refs; native `open` resets frame scope. This reloads the page with restored cookies/storage, not unsaved forms, JavaScript memory, or history. Live wrong-tab recovery does not navigate. Explicit URL reads, URL `a11y`/`vitals`, `diff url`, `window new`, URL-bearing recording commands, and explicit navigation/context changes do not require the old tab. Caller-owned/attached and restore-disabled sessions are not auto-reopened
|
|
739
486
|
- `"fresh"` rotates that managed session to a fresh upstream launch so startup-scoped flags like `--profile`, `--executable-path`, `--ca-cert`, `--no-ca-cert`, `--webgpu`, `--no-webmcp`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--headed`, `--enable`, `-p` / `--provider`, or iOS `--device` apply and later default calls follow the new browser; `--idle-timeout` must equal the Pi process's configured managed idle timeout or the wrapper rejects it with restart guidance
|
|
@@ -774,6 +521,24 @@ These calls return plain text and stay stateless: the extension does not inject
|
|
|
774
521
|
|
|
775
522
|
## Result shape
|
|
776
523
|
|
|
524
|
+
### Canonical observation
|
|
525
|
+
|
|
526
|
+
Direct and advanced results expose the same decision-relevant observation in model-visible JSON or prose; code receives it as data without rendering/reparsing prose. The base is `{ success, resultCategory, data?, error?, summary?, successCategory?, failureCategory? }`. Optional fields include exact `nextActions`, warnings, session/namespace, artifact verification, image observations, page-change evidence, timeout partial progress, and QA/source/diagnostic results. Batch observations retain per-step outcomes. Native help/version remains plain text.
|
|
527
|
+
|
|
528
|
+
`details` retains audit, process, ownership, and replay metadata; it is not the sole channel for actionable failures. JSON remains parseable, including early preparation failures and filters. Prose retains useful command output and appends canonical observation metadata. Final visible text is bounded to 16,000 characters. If necessary, the complete redacted observation is spilled and the result names `observationPath` plus retrieval instructions. Exact recovery objects are either included whole or remain in that complete observation; truncated argv/stdin is never presented as executable recovery. An unavailable spill is reported honestly.
|
|
529
|
+
|
|
530
|
+
Inner code calls use `modelVisible: false` internally: no full model prose rendering, observation spills, or full-image base64 encoding for every result. This is not a public flag or a different executor. Source chooses JSON via `emit` and image handles via `emitImage`; failures and verification evidence remain visible at the outer boundary.
|
|
531
|
+
|
|
532
|
+
### Image observations
|
|
533
|
+
|
|
534
|
+
`imageObservations[]` carries verified capture metadata: `path`, optional `id` (cell-local code handle), `mimeType`, optional image `pixels` dimensions, `capture` (`viewport`, `full-page`, `element`, or `unknown`), and `geometry`. `geometry.status` is `measured` or `unknown`, with a `reason` and available before/after samples. Samples include CSS viewport/document dimensions, scroll offsets, device pixel ratio, visual viewport, frame evidence, and a uniquely measured CSS element box when available.
|
|
535
|
+
|
|
536
|
+
A measured mapping requires matching pre/post samples and image dimensions that exactly match the CSS crop at the sampled DPR. It includes document-space `crop: { x, y, width, height }` and independent `pixelsPerCssPixel: { x, y }`. This is bracketed measurement, not an atomic capture guarantee. Viewport crop starts at sampled document scroll; full-page crop starts at document origin. Scrolled/ambiguous/non-CSS element captures, child-frame uncertainty, changed geometry, zoom/offset visual viewports, and captures without samples (including native batch captures) remain unknown rather than borrowing final-page geometry.
|
|
537
|
+
|
|
538
|
+
Native mouse coordinates are **current viewport CSS pixels**. For measured geometry, original image pixel `(px, py)` maps to document CSS `(crop.x + px / scale.x, crop.y + py / scale.y)`; subtract current document scroll for viewport coordinates only after rechecking frame/scroll context. Prefer current refs/locators. Pi resize notes map sent-image pixels to original-image pixels only, not CSS coordinates. No provider image-detail override or model-specific coordinate rewrite is applied.
|
|
539
|
+
|
|
540
|
+
Conditional `changed: false` captures have no new image/artifact attachment. A path in JSON is not vision input. Ordinary screenshot attachments use existing verification/size checks; code explicitly selects a handle from its current call. Unknown geometry permits inspection, but cannot justify coordinate input.
|
|
541
|
+
|
|
777
542
|
### Content
|
|
778
543
|
|
|
779
544
|
Primary content should be:
|
|
@@ -822,7 +587,7 @@ Stable category fields are part of the machine-readable contract:
|
|
|
822
587
|
- `successCategory`: present on successful results. Current values are `"completed"`, `"artifact-pending"`, `"artifact-saved"`, `"artifact-unverified"`, and `"inspection"`. `artifact-pending` means a recording started but its file is not expected until `record stop`; use the exact `stop-pending-recording` next action and verify the resulting file. Dispatched `record start` and URL-bearing `record restart` attempts append one proactive `Page state:` warning, even on failure, describing conservative ref invalidation, not an observed page change; caller-requested `--json` carries it in `warnings`. Batch warnings require a reached result row. Preflight failures, missing binaries, help, plain restarts and unconfirmed planned rows do not claim a recording page change; the wrapper also invalidates the session’s prior ref snapshot (`refSnapshotInvalidation.reason: "page-transition"`) so old `@e…` refs fail as `stale-ref` until a fresh `snapshot -i`; that invalidation is attempt-scoped to protect older supported natives (0.37 normally keeps the active page and heap) and also covers `record restart` with a URL operand, while a plain `record restart` keeps the page and refs. Failed results also retain that action whenever their artifact rollup still contains a pending recording, except when the live daemon policy permits only cleanup: those results offer `close-pending-recording` instead, explicitly abandoning the unverified recording. `artifact-unverified` means upstream reported success but the merged `artifactVerification` summary still has unverified non-missing rows; inspect its counts and per-entry `state` / optional `limitation` before treating artifacts as durable evidence.
|
|
823
588
|
- `failureCategory`: present on failed results. Current values are `"aborted"`, `"artifact-missing"`, `"cleanup-failed"`, `"confirmation-required"`, `"download-not-verified"`, `"missing-binary"`, `"parse-failure"`, `"policy-blocked"`, `"qa-failure"`, `"script-error"`, `"selector-not-found"`, `"selector-unsupported"`, `"stale-ref"`, `"tab-drift"`, `"tab-gone"`, `"timeout"`, `"upstream-error"`, and `"validation-error"`. `artifact-missing` means upstream reported a saved/completed artifact path, but the wrapper verified the non-pending file is absent and failed closed.
|
|
824
589
|
|
|
825
|
-
For `
|
|
590
|
+
For `agent_browser_code`, the top-level category describes source execution, not an assertion that every browser call succeeded. `codeRun` and visible `failures` retain handled failures; uncaught source exceptions use `script-error`. The selected browser retains ordinary ownership and is not closed at cell completion. See [code failure and persistence](#failure-and-persistence).
|
|
826
591
|
|
|
827
592
|
These categories are intentionally bounded and stable so agents can branch on them instead of parsing prose. They do not replace raw diagnostics: `details.error`, `details.stderr`, `details.parseError`, `details.validationError`, and visible content still preserve the specific upstream or wrapper message after normal redaction.
|
|
828
593
|
|
|
@@ -830,11 +595,11 @@ These categories are intentionally bounded and stable so agents can branch on th
|
|
|
830
595
|
|
|
831
596
|
For argv-supplied `--allowed-domains`, the wrapper treats domain containment as launch-scoped. Upstream owns request, worker, popup, and WebRTC containment plus incompatible-mode rejection; the wrapper passes the setting and upstream result through unchanged.
|
|
832
597
|
|
|
833
|
-
Real Pi custom tools only mark a tool result failed when the tool throws during `execute`; returned `isError` fields are not authoritative. The extension therefore also registers a `tool_result` handler that treats
|
|
598
|
+
Real Pi custom tools only mark a tool result failed when the tool throws during `execute`; returned `isError` fields are not authoritative. The extension therefore also registers a `tool_result` handler for the direct, code, and advanced browser tools that treats a result with `details.resultCategory: "failure"` as a real Pi tool error. For normal prose output, it appends `Result category: failure; failureCategory: …; Pi tool isError: true.` to model-visible text. For caller-requested `--json` output, it only patches `isError` and preserves visible parseable JSON content unchanged. The TUI renderer also repeats that category line at the top of failed rendered results so collapsed failed rows keep the outcome visible. The hook treats `--json` as requested when echoed `details.args` or the original tool `input.args` includes that flag; it skips appending the prose notice when any non-empty text content item is parseable JSON, even if other text items are not parseable. Invalid or non-JSON text still gets the visible prose notice. Implementation: `buildAgentBrowserToolResultPatch` in `extensions/agent-browser/lib/pi-tool-rendering.ts`; `extensions/agent-browser/index.ts` registers the handler. This keeps Pi transcript semantics aligned with the machine-readable result contract, including wrapper-side reclassifications such as `qa-failure` after an upstream-successful batch and `artifact-missing` after an upstream-successful artifact command whose requested file is absent.
|
|
834
599
|
|
|
835
600
|
For `batch`, top-level `details` still carries `resultCategory` plus `successCategory` or `failureCategory` for the **aggregate** tool outcome: if any step fails, the overall result is a failure (`resultCategory: "failure"`) even when later steps succeed—inspect `batchSteps[]` for per-step outcomes. Each `batchSteps[]` entry includes its own `resultCategory` and either `successCategory` or `failureCategory` for that step. When upstream reports it, successful and failed rows also expose a dedicated `lifecycle` field containing only the bounded `effectiveLaunch.browserLaunched` boolean; on failed rows this preserves launch evidence even though the ordinary result payload is omitted. Live state and transcript replay use that evidence to distinguish a terminal nested close from a post-close browser launch. `batchFailure.failedStep` duplicates the first failing step’s details, including its `failureCategory`, bounded lifecycle evidence, and any `nextActions`.
|
|
836
601
|
|
|
837
|
-
Top-level `details.data` on `batch` is a compact per-step roll-up (not a verbatim replay of raw upstream batch JSON): each element is `{ success, command, result? | error? }` where `command` is argv-redacted the same way as echoed invocation args (including `clipboard write` text, `cookies set` cookie values, `storage local|session set` values, and other sensitive flags/positionals), `result` is the presentation-layer data for that step after the same structured redaction as non-batch commands, and `error` is failure text with clipboard-write/cookie/storage/password literals stripped when those values appeared in argv. Prefer `batchSteps[]` for full per-step `details` (artifacts, categories, spill paths); use the roll-up when you only need a redacted matrix of what ran. If a large batch/
|
|
602
|
+
Top-level `details.data` on `batch` is a compact per-step roll-up (not a verbatim replay of raw upstream batch JSON): each element is `{ success, command, result? | error? }` where `command` is argv-redacted the same way as echoed invocation args (including `clipboard write` text, `cookies set` cookie values, `storage local|session set` values, and other sensitive flags/positionals), `result` is the presentation-layer data for that step after the same structured redaction as non-batch commands, and `error` is failure text with clipboard-write/cookie/storage/password literals stripped when those values appeared in argv. Prefer `batchSteps[]` for full per-step `details` (artifacts, categories, spill paths); use the roll-up when you only need a redacted matrix of what ran. If a large batch/QA result is compacted and spilled, the inline compacted text still includes bounded failed-step context (first failing step, failure category, failure detail, and any failed-step spill path) before the preview and top-level `Full output path:`.
|
|
838
603
|
|
|
839
604
|
`details.refSnapshot` may appear after successful `snapshot` calls and subsequent same-session calls. It records the latest page-scoped ref ids known to the wrapper, optional per-ref accessible `role`/`name` metadata from the same snapshot, and the page target they came from so mutation-prone `@e…` commands can fail fast instead of silently hitting recycled refs after navigation. For wrapper-tracked Electron sessions, `details.electronRefFreshness` may also appear after a successful `@e…` mutation as a softer same-URL rerender warning: run `snapshot -i` before reusing old refs even if the URL did not change.
|
|
840
605
|
|
|
@@ -844,23 +609,25 @@ Ref preflight details (command taxonomy in `extensions/agent-browser/lib/command
|
|
|
844
609
|
- **URL alignment:** `refSnapshot.target.url` and the session’s current tab URL are compared via `targetsMatch` / `normalizeComparableUrl` in `extensions/agent-browser/index.ts`: values are trimmed, parsed as URLs when possible, compared **after dropping the `#fragment`**, and the query string remains significant. If either side lacks a `url`, `targetsMatch` treats the pair as matching so early-session calls are not blocked.
|
|
845
610
|
- **Batch stdin ordering:** user `batch` JSON is scanned in order. Any step whose first token satisfies `isRefInvalidatingBatchCommand` sets a latch that blocks later steps whose first token satisfies `isRefGuardedCommand` and that mention `@e…` refs, except for same-snapshot native form-control steps whose current snapshot role metadata identifies all refs as safe controls (`check`/`uncheck` or direct `click`/`tap` on checkbox or radio refs, and `select` on combobox refs). A step whose first token is `snapshot` clears that latch for subsequent steps (pre-spawn intent only; it does not wait for upstream success). These predicates read explicit command capability flags from `command-taxonomy.ts`: navigation/mutation verbs such as `open` / `goto`, `reload`, non-form `click`, and related upstream commands have `invalidatesBatchRefs`, and `record start` steps (any outcome), `record restart` steps with a URL operand, plus WebMCP `invoke` / `result` / `cancel` steps also set the latch because upstream swaps or navigates the active page; same-snapshot `fill` rows and the role-checked native form-control rows stay guarded against missing/stale refs but do not set the latch, allowing ordinary form batches before a final click/submit step. Direct `click`/`tap @e…` is only treated as a safe form-control row when every ref in that step is a latest-snapshot checkbox or radio; other click/tap refs remain invalidating. Ref-guarded commands accept page-scoped refs for interaction (`click`, `fill`, `download`, `scrollintoview` / `scrollinto`, and others centralized in the command taxonomy). Changing either capability requires updating this contract, [`docs/SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) `RQ-0072`/`RQ-0087` notes, README and command-reference pitfalls, and `test/agent-browser.extension-validation.test.ts`.
|
|
846
611
|
|
|
847
|
-
**Presentation redaction (implementation map):** Successful non-`batch` tool calls and each successful `batchSteps[]` row run upstream `data` through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation/diagnostics.ts`: `cookies` still walk objects/arrays and replace case-insensitive `value` keys with `"[REDACTED]"`; `storage` redacts values when the key or value looks credential-like (token, cookie, auth, secret, JWT, bearer/basic credential, high-entropy token-like string, or nested sensitive JSON) but keeps low-risk primitive QA values such as booleans, numbers, and short strings visible. Redacted storage entries add `valueRedacted` plus `valueRedactionReason` in `details.data`; diagnostic formatters mirror the same decision. Every other command’s payload is recursively scrubbed with the shared `redactSensitiveValue`, which redacts known sensitive key names and applies string-level sensitivity heuristics so network, diff, trace/profiler, stream, dashboard, chat, and other structured results do not echo bearer tokens, proxy credentials, or similar fields verbatim into `details.data`. Echoed `command` arrays in `details` and in batch roll-ups use `redactInvocationArgs` from `extensions/agent-browser/lib/runtime.ts` to mask trailing values for sensitive global flags (including `--body`, `--headers`, `--password`, and `--proxy`), preserve the special positional rules for `cookies set`, `storage local|session set`, and `set credentials`, and scrub other argv tokens for URLs and inline secrets. Failed batch steps additionally run `redactExactValues` on structured step errors so literals taken from that step’s argv (cookie value, storage set value, `--password` / `--password=` tokens) cannot reappear inside formatted error blobs. When the full batch is large enough to need its own aggregate spill, that spill reapplies these per-command data and argv redactors before persistence rather than using generic batch redaction.
|
|
612
|
+
**Presentation redaction (implementation map):** Successful non-`batch` tool calls and each successful `batchSteps[]` row run upstream `data` through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation/diagnostics.ts`: `cookies` still walk objects/arrays and replace case-insensitive `value` keys with `"[REDACTED]"`; `storage` redacts values when the key or value looks credential-like (token, cookie, auth, secret, JWT, bearer/basic credential, high-entropy token-like string, or nested sensitive JSON) but keeps low-risk primitive QA values such as booleans, numbers, and short strings visible. Redacted storage entries add `valueRedacted` plus `valueRedactionReason` in `details.data`; diagnostic formatters mirror the same decision. Every other command’s payload is recursively scrubbed with the shared `redactSensitiveValue`, which redacts known sensitive key names and applies string-level sensitivity heuristics so network, diff, trace/profiler, stream, dashboard, chat, and other structured results do not echo bearer tokens, proxy credentials, or similar fields verbatim into `details.data`. Echoed `command` arrays in `details` and in batch roll-ups use `redactInvocationArgs` from `extensions/agent-browser/lib/runtime.ts` to mask trailing values for sensitive global flags (including `--body`, `--headers`, `--password`, and `--proxy`), preserve the special positional rules for `cookies set`, `storage local|session set`, and `set credentials`, and scrub other argv tokens for URLs and inline secrets. Raw batch row strings are parsed and command-redacted for presentation and ordered intent/completion journals; execution argv remains unchanged. Failed batch steps additionally run `redactExactValues` on structured step errors so literals taken from that step’s argv (cookie value, storage set value, `--password` / `--password=` tokens) cannot reappear inside formatted error blobs. When the full batch is large enough to need its own aggregate spill, that spill reapplies these per-command data and argv redactors before persistence rather than using generic batch redaction.
|
|
613
|
+
|
|
614
|
+
`nextActions` is an optional list of exact follow-ups. Each entry has a `tool`, `id`, `reason`, optional `safety`, and executable `params` or an `artifactPath`. Native actions use `agent_browser` with argv/stdin; Electron and network-source actions use their actual advanced tool name and flat params. Enable the corresponding capability with `agent_browser_tools` if needed. Both prose and JSON expose recovery; oversized observations retain complete payloads in the named observation spill rather than inventing shortened calls.
|
|
848
615
|
|
|
849
|
-
|
|
616
|
+
Browser actions preserve exact session/namespace (including empty namespace) unless fresh-session recovery deliberately omits the old session. Stale refs require a fresh snapshot; unknown targets require `get url` before snapshot; timeouts preserve partial progress and avoid blind mutation retries; covered clicks recommend overlay inspection; recordings retain stop/status/cleanup evidence; confirmations retain exact native IDs. Existing evidence predicates govern locator/ref fallbacks, rich-input recovery, tab correction, selector visibility, and artifact checks. Code-bound actions cannot change its fixed identity; execute cross-identity or namespace-wide recovery outside the cell.
|
|
850
617
|
|
|
851
618
|
**Unknown-command getter hints (failure presentation):** `buildErrorPresentation` in `extensions/agent-browser/lib/results/presentation/errors.ts` only runs this path when upstream error text (after model-facing redaction) matches `unknown command`, `unknown subcommand`, or `unrecognized command` (case-insensitive) **and** the failed invocation’s primary command token is one of `attr`, `count`, `html`, `text`, `title`, `url`, or `value`. Visible text then includes a grouped-`get` hint line plus per-token guidance (`get text <selector>`, `get html …`, `get attr …`, `get count …`, `get value …`, `get title`, `get url`). Machine `nextActions` with ids `use-get-title` / `use-get-url` are emitted only for `title` / `url`, with `params.args` optionally prefixed by `--session <name>` when the failed call targeted a named session. If the error string already contains `Agent-browser hint:` from selector recovery (stale-ref or unsupported selector dialect appendages), the getter block is skipped so two stacked `Agent-browser hint:` headers are not emitted.
|
|
852
619
|
|
|
853
|
-
For `network requests`, `details.nextActions` is bounded to one selected safe request ID, preferring actionable failed rows, then API/fetch-like rows, then benign failed rows, then the first request with a safe ID. Detail/filter/HAR actions use `params.args` and preserve known `--namespace <name>` / `--session <name>` prefixes when the current presentation has `details.namespace` / `details.sessionName`; source-candidate actions use `
|
|
620
|
+
For `network requests`, `details.nextActions` is bounded to one selected safe request ID, preferring actionable failed rows, then API/fetch-like rows, then benign failed rows, then the first request with a safe ID. Detail/filter/HAR actions use `params.args` and preserve known `--namespace <name>` / `--session <name>` prefixes when the current presentation has `details.namespace` / `details.sessionName`; source-candidate actions use flat `agent_browser_network_source` params with the selected `requestId` plus `namespace` / `session` when known and are only emitted for actionable failed rows that the failed-request analyzer can correlate. A `clear-network-requests-before-repro` action can run `network requests --clear` so the next reproduction starts with a current-page-focused diagnostic buffer. Wrapper-side `network requests --current-page` / `--current-origin` renders only rows whose URL matches the active page origin, while `--current-url` renders exact active-document URL matches; those wrapper-only flags are removed before upstream spawn and reported in `details.networkRequestsPageFilter`. Filtered results pass through the same `redactPresentationData` credential redactor as ordinary network diagnostics before entering `details.data` or an `outputPath` export. URLs and query strings are not copied into action params; path filters are skipped when they look sensitive or too large. If the wrapper has observed `network route` in the same session, matching failed, pending, or CORS-looking fetch/XHR rows also add `details.networkRouteDiagnostics[]` with `{ reason, routePattern, mode, requestId?, requestUrl?, summary }` and prepend executable route-mock next actions (`inspect-routed-network-request`, `start-network-har-capture-for-route-mock`) before generic request follow-ups; same-origin/CORS fixture retry guidance stays in visible prose. The route tracker is wrapper-session-local, updated on successful `network route`/`network unroute`, and cleared when that session closes or is replaced.
|
|
854
621
|
|
|
855
622
|
For `batch`, each `batchSteps[]` entry can carry its own `nextActions` for that step’s success or failure. Top-level `details.nextActions` on a failed batch duplicates `batchFailure.failedStep.nextActions` so callers can read one aggregate object. On a fully successful batch, top-level `nextActions` may still list artifact follow-ups derived from the combined step artifacts.
|
|
856
623
|
|
|
857
624
|
`pageChangeSummary` is an optional compact summary for mutation-prone and artifact-producing commands. It includes `changeType` (`"navigation"`, `"mutation"`, `"artifact"`, or `"confirmation"`), `observed`, `command`, a readable `summary`, optional `title`/`url`, optional `artifactCount` or `savedFilePath`, and `nextActionIds`. `observed: false` means upstream dispatched a mutation-capable action but the wrapper did not observe an application change; standalone results also append a visible `Action dispatched; application change unverified` warning. The wrapper maintains explicit command/subcommand capability checks through `isPageChangeSummaryCommand` in `extensions/agent-browser/lib/command-taxonomy.ts`: those commands still emit a `mutation`-typed summary when upstream JSON lacks navigation metadata, as long as no stronger signal (artifact, saved path, navigation fields, or pending confirmation) applies. That capability is independent from `invalidatesBatchRefs` and `triggersPostMutationSnapshot`, so artifact summaries like `download` / `screenshot` and guarded-but-non-invalidating `fill` are documented directly in the capability table instead of implied by broad set spreading. Commands outside that set omit `pageChangeSummary` unless the parsed payload shows navigation, a confirmation prompt, saved files, or artifacts—including read-only inspection commands, which normally have no summary unless one of those signals appears. For `batch`, the top-level summary favors artifact rollups when any step produced artifacts; otherwise it synthesizes an observed-or-unverified summary from step evidence. Visible batch output promotes dispatch-only mutation evidence before step details and states that fixed waits are not postconditions. Agents should verify URL/text/state or an external receipt for important mutations before continuing.
|
|
858
625
|
|
|
859
|
-
`clickDispatch` may appear after a **top-level non-Electron** direct `click` when the wrapper installed a target-specific DOM-event probe, upstream reported success, and the post-click probe found no trusted DOM event reached the resolved target. Target-specific probes cover `xpath=` targets and role-gated `@e…` refs when the latest wrapper-tracked snapshot has role/name metadata; eligible ref roles are `button`, `checkbox`, `menuitem`, `radio`, `switch`, and `tab`, and the role/name must be unique in both the saved snapshot and the live candidates. Before trusting either a ref or XPath candidate, the wrapper sets a per-probe temporary DOM attribute and requires native `get attr <original-selector> <marker>` readback to match it. Missing or mismatched identity, including XPath/frame scope mismatches, cleans up the probe and leaves the native click to run without dispatch verification. This does not implement a new accessible-name algorithm or promise that every ref is observable. Duplicate-name refs pass through without a probe because snapshot order can change and is not proof of target identity. Raw `find … click` locator calls, including compiled `
|
|
626
|
+
`clickDispatch` may appear after a **top-level non-Electron** direct `click` when the wrapper installed a target-specific DOM-event probe, upstream reported success, and the post-click probe found no trusted DOM event reached the resolved target. Target-specific probes cover `xpath=` targets and role-gated `@e…` refs when the latest wrapper-tracked snapshot has role/name metadata; eligible ref roles are `button`, `checkbox`, `menuitem`, `radio`, `switch`, and `tab`, and the role/name must be unique in both the saved snapshot and the live candidates. Before trusting either a ref or XPath candidate, the wrapper sets a per-probe temporary DOM attribute and requires native `get attr <original-selector> <marker>` readback to match it. Missing or mismatched identity, including XPath/frame scope mismatches, cleans up the probe and leaves the native click to run without dispatch verification. This does not implement a new accessible-name algorithm or promise that every ref is observable. Duplicate-name refs pass through without a probe because snapshot order can change and is not proof of target identity. Raw `find … click` locator calls, including compiled `agent_browser_action` clicks that still execute as upstream `find`, are not probed because the wrapper has no concrete element before upstream resolves the locator, and document-level probes can falsely fail frame-scoped clicks. It does **not** take a fresh pre-click snapshot because that could recycle upstream refs before the intended click. The wrapper does **not** replay clicks in-page. On a miss it marks the tool failed, appends `Click dispatch diagnostic: …`, and sets `clickDispatch.status` to `"no-native-event-observed"` with `reason: "native-click-produced-no-target-dom-event"`, `nativeEventCount`, and a redacted `target` descriptor (`kind: "xpath"` plus `selector`, or `kind: "accessible"` plus `refId`, `role`, and redacted `name`). `details.nextActions` gains `inspect-click-dispatch-miss` (`snapshot -i`) and `retry-click-after-dispatch-miss` (same upstream click argv, session-prefixed when applicable). If a local static fixture must be exercised despite this diagnostic, a caller may explicitly run a programmatic activation via `eval --stdin` such as `document.querySelector(...).click()`, but that emits an untrusted scripted event and is only a debugging/workaround path; it must not be used as proof that real user-like clicking works or to bypass prompt stop boundaries. This diagnostic is only for standalone top-level direct click calls; `find` locator clicks and native batch/QA click steps remain upstream-owned behavior.
|
|
860
627
|
|
|
861
628
|
`promptGuard` may appear on wrapper-blocked calls only for concrete, machine-checkable prompt requirements. `reason: "requested-artifacts-missing-before-close"` blocks `close` / `quit` / `exit` when the prompt used a direct screenshot/recording creation phrase with a destination such as `here`, `at`, `as`, or `to` and the session artifact manifest has not verified that exact path; a destination heading can carry that intent across contiguous plain or Markdown-bulleted path-only list lines. Bare, review-only, fenced-reference, conditional, permissive/uncertain, directly negated, and Pi clipboard/attachment image/video paths are treated as input, not output requirements, while subordinate requirements such as “do not close until you save” remain output intent. A recording-availability qualifier may precede, appear within, or follow its path/list and is scoped through the next path boundary; if the same path appears more than once, any required occurrence takes precedence. Markdown-link destinations resolve to the destination path. Explicitly optional artifacts are not close requirements. The classifier is deliberately conservative, so use wording such as `Save a screenshot here: <path>` when machine-enforced close blocking matters. Optional recording paths are only required when recording appears available. The wrapper does **not** parse broad user/business intent such as “do not place the order” or “do not post anything” into click/key blocks; agents must follow those instructions themselves. Prompt guards return `failureCategory: "policy-blocked"` and `validationError` text instead of invoking upstream.
|
|
862
629
|
|
|
863
|
-
`overlayBlockers` may appear on a successful `snapshot` whose own refs show strong modal context, or after a successful **top-level non-Electron** `click` (the unified `details.command` is `click`, not
|
|
630
|
+
`overlayBlockers` may appear on a successful `snapshot` whose own refs show strong modal context, or after a successful **top-level non-Electron** `click` (the unified `details.command` is `click`, not native batch/QA flows that compile to `batch`) only when upstream JSON includes a string `data.clicked` ref (`@e…` / `ref=`), no `clickDispatch` diagnostic fired for the same result, the session’s prior pinned tab URL (`priorSessionTabTarget.url`) and `details.navigationSummary.url` both exist and stay equal after the same URL normalization used for ref preflight (trimmed hosts/paths; **`#fragment` dropped** while the query string stays significant), and the wrapper did not apply session tab correction or an about-blank mismatch recovery in the same result. Wrapper-tracked Electron clicks prefer lifecycle health and ref-freshness diagnostics because desktop app chrome produced too many false overlay candidates in dogfood. For post-click diagnostics, the wrapper uses existing `details.navigationSummary.url` as same-URL evidence, then issues **one** extra session-scoped `snapshot -i`; CSS selector clicks do not run this overlay probe. For snapshot diagnostics, it scans that snapshot result directly. It only emits diagnostics when **both** are true: at least one ref has a strong modal role (`dialog` or `alertdialog`), and there are up to **three** separate `button`/`link`/`menuitem` refs whose names match close/dismiss-style patterns (for example “Close”, “Dismiss”, “No thanks”, or a lone `×`). Page-wide text such as “privacy”, “sign in”, or “banner” without a dialog role is not enough, which avoids warning on ordinary same-page menu opens or app button mutations. Each candidate carries `ref` (`@eN`), optional `role`/`name`, exact `click` argv in `args`, and a short evidence `reason`. The struct also includes a `summary` string (one sentence describing the snapshot/click evidence and likely dismiss controls) plus a `snapshot` object (same shape as `details.refSnapshot` after a normal snapshot): on success the wrapper may treat that snapshot as the session’s latest ref map for subsequent calls, so agents should assume refs can move to match this post-diagnostic tree. Visible text appends the same bullets under `Possible overlay blockers`, and `details.nextActions` gains `inspect-overlay-state` plus `try-overlay-blocker-candidate-1`…`3` after any presentation `nextActions` (for example `inspect-after-mutation`); when `details.sessionName` is set, those appended actions preserve session context; namespaced sessions use `--namespace <namespace> --session <name>` and non-namespaced sessions use `--session <name>` unless argv already carries that context. This is conservative evidence, not proof the candidate should be clicked; prefer `inspect-overlay-state` first unless the dismiss control is clearly safe.
|
|
864
631
|
|
|
865
632
|
Example shape (fields vary by scenario):
|
|
866
633
|
|
|
@@ -876,7 +643,7 @@ Example shape (fields vary by scenario):
|
|
|
876
643
|
]
|
|
877
644
|
```
|
|
878
645
|
|
|
879
|
-
When `
|
|
646
|
+
When `agent_browser_action` produced compiled `find` argv and the unified result is `failureCategory: "stale-ref"` with `details.compiledSemanticAction` still present, `nextActions` chains snapshot refresh then the compiled `find` retry; `select` shorthands with stale `@refs` stop at refresh guidance. `reason` / `safety` strings match `buildAgentBrowserNextActions` in `extensions/agent-browser/lib/results/action-recommendations.ts` and the append in `extensions/agent-browser/index.ts`:
|
|
880
647
|
|
|
881
648
|
```json
|
|
882
649
|
"nextActions": [
|
|
@@ -924,44 +691,45 @@ Additional structured fields can appear when relevant:
|
|
|
924
691
|
- `lifecycle: { effectiveLaunch: { browserLaunched } }` when upstream returned that boolean. It separates starting the requested `agent-browser` CLI process from the effective Chrome session context. `readSource` exposes upstream's string `data.source` for direct `read` calls and identifies the raw HTTP fetch path; its lifecycle boolean can be `false` before a browser launch or `true` when the same managed session already has an active browser. Direct reads also append one visible `Read execution` line with the source, CLI-start result, managed browser lifecycle, and managed-session outcome so Pi models do not have to infer model-invisible details.
|
|
925
692
|
- `browserWindow: { mode: "headed", ownership: "wrapper-managed", sessionName, visibility: "unverified" }` only after a successful first/fresh local wrapper-managed headed call (including `batch`) that is not an attachment and whose lifecycle proves a browser launched and whose managed-session outcome is `created` or `replaced`. One visible handoff sentence tells the user to complete the login in that window if they can see it, then continue with `sessionMode: "auto"`; the field never claims OS desktop visibility.
|
|
926
693
|
- `sessionTabTargetUnknown: true` after a spawned `connect`, `state load`, history navigation, tab-selection/close, `window new`, or `diff url` call changes the active page without a trustworthy observed target. Direct `window new` / `diff url` and reached native batch rows retire the old target and refs; an intentional new blank window or an observed blank URL-diff destination never triggers old-tab recovery. URL diff inputs do not prove the final URL after redirects. Successful standalone tab selection/close now live-probes URL and a fresh non-blank title before state is committed, even when the new tab shares the prior URL; explicit selection of an existing `about:blank` tab and a post-close blank target are retained, so this marker remains only when the probe cannot verify the target. It is persisted and restored across branch/reload replay, clears stale refs and tab pinning, and blocks page inspection until `get url` or explicit navigation observes a target; `tab list`, tab selection, close, and blocking-dialog `status` / `accept` / `dismiss` remain available. A timeout against an unknown target removes standalone snapshot actions and returns `verify-page-target-after-timeout`, a session-scoped `batch --bail` whose stdin runs `get url` before `snapshot -i`.
|
|
927
|
-
- `compiledSemanticAction` when the call used `
|
|
928
|
-
- `compiledJob`
|
|
929
|
-
- `compiledQaPreset` when the call used `
|
|
930
|
-
- `compiledSourceLookup` when the call used `
|
|
931
|
-
- `sourceLookup` when the call used `
|
|
932
|
-
- `compiledNetworkSourceLookup` / `networkSourceLookup` when the call used `
|
|
933
|
-
- `qaPreset` when the call used `
|
|
694
|
+
- `compiledSemanticAction` when the call used `agent_browser_action` and the result includes the unified `details` merge: `{ action, locator, args }` for `find` actions or `{ action: "select", selector?, locator?, values, args }` for `select`, with the same redaction rules as `args` / `effectiveArgs`; omitted for plain direct native calls and omitted on some early error returns that omit this field (see the `agent_browser_action` section above)
|
|
695
|
+
- `compiledJob` is internal QA batch-plan metadata: `{ args: ["batch", "--bail"], failFast: true, stdin, steps }`, with redacted step argv. It does not expose a public job input or compiler.
|
|
696
|
+
- `compiledQaPreset` when the call used `agent_browser_qa`: the compiled native batch fields plus the QA `checks` object. `args` is `batch --bail` and `failFast` is `true` for QA presets. `checks.attached` is `true` for current-session QA, `checks.url` is present only for URL-opening QA, and `checks.diagnosticsResetAtStart` is `true` only for URL-opening QA because `agent_browser_qa` with `attached: true` preserves existing session diagnostics.
|
|
697
|
+
- `compiledSourceLookup` when the call used `agent_browser_source`: `{ args: ["batch"], stdin, steps, query }` with the generated local-evidence plan and original query fields (`selector?`, `reactFiberId?`, `componentName?`, `includeDomHints?`, `maxWorkspaceFiles?`).
|
|
698
|
+
- `sourceLookup` when the call used `agent_browser_source`: `{ status, candidates, limitations, summary, workspaceRoot?, electronContext? }`; wrapper-tracked packaged Electron no-candidate diagnostics may carry `workspaceRoot` plus `electronContext` and live Electron nextActions without marking the successful batch as a tool failure.
|
|
699
|
+
- `compiledNetworkSourceLookup` / `networkSourceLookup` when the call used `agent_browser_network_source`: the generated batch plan plus bounded failed-request/candidate evidence as described above.
|
|
700
|
+
- `qaPreset` when the call used `agent_browser_qa`: `{ passed, failedChecks, warnings, summary }`. `failedChecks` includes “page-error check could not be verified” when final errors match a nonempty post-clear baseline; this sets `passed: false` without claiming those matched rows are new application errors. Definitely new rows are counted separately as page errors. Network rows inside the `network requests` batch step use `summarizeNetworkFailures` / `classifyNetworkRequestFailure` in `network.ts`: actionable failures appear in `failedChecks` (and fail the tool when the upstream batch still succeeded); benign icon-classified failures appear only in `warnings` and in `summary` as `QA preset passed with warnings: …` when nothing else failed.
|
|
934
701
|
- `networkRouteDiagnostics` after successful `network requests` when the wrapper has observed active `network route` patterns for that session and a matching request row is pending/no-status or carries CORS/preflight-looking error text. Each row includes `reason` (`"pending-routed-request"` or `"cors-likely-routed-request"`), `routePattern`, `mode`, optional `requestId`, optional `requestUrl`, and `summary`; visible text starts with `Network route diagnostics`, and `details.nextActions` prepends executable route-mock inspection/HAR follow-ups before generic request follow-ups.
|
|
935
|
-
- `compiledElectron` when the call used `
|
|
936
|
-
- `electron` when the call used `
|
|
702
|
+
- `compiledElectron` when the call used `agent_browser_electron`: redacted action plan for `list`, `launch`, `status`, `cleanup`, or `probe`.
|
|
703
|
+
- `electron` when the call used `agent_browser_electron`: action-specific lifecycle, discovery, probe, and cleanup data; see the `agent_browser_electron` contract above.
|
|
937
704
|
- `batchFailure` and `batchSteps` for `batch` rendering, including mixed-success runs
|
|
938
705
|
- `navigationSummary` for navigation-style commands like `click`, `back`, `forward`, `reload`, `window new`, `diff url`, and successful standalone tab selection/close; reached window-new/URL-diff rows also probe the final page after native batches, including failed batches that may already have changed the page (not aborted or timed-out calls). Unreached rows do not change target or ref state; `urlChanged` records whether the live URL differs from a known pinned pre-command URL, so same-URL clicks and clicks without a comparison baseline remain dispatch-only rather than being mislabeled as observed navigation. Helper probes run `get url` first and run `get title` for any verified non-`about:blank` URL. The title read is skipped when the probed URL already carries a wrapper-observed title for this session, except after a tab selection/close, which always refreshes a non-blank title even when the URL is unchanged: titles are last-observed labels for that URL, while the URL itself is live-probed on every call. Href-less CSS selector clicks use this same post-command helper so `sessionTabTarget` cannot stay on the pre-click page; any click-dispatch check still runs first. A failed non-batch `eval`, `back`, `forward`, `reload`, `connect`, `state load`, or `tab` selection also runs this helper (browser started, not aborted, not watchdog-timed-out), so an observed page stays verified instead of forcing a manual `get url` round trip; a failed or empty probe keeps the prior unverified-page behavior. Because a failed transition can still have mutated or replaced the document, a successful probe also invalidates the prior page-scoped ref snapshot (matching the previous unknown-target behavior, which dropped refs), so the next `@e…` use requires a fresh `snapshot -i`.
|
|
939
706
|
- `pageChangeSummary` for compact mutation/artifact/navigation summaries on commands that can change browser state
|
|
940
707
|
- `clickDispatch` when a top-level non-Electron direct `click` reported upstream success but the target-specific probe found no trusted event reached the resolved XPath or accessible `@ref` target; shape follows `ClickDispatchDiagnostic` in `extensions/agent-browser/lib/orchestration/browser-run/types.ts`
|
|
941
708
|
- `promptGuard` when the requested-artifact-before-close guard blocks browser close before required prompt artifact paths are verified; implementation lives in `extensions/agent-browser/lib/orchestration/browser-run/prompt-guards.ts`
|
|
942
709
|
- `overlayBlockers` for conservative overlay/banner/dialog blocker candidates when a successful snapshot itself contains strong modal evidence, or after a qualifying top-level `@e…` / `ref=` click stays on the same URL, no `clickDispatch` diagnostic fired, and a fresh snapshot provides evidence (`candidates`, `summary`, and `snapshot` per `OverlayBlockerDiagnostic` in `extensions/agent-browser/index.ts`). CSS selector clicks do not run this overlay probe.
|
|
943
|
-
- `visibleRefFallback` after a raw `find` or compiled `
|
|
710
|
+
- `visibleRefFallback` after a raw `find` or compiled `agent_browser_action` fails with `selector-not-found` and a fresh snapshot finds exact role/name `@ref` matches. Shape follows `VisibleRefFallbackDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, snapshot, summary, target }`, where each candidate has `ref`, `role`, `name`, optional direct ref `args`, and `reason`; visible text appends `Current snapshot ref fallback`. Non-fill candidates with direct args add `try-current-visible-ref` or numbered `try-current-visible-ref-N` actions. Fill candidates omit direct args and target text so recovery details do not repeat potentially sensitive fill text.
|
|
944
711
|
- `refSnapshotInvalidation` after a confirmed cold managed-session shutdown (`reason: "page-transition"`, including when reopening fails), a session `snapshot` fails with `No active page`, any upstream-executed `record start` attempt or URL-bearing `record restart` conservatively invalidates refs (including failures, for older-native protection rather than proof of a page change), a direct or reached batch `window new` / `diff url` attempt changes the page, or a failed non-batch transition command (`eval`, `back`, `forward`, `reload`, `connect`, `state load`, `tab` selection) whose live URL re-verification probe observed the page (a failed transition can still have mutated the document, so the verified URL is kept but the prior refs are not). Shape follows `SessionRefSnapshotInvalidation` in `extensions/agent-browser/lib/session-page-state.ts`: `{ reason: "no-active-page" | "page-transition", summary }`; replay preserves the persisted summary. The wrapper deletes prior refs for that session, persists the invalidation for resume, and blocks mutation-prone `@e…` preflight with `failureCategory: "stale-ref"` until a successful fresh `snapshot -i` records refs again.
|
|
945
712
|
- `snapshotFilter` after wrapper-side `snapshot -i --search <text>` or `snapshot -i --filter role=<role>`. Shape: `{ cleanArgs, search?, role?, matchedRefs, totalRefs, visibleLines, totalLines, renderedTextMatches?, renderedTextTotalMatches?, renderedTextTruncated? }`. Search runs one bounded read-only rendered-DOM probe across the full document; each visible match carries bounded `text`, `tagName`, `kind` (`text` or prioritized `validation`), `offscreen`, optional `role`/accessible `name`, and a unique mapped `ref` when the full snapshot supports it. Hidden elements are excluded. The filtered accessibility snapshot remains separate, while `details.refSnapshot` still records the full upstream ref map for later stale-ref checks.
|
|
946
713
|
- `snapshotViewport` after wrapper-side `snapshot --viewport` (with or without `-i`, `--search`, or `--filter`). Shape matches the scroll-position probe: viewport scroll offsets, inner/document dimensions, sampled scrollable-container count, and bounded container offsets. The wrapper strips `--viewport` before upstream spawn and gathers this with a read-only `eval --stdin` call.
|
|
947
714
|
- `snapshotDiff` after wrapper-side `snapshot --diff` (with or without `-i`, `--search`, `--filter`, or `--viewport`). Shape: `{ addedRefs, removedRefs, changedRefs, unchangedRefs, summary }`, comparing ref ids plus role/name metadata from the previous wrapper-tracked snapshot for the session with the newly returned full ref map. It is a quick ref-map delta, not a visual diff.
|
|
948
715
|
- `networkRequestsPageFilter` after wrapper-side `network requests --current-page`, `--current-origin`, or `--current-url`. Shape: `{ cleanArgs, currentUrl, mode, matchedRows, totalRows }`; the visible rows and `details.data.requests` / `items` / `entries` are filtered while the active session page target is read with `get url`.
|
|
949
|
-
- `richInputRecovery` after a raw `find` or compiled `
|
|
716
|
+
- `richInputRecovery` after a raw `find` or compiled `agent_browser_action` `fill` fails with `selector-not-found` and the same current-ref diagnostic finds exact editable `searchbox` / `textbox` candidates. Shape follows `RichInputRecoveryDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, inputMethodHint, nextActionIds, summary, target }`, where each candidate has `ref`, `role`, `name`, `focusArgs`, `clickArgs`, and `reason`. Visible text appends `Rich input recovery`, and `details.nextActions` gains ids from `getAgentBrowserRichInputRecoveryNextActionIds`: `focus-current-editable-ref` / `click-current-editable-ref` (or numbered variants). These actions are bounded to focus/click/inspect-style recovery: they do not include the fill text, do not press `Enter`, and do not submit. After the right current editable ref is focused, use `keyboard type` for framework-controlled editors that require real key events. Use paste-like `keyboard inserttext` only with separate application-state verification, and submit only when explicitly required by the flow.
|
|
950
717
|
- unsupported `scrollintoview text=<label>` / `scrollinto text=<label>` fails before upstream dispatch, directly or inside an effective raw/stdin batch row, because current upstream can report success without moving the page; help forms pass through unchanged. Visible failure text and `details.nextActions` both return the session-scoped `scroll-semantic-text-target` (`find text <label> hover`) when hover side effects are acceptable, plus `refresh-refs-for-scroll-target` (`snapshot -i`) before `scrollintoview <@ref>`. CSS, `xpath=...`, and current `@e…` targets remain native pass-through.
|
|
951
718
|
- `scrollPage` when the wrapper moves `document.scrollingElement` directly for `scroll <up|down|left|right> [px|percent]` or `scroll to end|top`; it temporarily disables smooth scrolling so immediate before/after offsets are reliable, returns `{ request, result }`, and includes `exitCode: 0` on success. Directional document no-movement falls through to upstream wheel behavior so nested panes still work. Explicit CSS-container calls `scroll <selector> <up|down|left|right> [px|percent]` remain wrapper-handled, use `Element.scrollBy` with `behavior: "instant"` before measuring movement so smooth-scroll CSS cannot cause a premature no-movement failure, and report `details.scrollContainer`. All scroll helper shims are skipped when startup-scoped flags are present so the requested browser/profile launches before any helper command.
|
|
952
719
|
- `scrollNoop` after a nominally successful large **top-level** upstream scroll fallback on an existing or fresh managed session when wrapper-side read-only probes before and after the command show no change in `window.scrollX` / `window.scrollY` and no change in the sampled prominent scrollable containers. The wrapper reclassifies this outcome as `failureCategory: "upstream-error"` rather than claiming the page scrolled. To avoid pre-launching a session without caller startup state, this probe is skipped for small pixel scrolls, calls that would create a managed session only for the probe, and invocations with startup-scoped flags such as `--profile`, `--state`, `--restore`, `--namespace`, `--session-name`, `--cdp`, providers, init scripts, or similar launch settings. Shape: `{ reason: "no-observed-scroll-position-change", message, before, after, recommendations }`; `before` / `after` include viewport dimensions, document scroll dimensions, and up to ten sampled container descriptors plus scroll offsets. Container descriptors use only sample index, tag name, and ARIA role; DOM ids/classes are intentionally not stored. This diagnostic is conservative evidence that the page-level scroll likely missed a nested pane, not proof that every app-specific region is unchanged. Visible text starts with `Scroll completed with no observed movement`, appends `Scroll diagnostic: no observed scroll movement`, sets `details.data.scrolled` to `false` / `details.data.noMovement` to `true`, and `details.nextActions` gains `inspect-after-noop-scroll` (`snapshot -i`) plus `verify-noop-scroll-visually` (`screenshot`), session-prefixed when applicable.
|
|
953
|
-
- `comboboxFocus` after a successful explicit combobox-targeted `click` / `fill` / `find … click|fill` (for example `
|
|
720
|
+
- `comboboxFocus` after a successful explicit combobox-targeted `click` / `fill` / `find … click|fill` (for example `agent_browser_action` with role `combobox`, including when that semantic action resolves through a current visible `@ref` before execution) when a read-only probe sees the active element is combobox-like, `aria-expanded` is explicitly present (`false` or `true`), and no visible `listbox` / `option` / menu option elements are open. Shape: `{ reason: "focused-combobox-without-visible-options", message, activeElement, visibleListboxCount, visibleOptionCount, recommendations }`; `activeElement` includes bounded role/tag/expanded/hasPopup/name metadata with normal text redaction. Visible text appends `Combobox diagnostic: focused combobox did not expose visible options`, and `details.nextActions` gains `inspect-focused-combobox` (`snapshot -i`), `try-open-combobox-with-arrow` (`press ArrowDown`), and `try-open-combobox-with-enter` (`press Enter`), session-prefixed when applicable. The diagnostic is deliberately gated to explicit combobox-targeted calls to avoid extra probes or false positives on ordinary clicks/textboxes.
|
|
954
721
|
- `recordingDependencyWarning` after a successful `record start` or `record restart` when the wrapper cannot find an executable `ffmpeg` on the Pi process `PATH`. Shape: `{ reason: "ffmpeg-missing-for-recording", dependency: "ffmpeg", command, message, recommendations }`. Visible text appends `Recording dependency warning: ffmpeg not found on PATH`. This is a non-blocking warning after native success, not a preflight or encoding check. Native 0.37 validates ffmpeg at startup; older supported natives may defer failure. Treat the pending output as unverified, stop and check its result, then install ffmpeg before starting a new recording.
|
|
955
722
|
- `selectorTextVisibility` after a **successful** upstream `get text <selector>` (standalone or inside a successful `batch`) when the wrapper’s follow-up probe finds a hazard: more than one DOM match (upstream reads the first `querySelectorAll` hit, which may be the wrong tab/panel), or the first match is hidden while at least one other match is visible (requires multiple DOM nodes so a visible peer exists; a lone hidden match is not flagged). The probe is a read-only `eval --stdin` script (`buildVisibleTextProbeScript` in `extensions/agent-browser/lib/orchestration/browser-run/diagnostics.ts`) that counts matches, applies a small visibility heuristic (`display`/`visibility`/`opacity` plus non-zero client rects), may include a redacted `firstVisibleTextPreview`, and may include up to eight `visibleCandidates` entries (`index` in `querySelectorAll`, `tagName`, optional `role`, optional redacted `textPreview`). It is **not** run for simple id selectors, page-scoped `@e…` selectors, or when the selector string is withheld because `selectorMayExposeSensitiveLiteral` would risk echoing secrets in probe output. `details.selectorTextVisibility` mirrors the primary diagnostic (first sorted entry); when several selectors in one `batch` qualify, `selectorTextVisibilityAll` lists every diagnostic sorted so hidden-first cases precede generic multi-match ambiguity. Appended visible warning text names the matching `details.nextActions` id and may list visible candidate previews. Appended `details.nextActions` use ids `inspect-visible-text-candidates` and `inspect-visible-text-candidates-2`, … with the probe replayed via `eval --stdin` for each hazardous selector. If the probe still leaves more than one visible candidate, it is only ambiguity evidence; agents should narrow the selector, use a current visible `@ref`, or run a targeted visible-element `eval --stdin` rather than trusting the broad selector.
|
|
956
723
|
- `electronGetTextScopeWarning` after a successful wrapper-tracked attached Electron `get text <selector>` (standalone or successful `batch`) when a broad non-ref CSS selector such as `body`, `html`, `main`, `div`, or `[role=application]` may read the whole app shell. Ordinary browser pages do not qualify without wrapper-owned Electron launch provenance. Shape: `{ selector, summary, electronContext: { launchId?, sessionName?, url? } }`; multiple batched diagnostics use `electronGetTextScopeWarnings`. Visible text appends `Broad Electron get text selector warning`, and next actions use `snapshot-for-electron-text-scope` ids with session-scoped `snapshot -i` payloads.
|
|
957
724
|
- `evalStdinHint` after a successful `eval --stdin` when caller stdin (trimmed) looks function-shaped to the wrapper’s lightweight detector (in `extensions/agent-browser/lib/orchestration/browser-run/diagnostics.ts`: leading `function` / `async function`, parenthesized arrow `(…) =>`, or a concise `name =>` / `async name =>` form) **and** upstream JSON `data` is an object whose `result` field is a plain empty object (`{}`). Arrays such as `[]` do not qualify. It includes `reason` and `suggestion`; visible output appends `Eval stdin hint` with the same guidance. This is a heuristic for the common mistake of returning a function object instead of invoking it or passing a plain expression, not a JavaScript parser or proof that the page returned no useful data. Before this diagnostic path runs, the wrapper also recovers the common malformed native-tool call `args: ["eval", "--stdin", "..."]` with no top-level `stdin` by moving trailing `args` tokens after `--stdin` into the process stdin stream.
|
|
958
725
|
- `evalResultWarning` after a successful `eval --stdin` when the current or prior page URL is `file:` (from navigation summary, session tab target, or persisted session page state), upstream JSON `data.result` is strictly `null`, and stdin is non-empty and not a trivial literal `null`/`undefined`. Fields: `reason`, `suggestion`. Visible output appends `Eval result warning` without failing the tool. Use snapshot -i, ref-based getters, screenshots, or http(s) fixtures when file:// null results are inconclusive.
|
|
959
|
-
- `timeoutPartialProgress` after `runAgentBrowserProcess` reports `timedOut` (wrapper child-process watchdog) when best-effort recovery finds useful context. `summary` is a short sentence counting recovered planned-step state and declared artifact paths, plus whether page context came from live session reads or only from a planned URL (when nothing in the plan declares an artifact path, the fraction may read `0/0` while `currentPage` can still carry session or planned URL context). `steps` lists planned argv from the compiled
|
|
726
|
+
- `timeoutPartialProgress` after `runAgentBrowserProcess` reports `timedOut` (wrapper child-process watchdog) when best-effort recovery finds useful context. `summary` is a short sentence counting recovered planned-step state and declared artifact paths, plus whether page context came from live session reads or only from a planned URL (when nothing in the plan declares an artifact path, the fraction may read `0/0` while `currentPage` can still carry session or planned URL context). `steps` lists planned argv from the compiled QA batch plan (`compiledJob` remains its internal metadata name) or, when that object is absent, from the effective upstream `batch` source: raw argument command strings exclusively when present, otherwise JSON-array stdin, whether caller-authored or wrapper-generated by advanced source tools (1-based indices). Ignored stdin does not contribute recovery steps or artifact evidence. Generated QA rows may include `generatedFrom`. Each step includes `status` (`completed`, `failed`, `pending`, or `unknown`) and optional `reason`; the first incomplete step becomes `retryStep`, but `retry` and top-level `retry-timeout-step` are emitted only for read-only or idempotent commands such as waits, snapshots, screenshots, navigation, and diagnostics. Each retry uses `args: ["batch"]` with `stdin` containing the one original row, preserving native row operands instead of reinterpreting them as outer CLI globals. Mutating steps such as clicks, fills, keyboard typing, presses, selects, or checks are still identified as the first incomplete step but omit executable retry args because they may already have run; when the timed-out session is still usable and its target is already verified, `details.nextActions` can instead include `inspect-current-page-after-timeout` (`snapshot -i`) so the agent verifies current state before continuing with a shorter split flow. When the target is unknown, every standalone snapshot action is removed and replaced by one executable `verify-page-target-after-timeout` action: a session-scoped `batch --bail` with stdin `[["get","url"],["snapshot","-i"]]`, so snapshot runs only after the wrapper's page-target guard is satisfied; visible failure text prints those redacted args and short stdin rather than pointing only to structured details. Dialog `status`, `accept`, and `dismiss` remain allowed while the target is unknown so timeout dialog recovery actions are executable. When a retryable step timed out during `sessionMode: "fresh"` and no live URL was recovered, `retry-timeout-step` uses top-level `sessionMode: "fresh"` instead of prefixing the abandoned generated session name. `currentPage` comes from session-scoped `get url` followed by `get title` when the session answers, otherwise a fallback URL may be inferred from the last `open` / `navigate` / `pushstate` step in the plan; `liveUrlRecovered` is true only when the wrapper recovered a live URL, so planned URLs are not treated as proof that the page actually opened. `openedButPostOpenTimedOut` is set when a live opened page was recovered and a later step appears to have timed out. `artifacts` covers declared output paths on `screenshot`, `pdf`, `download`, and `wait --download` steps (absolute path, existence, `state`, optional `sizeBytes`, `stepIndex`). It uses native operand positions, including literal dash-leading paths: first operand for `pdf`, second for `download`, and the next retained operand after the first timeout pair is removed for `wait --download` / `-d`. Visible text repeats the same block under `Timeout partial progress`, applying URL and path-segment redaction; the prose `Planned steps` list shows at most six steps, then an omitted-count line when the plan is longer. This is recovery evidence only; missing entries do not prove the upstream step never ran or that no other side effects occurred.
|
|
960
727
|
- `managedSessionHeadedAutosaveInterval` on active/current-after-failure wrapper-owned headed session rows, containing the canonical effective launch-time `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` string (invalid or out-of-range explicit values resolve to upstream's `"30000"` default). It is `"0"` for the wrapper default and can hold an explicit interval such as `"1000"`; transcript replay and still-owned off-current helpers reapply the recorded value. It is omitted for sessionless, headless, caller-owned, abandoned, and closed calls. If a resumed Pi process explicitly requests a different value in either direction, non-close calls fail with close-plus-fresh recovery guidance while close still uses the recorded daemon value.
|
|
961
728
|
- `managedSessionHeadedAutosaveDisabled: true` is the narrower compatibility marker that the targeted session uses the wrapper's default interval `0`, rather than an explicit caller interval. It accompanies `managedSessionHeadedAutosaveInterval: "0"` on active rows and remains omitted for explicitly configured autosave.
|
|
962
|
-
- `managedSessionOutcome` after a managed-session plan reaches process execution (`buildManagedSessionOutcome` / `formatManagedSessionOutcomeText` in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`). Populated when `buildExecutionPlan` injects an extension-managed implicit or fresh `--session`, and also when a successful explicit `--session <current-wrapper-managed-session> close` closes the current managed session. It remains omitted for unrelated explicit user-managed sessions and for sessionless inspection/local paths that skip injection. Successful nested-batch lifecycle rows are evaluated in order: a terminal close reports and replays `status: "closed"` even when aggregate artifact verification makes the tool result fail; a later lifecycle-proven browser launch (including a post-close `record stop`) keeps the session active, an explicitly non-launching diagnostic leaves it closed, and an unknown row stays conservatively active even when the failed batch was the first managed call. Fields: `status` (`created`, `replaced`, `unchanged`, `closed`, `preserved`, or `abandoned`), `sessionMode`, `attemptedSessionName`, `previousSessionName`, `currentSessionName`, optional `currentSessionNamespace`, optional `replacedSessionName`, optional `replacedSessionNamespace`, optional `replacedSessionClosed` (false means automatic close failed and the previous session remains wrapper-owned/restorable for explicit cleanup), `activeBefore`, `activeAfter`, `succeeded`, and `summary` (machine-oriented; may include generated session names). Use `currentSessionNamespace` with `currentSessionName` when following preserved-session recovery actions; retry-fresh actions stay in the attempted namespace. Model-visible echo: when `sessionMode` is `"fresh"` **and** `succeeded` is false, or when `replacedSessionClosed` is false after a replacement, the wrapper appends action-oriented `Managed session outcome` and `Recovery` lines without repeating generated session ids in visible prose; session names remain in `details.managedSessionOutcome`. Failed fresh launches may also append `details.nextActions` such as `run-agent-browser-doctor`, `verify-current-managed-session`, `snapshot-current-managed-session`, or `retry-fresh-managed-session`. When other trailing diagnostic prose is also emitted in the same result, that block is concatenated **after** semantic-action candidate lines, overlay/selector-visibility tails, eval hints/warnings, and `Timeout partial progress` (see `rawAppendedDiagnosticText` in `extensions/agent-browser/lib/orchestration/browser-run/final-result.ts`). For `"auto"` failures the same struct may appear on `details` without that extra line. When post-upstream analysis (for example **`qa`** preset failure) flips the overall tool result after a successful batch, or a fresh
|
|
729
|
+
- `managedSessionOutcome` after a managed-session plan reaches process execution (`buildManagedSessionOutcome` / `formatManagedSessionOutcomeText` in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`). Populated when `buildExecutionPlan` injects an extension-managed implicit or fresh `--session`, and also when a successful explicit `--session <current-wrapper-managed-session> close` closes the current managed session. It remains omitted for unrelated explicit user-managed sessions and for sessionless inspection/local paths that skip injection. Successful nested-batch lifecycle rows are evaluated in order: a terminal close reports and replays `status: "closed"` even when aggregate artifact verification makes the tool result fail; a later lifecycle-proven browser launch (including a post-close `record stop`) keeps the session active, an explicitly non-launching diagnostic leaves it closed, and an unknown row stays conservatively active even when the failed batch was the first managed call. Fields: `status` (`created`, `replaced`, `unchanged`, `closed`, `preserved`, or `abandoned`), `sessionMode`, `attemptedSessionName`, `previousSessionName`, `currentSessionName`, optional `currentSessionNamespace`, optional `replacedSessionName`, optional `replacedSessionNamespace`, optional `replacedSessionClosed` (false means automatic close failed and the previous session remains wrapper-owned/restorable for explicit cleanup), `activeBefore`, `activeAfter`, `succeeded`, and `summary` (machine-oriented; may include generated session names). Use `currentSessionNamespace` with `currentSessionName` when following preserved-session recovery actions; retry-fresh actions stay in the attempted namespace. Model-visible echo: when `sessionMode` is `"fresh"` **and** `succeeded` is false, or when `replacedSessionClosed` is false after a replacement, the wrapper appends action-oriented `Managed session outcome` and `Recovery` lines without repeating generated session ids in visible prose; session names remain in `details.managedSessionOutcome`. Failed fresh launches may also append `details.nextActions` such as `run-agent-browser-doctor`, `verify-current-managed-session`, `snapshot-current-managed-session`, or `retry-fresh-managed-session`. When other trailing diagnostic prose is also emitted in the same result, that block is concatenated **after** semantic-action candidate lines, overlay/selector-visibility tails, eval hints/warnings, and `Timeout partial progress` (see `rawAppendedDiagnosticText` in `extensions/agent-browser/lib/orchestration/browser-run/final-result.ts`). For `"auto"` failures the same struct may appear on `details` without that extra line. When post-upstream analysis (for example **`qa`** preset failure) flips the overall tool result after a successful batch, or a fresh native batch opens the requested page and then a later step fails, the managed-session transition still reflects that the fresh browser became current. The visible recovery says the fresh launch became current and points to `failureCategory` / `qaPreset` / `batchFailure` for the post-launch failure instead of telling the agent that the old session was preserved.
|
|
963
730
|
- `imagePath` / `imagePaths` for Pi inline image attachments from the **`screenshot`** command (including batched screenshot steps). **`diff screenshot`** still records the diff output as an `image`-kind entry in `details.artifacts`, but it does **not** populate `imagePath` / `imagePaths` or attach an inline image: only plain `screenshot` is treated as a trusted live-capture path for automatic inlining (`isTrustedScreenshotOutput` in `extensions/agent-browser/lib/results/presentation/artifacts.ts`).
|
|
964
|
-
- `artifacts` for saved files such as screenshots, `state save` outputs, `diff screenshot` diff images, PDFs, downloads, `wait --download` / `wait -d` files, traces, CPU profiles, completed video recordings, path-bearing HAR captures, and future recording output paths reported by `record start` / `record restart`. Non-file URL payloads such as `data:` / `blob:` / `http(s):` values are not treated as verified local artifacts. For direct artifact commands and batch artifact steps, the wrapper creates parent directories for requested paths before spawning upstream. Filesystem `mkdir` failures at this shared preparation boundary return `validation-error`, `agentBrowserStarted: false`, the attempted directory and `verify-artifact-path` guidance. When execution cwd differs from the launch/project root, relative file operands in raw batch strings are bound to the captured invocation root; otherwise raw strings stay unchanged and absolute artifact paths are preferred. Each artifact includes the original saved or requested `path`, resolved `absolutePath`, `kind`/`artifactType`, optional `mediaType`, optional `extension`, best-effort disk metadata such as `exists`, `sizeBytes`, and `updatedAtMs`, plus `requestedPath`, `status`, `cwd`, `session`, `namespace`, and `tempPath` when applicable. `requestedPath` is retained only when known from the caller, separately from reported/resolved locations; a differing screenshot report remains in `tempPath` and is displayed as `Reported path`, whether it is a temporary file or a canonical path alias. Ordinary file `mediaType` values come from bounded PNG/JPEG/GIF/WebP header recognition, not suffixes; unknown, missing, unreadable or truncated headers leave it undefined. Header recognition is not full-file format validation. Inline screenshot attachments use the same byte classifier and existing size limit. For commands that create/update artifacts, a path that existed but was not updated during this command uses `status: "stale"`; observational `wait --download` may accept a file completed just before the wait began. Pending `record start` / `record restart` artifacts use `status: "pending"`, omit `exists` rather than reporting false, and include `recordingState: "openRecording"` / `willExistOnStop: true`. Within one Pi extension process, the wrapper keeps an unbounded transcript-backed active-recording reservation index separate from the bounded artifact manifest, keyed by canonical namespace plus session; still-live process-owned reservations survive branch switches, while known closures are appended after tree navigation and during shutdown/reload so a close on one branch cannot be resurrected after returning to an older branch. Persisted active reservations require absolute storage paths and cwd; their display paths may remain relative. If a journal append fails, the next serialized browser boundary, tree navigation, or shutdown retries all current reservations and known closures. `recordingPersistenceWarning` and visible warning text remain present while restart protection is not durable; successful recovery is quiet and cleanup still runs. Artifact lifecycle calls, explicit `wait --download <path>` / `wait -d <path>` destinations, and result `outputPath` writes serialize around the global destination check/update, every successful direct, ordered nested-batch, fresh-replacement,
|
|
731
|
+
- `artifacts` for saved files such as screenshots, `state save` outputs, `diff screenshot` diff images, PDFs, downloads, `wait --download` / `wait -d` files, traces, CPU profiles, completed video recordings, path-bearing HAR captures, and future recording output paths reported by `record start` / `record restart`. Non-file URL payloads such as `data:` / `blob:` / `http(s):` values are not treated as verified local artifacts. For direct artifact commands and batch artifact steps, the wrapper creates parent directories for requested paths before spawning upstream. Filesystem `mkdir` failures at this shared preparation boundary return `validation-error`, `agentBrowserStarted: false`, the attempted directory and `verify-artifact-path` guidance. When execution cwd differs from the launch/project root, relative file operands in raw batch strings are bound to the captured invocation root; otherwise raw strings stay unchanged and absolute artifact paths are preferred. Each artifact includes the original saved or requested `path`, resolved `absolutePath`, `kind`/`artifactType`, optional `mediaType`, optional `extension`, best-effort disk metadata such as `exists`, `sizeBytes`, and `updatedAtMs`, plus `requestedPath`, `status`, `cwd`, `session`, `namespace`, and `tempPath` when applicable. `requestedPath` is retained only when known from the caller, separately from reported/resolved locations; a differing screenshot report remains in `tempPath` and is displayed as `Reported path`, whether it is a temporary file or a canonical path alias. Ordinary file `mediaType` values come from bounded PNG/JPEG/GIF/WebP header recognition, not suffixes; unknown, missing, unreadable or truncated headers leave it undefined. Header recognition is not full-file format validation. Inline screenshot attachments use the same byte classifier and existing size limit. For commands that create/update artifacts, a path that existed but was not updated during this command uses `status: "stale"`; observational `wait --download` may accept a file completed just before the wait began. Pending `record start` / `record restart` artifacts use `status: "pending"`, omit `exists` rather than reporting false, and include `recordingState: "openRecording"` / `willExistOnStop: true`. Within one Pi extension process, the wrapper keeps an unbounded transcript-backed active-recording reservation index separate from the bounded artifact manifest, keyed by canonical namespace plus session; still-live process-owned reservations survive branch switches, while known closures are appended after tree navigation and during shutdown/reload so a close on one branch cannot be resurrected after returning to an older branch. Persisted active reservations require absolute storage paths and cwd; their display paths may remain relative. If a journal append fails, the next serialized browser boundary, tree navigation, or shutdown retries all current reservations and known closures. `recordingPersistenceWarning` and visible warning text remain present while restart protection is not durable; successful recovery is quiet and cleanup still runs. Artifact lifecycle calls, explicit `wait --download <path>` / `wait -d <path>` destinations, and result `outputPath` writes serialize around the global destination check/update, every successful direct, ordered nested-batch, fresh-replacement, code, Electron, or shutdown close retires only its exact identity at that lifecycle point, and destination reuse is rejected through lexical, existing or dangling symlink, hardlink, full Unicode-fold, or macOS/Windows case aliases. Batch preflight rejects `record start` / `record restart` after a close row because upstream can report a recording that did not start; split those operations into separate calls. A `No recording in progress` stop failure, direct or nested, first checks the matching native receipt once. Retirement preserves the receipt and freshly checked file metadata; it never turns an existing file into a missing file merely because no recording is active. A later successful batch recording row opens its new pending path normally. Batch preflight applies the same distinct-destination rule to the steps upstream will execute: raw argument command strings exclusively when any exist, stdin arrays only otherwise; upstream-ignored stdin rows cannot fail artifact preflight, add pending recordings, or create parent directories. Parent directories are prepared for the effective steps in both modes; outside execution-directory binding, raw argument strings stay unchanged, so screenshot normalization and tracked path requests apply to stdin rows only. Outer CLI globals are removed before artifact parsing, but native batch row operands stay literal: `pdf --quick ignored.pdf` targets `--quick`, not `ignored.pdf`. Reservation checks, preparation, and requested-path presentation follow that same distinction. Recording path/URL consumers skip complete numeric `--fps` pairs without rewriting argv; native still validates rate, format and extra arguments. FPS-only calls keep the intended pinned tab.
|
|
732
|
+
|
|
965
733
|
|
|
966
734
|
Recording destinations are reserved within one Pi process, not across processes. Use unique paths for concurrent Pi processes: different explicit sessions can overwrite one file even when both `record stop` results are verified. Upstream’s same-session `record start` guard does not reserve the filename across other sessions.
|
|
967
735
|
- `savedFilePath` / `savedFile` for direct `download`, `pdf`, and `wait --download` / `wait -d` saved-file workflows when a host file path is reported or wrapper-verified. Batch results preserve the same fields on the relevant `batchSteps` entry. These fields are metadata only until `artifactVerification` verifies the file. Native `download <selector> <path>` always owns the click and download, including loopback links, generated Blob exports, and redirects; the wrapper does not substitute a fetch of an anchor's `href`.
|
|
@@ -976,13 +744,13 @@ Additional structured fields can appear when relevant:
|
|
|
976
744
|
- `inspection: true` plus `stdout` for successful plain-text inspection commands like `--help` and `--version`
|
|
977
745
|
- valid sessionless `upgrade` commands accept native text output with surrounding whitespace trimmed and normal redaction, using the existing scalar `details.data` result. They are not inspection calls and do not claim or replace a managed browser session. Caller-requested `--json` stays a parseable `{ success, data, error? }` result. Nonzero exits, structured upstream errors, spawn failures, timeout and cancellation remain failures even when text looks successful or a terminated child exits zero; failed native text remains diagnostic data alongside the error and stderr. Caller argv and session planning are unchanged.
|
|
978
746
|
- `versionValidation` on a browser-backed preflight failure when installed upstream output is not a stable version at or above the supported floor; it includes `expected` and optional parsed `observed`, while top-level `expectedVersion` remains the recommended current baseline, `minimumSupportedVersion` reports the floor, and `observedVersion` reports the installed version. The extension caches a successful `agent-browser --version` probe per cwd/PATH for the Pi process; plain help/version, close recovery, and sessionless local commands remain available without this browser-backed gate.
|
|
979
|
-
- `agentBrowserStarted` on results that reached browser-run processing: `false` proves the requested main subprocess never started (for example a socket-path, policy, or spawn preflight failure); `true` proves only that the CLI started, not that Chrome launched. Use `details.lifecycle.effectiveLaunch.browserLaunched` for the latter. Preparation helpers may already have touched the
|
|
747
|
+
- `agentBrowserStarted` on results that reached browser-run processing: `false` proves the requested main subprocess never started (for example a socket-path, policy, or spawn preflight failure); `true` proves only that the CLI started, not that Chrome launched. Use `details.lifecycle.effectiveLaunch.browserLaunched` for the latter. Preparation helpers may already have touched the selected browser; an interrupted code intent remains unknown until inspection.
|
|
980
748
|
|
|
981
749
|
When the tool echoes `args` or `effectiveArgs` back into Pi, sensitive values such as `--headers`, proxy credentials, and auth-bearing URL parameters are redacted first. Replacements use `[REDACTED]` (URL-encoded in parsed URLs). Ordinary technical prose such as `bearer token`, `bearer authentication`, and `bearer credentials` stays verbatim; credential fields, Authorization / Proxy-Authorization headers, and explicit header arguments such as `curl -H` still redact their values. Outside those contexts, bearer redaction requires a value matching bearer-token syntax with digits or token punctuation, not just a following word, HTML, or URL.
|
|
982
750
|
|
|
983
751
|
URL redaction covers `code`, SAMLRequest, SAMLResponse, RelayState, and `authorization_session_id` (including common separator/case variants); `state` and `nonce` are redacted only when the same URL token has an auth/login/OAuth/OIDC/SAML/SSO context or another known sensitive query name, so ordinary application state URLs remain useful. URLs needing no redaction keep their original spelling. Exact internal page-target URLs stay unredacted for browser correctness; model-facing content, structured details, persisted spills, and explicit `outputPath` exports receive the same redacted copies.
|
|
984
752
|
|
|
985
|
-
For parse-valid oversized snapshots and other oversized tool outputs, details should switch to a compact metadata object and include `fullOutputPath` pointing at a private spill file with the full redacted upstream payload. When the caller supplied `outputPath`, only matching live wrapper-manifest spills may provide pre-compaction payloads; direct compacted data, compacted result rows, and command-redacted whole-batch data are rehydrated in place, while any unavailable required spill fails without writing compact metadata. Malformed oversized output is not safe to redact structurally, so its temporary subprocess spill is deleted and no durable `fullOutputPath` is returned. The model-facing tool text should print the actual spill-file path when one exists instead of only saying to inspect a details key. Oversized batch/
|
|
753
|
+
For parse-valid oversized snapshots and other oversized tool outputs, details should switch to a compact metadata object and include `fullOutputPath` pointing at a private spill file with the full redacted upstream payload. When the caller supplied `outputPath`, only matching live wrapper-manifest spills may provide pre-compaction payloads; direct compacted data, compacted result rows, and command-redacted whole-batch data are rehydrated in place, while any unavailable required spill fails without writing compact metadata. Malformed oversized output is not safe to redact structurally, so its temporary subprocess spill is deleted and no durable `fullOutputPath` is returned. The model-facing tool text should print the actual spill-file path when one exists instead of only saying to inspect a details key. Oversized batch/QA failures include bounded failed-step context inline before the preview so agents can see the failed assertion/error and failure category without opening the spill file. Persisted sessions should keep that spill file under a private session-scoped artifact directory so the path remains usable after reload/restart. The oldest persisted spill files are evicted as needed to stay within `PI_AGENT_BROWSER_SESSION_ARTIFACT_MAX_BYTES` (default 32 MiB), and those evictions are reported as `artifactManifest.entries[].retentionState: "evicted"` instead of silently disappearing from the session inventory. Set `PI_AGENT_BROWSER_SESSION_ARTIFACT_MAX_BYTES=0` to retain existing and new persistent spill files without automatic byte-budget eviction; unset or invalid values use the default, and positive integer limits retain oldest-first eviction. This does not recover previously evicted files or change temporary subprocess spill cleanup. This persisted-spill byte budget is separate from the recent metadata window controlled by `PI_AGENT_BROWSER_SESSION_ARTIFACT_MANIFEST_MAX_ENTRIES`.
|
|
986
754
|
|
|
987
755
|
## Recording receipts and recovery
|
|
988
756
|
|
|
@@ -1007,7 +775,7 @@ Matching requires the actual namespace/session, expected path and known recordin
|
|
|
1007
775
|
|
|
1008
776
|
`details.readConfirmation` stores only native control-response provenance for an actual explicit URL read: `{ id, sessionName, namespace?, source: "native-explicit-url-read", state: "pending" | "cleared", capabilities? }`. Page content, nested JSON text and bare DOM reads do not establish this provenance. It is replayed in existing per-session transcript state on resume/branch changes and retired by successful confirmation/denial or session close.
|
|
1009
777
|
|
|
1010
|
-
Routing uses the actual native session, including `default` when the original read did not allocate a managed browser. Explicit caller session/namespace choices still win, and
|
|
778
|
+
Routing uses the actual native session, including `default` when the original read did not allocate a managed browser. Explicit caller session/namespace choices still win, and code calls cannot borrow another session's confirmation. Legacy native prompts retain this routing. Only a control response with `capabilities.readRequiresConfirmation: true` also proves native explicit-ID matching and enables matching confirm/deny without page helpers or managed-browser replacement. Legacy/unproven and DOM confirmations retain normal page checks. The advertised capability requires native ID validation before consuming a pending action, so a stale read ID cannot approve a later DOM action. A failed or expired confirmation returns exact session-status guidance, not an automatic retry of another ID.
|
|
1011
779
|
|
|
1012
780
|
A confirmed HTTP failure is a tool failure regardless of that capability, including inside a batch. A pending confirmation is not success even when the native outer envelope says otherwise. If a new DOM confirmation replaces a pending read, the old read marker clears and the new confirmation keeps its own approve/deny actions and normal page checks.
|
|
1013
781
|
|
|
@@ -1023,7 +791,7 @@ Worth doing in v1:
|
|
|
1023
791
|
- `diff screenshot` → same file-artifact pattern as above for the **diff** image path only (summary text uses “Saved diff image” only when the diff output exists; missing output says “Diff image reported; file not verified” and fails as `artifact-missing`); baseline paths and other fields stay in the structured payload but are not echoed as separate saved artifacts in the visible artifact block, and there is no Pi inline image attachment for the diff output
|
|
1024
792
|
- `state load` → completion text may mention the loaded path, but the wrapper does **not** treat that path as a new saved artifact (`artifacts` / `artifactManifest` stay unset) the way `state save` does
|
|
1025
793
|
- auth, cookies, storage, clipboard, dialog, frame, state, network, debug, diff, stream, dashboard, chat, and other structured results → concise summaries that avoid expanding secret-bearing payloads; `state show` exposes metadata only in visible text and redacts every cookie/localStorage/sessionStorage `value` in structured details; credential-like keys, values, URLs, body snippets, bearer/basic credentials, clipboard write text, cookie values, and likely secret storage values are redacted before model-facing output and `details.data`, while benign primitive storage values may remain visible for local QA
|
|
1026
|
-
- TUI display → custom `agent_browser` call/result rendering with colorized command/output text and a built-in-style collapsed view for long visible output;
|
|
794
|
+
- TUI display → custom `agent_browser` call/result rendering with colorized command/output text and a built-in-style collapsed view for long visible output; advanced tools retain their named call rows and compiled plans in audit details; code call rows show terminal-safe source; failed results keep `resultCategory` / `failureCategory` visible before truncated output; `ctrl+o` expansion reveals the full rendered tool result without changing the model-facing content
|
|
1027
795
|
- snapshots → origin + ref count + main-content-first compact preview, with the redacted snapshot spill path printed directly in content and kept in `details.fullOutputPath` plus `details.artifactManifest` when the inline result would otherwise be too large
|
|
1028
796
|
- oversized generic outputs such as large `eval --stdin` payloads → compact preview plus the actual spill file path instead of dumping the whole payload into model context
|
|
1029
797
|
- `read [url]` → upstream `data.content` first, with source/content-type/status/final-URL metadata retained in `details.data`. Explicit URL reads and all-read batches neither allocate/replace a managed browser nor run pre-, post- or timeout page helpers; malformed read syntax is left to native validation rather than treated as a DOM read. Calls targeting an already-owned session retain its daemon settings and launch metadata, including after reload/resume, without changing restore policy or consuming a pending page reopen. Fetched URLs do not replace the browser target or invalidate refs. `Read execution` and `details.readSource` / `agentBrowserStarted` / native `lifecycle` report command evidence, not shared-browser liveness; absent launch evidence stays unknown. Bare `read` keeps normal DOM verification. Explicit read timeouts retain the native `.md` / ancestor-`llms.txt` request budget.
|
|
@@ -1045,8 +813,8 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
1045
813
|
|
|
1046
814
|
`session info` is one explicit, read-only preflight, not an automatic probe before every command. Its text distinguishes daemon `active` / `pid` from native `runtime.browser`: `status`, nullable `alive`, Chrome `pid`, exact `userDataDir`, `tabs`, native `ownership` (`launched`, `attached`, `none`, or `unknown`), and `error`. It also preserves native `runtime.recording.current` / `last` receipts and protocol `capabilities`. `data.piCleanupOwnership` separately reports `caller-owned` or `wrapper-managed` from Pi's actual ownership records; an explicit name can still target a wrapper-managed session. The integration does not infer a Chrome PID/profile from config, scan host processes, launch Chrome or change tabs for this preflight. Missing native fields, an active daemon with `runtime: null`, and legacy `browserLaunched` alone do not prove live browser identity. Name-only `session` responses select a session without proving liveness. Restore check URLs, text and code remain omitted from status text.
|
|
1047
815
|
|
|
1048
|
-
-
|
|
1049
|
-
-
|
|
816
|
+
- use the root-Pi named browser for ordinary unconfigured calls, with a separate current managed slot for explicit fresh/owned launches
|
|
817
|
+
- keep root identity independent of cwd; generated managed-session bases retain their existing Pi-session/cwd scope
|
|
1050
818
|
- respect explicit upstream `--session` and configured native session/namespace defaults with minimal interference; per-call `--config` reaches helpers without changing native precedence
|
|
1051
819
|
- treat the extension-managed session as convenience state owned by the wrapper
|
|
1052
820
|
- preserve the current branch-visible extension-managed session across `/reload`, exact-session relaunch, `/resume`, and Pi `session_tree` branch transitions so persisted sessions can keep following the live browser after lifecycle changes
|
|
@@ -1054,14 +822,14 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
1054
822
|
- set one idle timeout on extension-managed sessions as a backstop for abnormal exits or cleanup failures, and pass that same `AGENT_BROWSER_IDLE_TIMEOUT_MS` to top-level commands plus every helper targeting that owned session; caller-owned sessions retain native idle policy so upstream does not restart the background browser, reset the active tab, or discard current refs between a snapshot and action
|
|
1055
823
|
- clean up process-private temp spill artifacts on shutdown, while keeping persisted-session snapshot spill files in a private session-scoped artifact directory so `details.fullOutputPath` survives reload/restart and the oldest spill files are evicted if the per-session artifact budget is exceeded
|
|
1056
824
|
- reconstruct the current branch-visible extension-managed session, every transcript-proven still-active wrapper-owned identity, latest page-scoped refs, newest-revision aggregate `artifactManifest`, and wrapper-tracked Electron launch records from the active transcript branch on `session_start` and Pi `session_tree` so later default and explicit off-current calls keep following owned managed browsers and can continue reporting artifact retention state; successful explicit wrapper-owned close rows and `electron.cleanup` managed-session steps are restore-visible close events
|
|
1057
|
-
- keep runtime cleanup ownership separate from branch-visible state: `session_tree` restore and wrapper-owned browser commands are serialized with managed-session work; caller-owned explicit-session commands use
|
|
825
|
+
- keep runtime cleanup ownership separate from branch-visible state: `session_tree` restore and wrapper-owned browser commands are serialized with managed-session work; caller-owned explicit-session commands use local queues plus a cooperating-process execution lock keyed by actual socket context and effective canonical namespace/session (explicit namespace argv wins over inherited `AGENT_BROWSER_NAMESPACE`, including an explicit empty default), so the live URL probe, preparation helpers, semantic snapshot, main command, and state commit for one identity cannot interleave while different identities remain concurrent. Namespace-scoped `close --all` is the exception: it drains and exclusively barriers managed plus matching caller-owned work before clearing global namespace state. Namespace and session identity components are additionally Unicode-normalized and case-folded on macOS and Windows to match their case-insensitive daemon paths. Only the outer tool execution acquires that key; nested helpers run under it without re-entry. Policy, route, and artifact deltas survive unrelated managed-state commits, while a separate branch-restore generation guard prevents stale completions from overwriting a newer branch. Concurrent artifact-producing results carry a monotonic aggregate manifest revision so transcript replay selects the complete bounded manifest rather than whichever call happened to occupy the last row. Extension-managed sessions and wrapper-launched Electron records owned by the current process remain eligible for quit/cleanup, and fresh-session allocation stays monotonic across branch restores, including auto rows and close rows that reference wrapper-generated fresh names
|
|
1058
826
|
- when a close command or `electron.cleanup` successfully closes the current wrapper-managed session, clear live page/ref state, reserve the next generated fresh-session ordinal, and rotate the next default auto call to a fresh wrapper-generated session name rather than reusing the closed name
|
|
1059
827
|
- when `/reload` shuts down an extension instance, close off-branch owned managed sessions and off-branch owned Electron launches before clearing process-local ownership; retain attached-browser context for those still-owned off-branch resources so cleanup cannot resend local-launch defaults. Preserve only the current branch-visible active managed session and active Electron launch plus its isolated `userDataDir` for reload continuity, and also persistently protect `userDataDir` paths when partial cleanup intentionally skips or fails profile removal so later temp cleanup, process exit, and stale temp-root pruning after restart do not violate Electron cleanup's safety decision; rebuild active branch state from the active branch on the next `session_start`
|
|
1060
828
|
- when an unnamed `sessionMode: "fresh"` launch succeeds, make it the new extension-managed session so later default calls keep using it
|
|
1061
829
|
- when an unnamed `sessionMode: "fresh"` launch fails or times out, preserve the previous managed session when one was active or report the attempted fresh session as abandoned when no managed session was active (`details.managedSessionOutcome`; visible `Managed session outcome: …` when the final tool call used `sessionMode: "fresh"` and failed, or when automatic close of its replaced session failed—see `#details`)
|
|
1062
830
|
- if that unnamed fresh launch replaced an already-active managed session, best-effort close the old managed session after the switch succeeds; `details.managedSessionOutcome.replacedSessionClosed` records the cleanup result, and `false` keeps the older identity wrapper-owned across transcript resume for explicit follow-up or cleanup
|
|
1063
|
-
- treat every explicit caller-provided `--session` as user-managed, including `piab-*` names. Wrapper-owned implicit sessions set a Pi-transcript- and Git-checkout-generation-scoped `AGENT_BROWSER_RESTORE` key automatically unless disabled with `PI_AGENT_BROWSER_MANAGED_SESSION_RESTORE=0`; explicit caller sessions do not receive that injection unless they exactly target the current wrapper-owned identity. Caller state/restore paths, profiles, upstream config, file access, launch arguments, environment variables, local file pages, `outputPath`, and close arguments pass through unchanged. `session list` and `state list` keep all upstream rows and restore identifiers visible. Automatic restore still validates and pins its own checkout/storage/namespace identity and coordinates same-daemon reuse so the wrapper cannot mix restore pools or corrupt managed lifecycle state. Ambiguous tab, attachment, history,
|
|
1064
|
-
- before a DOM/content-bearing read or interaction against a caller-owned explicit session or established attachment, run a session-scoped `get url` probe so stale transcript state cannot target the wrong page. A failed or non-URL probe blocks the requested content command. The
|
|
831
|
+
- treat every explicit caller-provided `--session` as user-managed, including `piab-*` names. Wrapper-owned implicit sessions set a Pi-transcript- and Git-checkout-generation-scoped `AGENT_BROWSER_RESTORE` key automatically unless disabled with `PI_AGENT_BROWSER_MANAGED_SESSION_RESTORE=0`; explicit caller sessions do not receive that injection unless they exactly target the current wrapper-owned identity. Caller state/restore paths, profiles, upstream config, file access, launch arguments, environment variables, local file pages, `outputPath`, and close arguments pass through unchanged. `session list` and `state list` keep all upstream rows and restore identifiers visible. Automatic restore still validates and pins its own checkout/storage/namespace identity and coordinates same-daemon reuse so the wrapper cannot mix restore pools or corrupt managed lifecycle state. Ambiguous tab, attachment, history, code, or state-load transitions remain page-target correctness boundaries: content calls live-check `get url` or require explicit navigation before acting. Windows uses `cross-spawn` for native executable and `.cmd` argument transport, rather than PowerShell or wrapper-owned argument reordering. Empty operands such as `fill #field ""`, `--args ""`, and explicit default `--namespace ""`, literal doublequotes in fill text, and command/subcommand adjacency are retained. Upstream receives the empty namespace rather than a wrapper omission or environment workaround. The selected child `PATH` shim is not bypassed; POSIX keeps native Node `spawn`.
|
|
832
|
+
- before a DOM/content-bearing read or interaction against a caller-owned explicit session or established attachment, run a session-scoped `get url` probe so stale transcript state cannot target the wrong page. A failed or non-URL probe blocks the requested content command. The shared execution lock orders that probe, semantic snapshot resolution, and the main command across cooperating updated Pi processes. Nested `batch` steps remain unsupported; raw batch command strings mirror upstream's ASCII-space tokenizer, including quoting and backslash handling.
|
|
1065
833
|
- pass explicit `--profile` straight through to upstream `agent-browser`; no profile-cloning or isolation layer is added in v1
|
|
1066
834
|
<!-- agent-browser-playbook:start wrapper-tab-recovery -->
|
|
1067
835
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
@@ -1073,16 +841,59 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
1073
841
|
- If upstream reports tab_gone, the pinned bound tab is gone; use details.nextActions (tab list / tab new) instead of assuming another tab is yours.
|
|
1074
842
|
<!-- agent-browser-playbook:end wrapper-tab-recovery -->
|
|
1075
843
|
- caller-owned sessions honor native `AGENT_BROWSER_SOCKET_DIR` unless the wrapper-specific socket override is set, using the same integrity checks; on other local Unix launches, set a short private socket directory for wrapper-spawned `agent-browser` processes so extension-generated session names do not fail the upstream Unix socket-path length limit in longer cwd/session-name combinations; require an absolute non-symlink directory owned by the current uid with mode `0700`, otherwise fail before spawn. Socket checks trust the operating environment's actual `/`, not its reported UID; all non-root ownership, permission and alias-destination checks remain in force. This is not protection from the controller of the root filesystem; see [socket trust](ARCHITECTURE.md#ownership). Android/Termux uses a short directory under the owner-only `/data/data/<package>` app sandbox, compacts generated managed identities to one 80-bit digest so ordinary namespace plus fresh-session paths remain within the limit, stores policy-lock coordination under `os.tmpdir()`, and probes process identity with Termux's `ps` beside Node instead of unavailable `/bin/ps`
|
|
1076
|
-
- keep wrapper-spawned commands bounded by clamping `AGENT_BROWSER_DEFAULT_TIMEOUT` to the upstream documented default of 25 seconds while the default wrapper child-process watchdog is 35 seconds (`PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides it, and top-level `timeoutMs` overrides it per call for browser CLI subprocesses). Explicit `wait <ms>`, `wait --timeout <ms>`, and WebMCP `invoke` / `result --timeout <ms>` calls can exceed that default; when top-level `timeoutMs` is omitted, the wrapper derives a per-call subprocess watchdog from the requested command duration plus a small grace window. Dialog commands use `PI_AGENT_BROWSER_DIALOG_PROCESS_TIMEOUT_MS` (default 5000 ms), and click/tap/find refs or tokens plus `eval --stdin` snippets whose text looks like alert/confirm/prompt/dialog triggers use `PI_AGENT_BROWSER_DIALOG_TRIGGER_PROCESS_TIMEOUT_MS` (default 8000 ms). Timed-out
|
|
844
|
+
- keep wrapper-spawned commands bounded by clamping `AGENT_BROWSER_DEFAULT_TIMEOUT` to the upstream documented default of 25 seconds while the default wrapper child-process watchdog is 35 seconds (`PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides it, and top-level `timeoutMs` overrides it per call for browser CLI subprocesses). Explicit `wait <ms>`, `wait --timeout <ms>`, and WebMCP `invoke` / `result --timeout <ms>` calls can exceed that default; when top-level `timeoutMs` is omitted, the wrapper derives a per-call subprocess watchdog from the requested command duration plus a small grace window. Dialog commands use `PI_AGENT_BROWSER_DIALOG_PROCESS_TIMEOUT_MS` (default 5000 ms), and click/tap/find refs or tokens plus `eval --stdin` snippets whose text looks like alert/confirm/prompt/dialog triggers use `PI_AGENT_BROWSER_DIALOG_TRIGGER_PROCESS_TIMEOUT_MS` (default 8000 ms). Timed-out advanced QA or native batch calls may add `details.timeoutPartialProgress` and visible `Timeout partial progress` evidence with per-step status, retry payloads, current page title/URL, and declared artifact path checks; timed-out dialog-like commands may add dialog status/dismiss/fresh-session recovery next actions
|
|
1077
845
|
- interactive or long-running upstream families such as `chat` without a prompt, `dashboard start`, `stream enable`, `trace start`, `profiler start`, `record start`, `inspect`, `install`, `upgrade`, `doctor --fix`, and `confirm-interactive` are passed through thinly but remain bounded by the same wrapper timeout/session planning rules; prefer explicit arguments, single-shot `chat <message>`, non-interactive flags like `doctor --offline --quick` or `doctor --json`, and cleanup pairs such as `dashboard stop`, `stream disable`, `trace stop`, `profiler stop`, and `record stop`
|
|
1078
846
|
- treat successful plain-text inspection commands like `--help` and `--version` as stateless: do not inject the implicit managed session and do not let those calls claim the managed-session slot
|
|
1079
847
|
- if startup-scoped flags like `--profile`, `--args`, `--user-agent`, `--executable-path`, `--ca-cert`, `--no-ca-cert`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--enable`, `-p` / `--provider`, or iOS `--device` target the current active managed session, return a validation error with a structured recovery hint that recommends `sessionMode: "fresh"`; when the call explicitly names the current managed session, remove that `--session` from the recovery payload so rotation is actionable. If daemon inspection proves an explicitly targeted older wrapper-owned session active, reject its startup-scoped flags with close-first or fresh-rotation guidance before spawn
|
|
1080
848
|
- for direct headless local Chrome launches to `chat.com` / `chatgpt.com` / `chat.openai.com` or `dash.cloudflare.com`, allow a narrow compatibility fallback that injects a normal Chrome `--user-agent` only when the caller did not explicitly provide one and did not choose raw Chrome arguments, headed, CDP, auto-connect, provider-backed, custom-UA, or non-Chrome behavior through argv or matching upstream environment. Wrapper-managed sessions retain that wrapper-owned user agent as per-session state across follow-up calls, failed replacement closes, and branch reload/resume. Active daemons omit both launch forms so upstream does not replace a launch-configured browser; a session proven inactive receives the retained compatibility launch values, including a fixed, comma-safe Chrome launch argument because the per-page override does not propagate to new tabs or SSO popups.
|
|
1081
849
|
|
|
850
|
+
## Shared execution coordination
|
|
851
|
+
|
|
852
|
+
Updated cooperating Pi processes acquire the same lock for the actual socket context, canonical namespace, and session. It spans helper checks, action, and state commit; a code cell holds it across all reads, branches, and actions. Independent identities can proceed concurrently. Fresh replacement or multi-session cleanup acquires the required identity set in sorted order; direct namespace-wide `close --all` takes the namespace barrier.
|
|
853
|
+
|
|
854
|
+
This is neither a browser transaction nor rollback. It does not coordinate humans, third-party/native CLI clients, older package versions, or distinct native identities attached to the same external Chrome target. Restart all participating Pi processes with the updated package before relying on the protocol. Separate tool calls still require workflow coordination. Inspect unknown mutation outcomes rather than replaying automatically.
|
|
855
|
+
|
|
856
|
+
## 0.7 migration
|
|
857
|
+
|
|
858
|
+
The 0.7 public contract removes old multimode `agent_browser` inputs. Native CLI coverage, authentication, recordings, artifacts, diagnostics, and specialized outcomes remain available.
|
|
859
|
+
|
|
860
|
+
| Previous 0.6 input | 0.7 replacement |
|
|
861
|
+
| --- | --- |
|
|
862
|
+
| `agent_browser { args, stdin?, outputPath?, timeoutMs?, sessionMode? }` | Same compact direct shape |
|
|
863
|
+
| `agent_browser { script: source }` | `agent_browser_code { code: source, session?, namespace?, timeoutMs?, outputPath? }`; check `.success`, not `.ok`; browser now persists |
|
|
864
|
+
| `agent_browser { job: { steps, failFast? } }` | Native `batch --bail` with argv rows; omit `--bail` only for deliberate safe continuation, or use code for branching |
|
|
865
|
+
| `agent_browser { semanticAction: fields }` | Enable `action`, then `agent_browser_action { ...fields }` |
|
|
866
|
+
| `agent_browser { qa: fields }` | Enable `qa`, then `agent_browser_qa { ...fields }` |
|
|
867
|
+
| `agent_browser { electron: fields }` | Enable `electron`, then `agent_browser_electron { ...fields }` |
|
|
868
|
+
| `agent_browser { sourceLookup: fields }` | Enable `source`, then `agent_browser_source { ...fields }` |
|
|
869
|
+
| `agent_browser { networkSourceLookup: fields }` | Enable `network`, then `agent_browser_network_source { ...fields }` |
|
|
870
|
+
|
|
871
|
+
Move former outer `outputPath`/`timeoutMs` alongside advanced flat fields where supported. Action has optional `session` but no `sessionMode`; QA/source/network retain `sessionMode`; Electron `timeoutMs` is flat and action-specific (`list` rejects it). Code chooses session/namespace in the parent.
|
|
872
|
+
|
|
873
|
+
| Former job step | Native argv rows |
|
|
874
|
+
| --- | --- |
|
|
875
|
+
| `open` plus `loadState` | `open <url>`, then `wait --load <state>` |
|
|
876
|
+
| Selector `click` / `fill` | Ordinary command with selector and optional text |
|
|
877
|
+
| Locator click/fill | `find <locator> <value> <action> ...` |
|
|
878
|
+
| `select` | `select <selector> <value...>` |
|
|
879
|
+
| `wait` | `wait <milliseconds>` |
|
|
880
|
+
| `assertText` / `assertUrl` | `wait --text <text>` / `wait --url <url-or-glob>` |
|
|
881
|
+
| `waitForDownload` | `wait --download <path>` |
|
|
882
|
+
| `snapshot` / `screenshot` | `snapshot -i` / `screenshot <path>` |
|
|
883
|
+
| Paced `type` | Optional `focus <selector>`, interleaved `keyboard type <character>` / `wait <delayMs>`, optional final `press <key>` |
|
|
884
|
+
|
|
885
|
+
For exact delayed-input behavior, build native rows instead of relying on unverified native `--delay` equivalence. Keep the former 200-character pacing bound when porting that workflow. Typing `hi` with a 20 ms gap:
|
|
886
|
+
|
|
887
|
+
```json
|
|
888
|
+
{ "args": ["batch", "--bail"], "stdin": "[[\"focus\",\"#prompt\"],[\"keyboard\",\"type\",\"h\"],[\"wait\",\"20\"],[\"keyboard\",\"type\",\"i\"]]" }
|
|
889
|
+
```
|
|
890
|
+
|
|
891
|
+
Fixed waits are elapsed time, not postconditions. Preserve explicit URL/text waits, same-snapshot fill ordering, fresh refs after mutations, and partial-progress inspection. There is no legacy public script/job runtime. Exact isolated-script cleanup leases are read only to retire pre-upgrade resources. Saved auth and Pi transcripts are not migrated or deleted; [rollback](RELEASE.md#07-upgrade-and-rollback) changes the package version, not browser side effects.
|
|
892
|
+
|
|
1082
893
|
## Non-goals
|
|
1083
894
|
|
|
1084
895
|
- no giant action enum mirroring the whole upstream CLI
|
|
1085
896
|
- no support for older `agent-browser` versions
|
|
1086
897
|
- no compatibility shims
|
|
1087
|
-
- no first-class reusable named browser recipe runtime above
|
|
898
|
+
- no first-class reusable named browser recipe runtime above code, advanced tools, or native batch; see [`ARCHITECTURE.md`](ARCHITECTURE.md#no-reusable-recipe-layer-yet) (closed `RQ-0068`)
|
|
1088
899
|
- no embedded browser UI inside `pi`
|