pi-agent-browser-native 0.6.6 → 0.6.7
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -1
- package/README.md +20 -5
- package/dist/extensions/agent-browser/index.js +138 -75
- package/dist/extensions/agent-browser/lib/command-taxonomy.js +6 -5
- package/dist/extensions/agent-browser/lib/electron/cleanup.js +10 -1
- package/dist/extensions/agent-browser/lib/input-modes/params.js +20 -7
- package/dist/extensions/agent-browser/lib/launch-scoped-flags.js +0 -1
- package/dist/extensions/agent-browser/lib/managed-session-restore.js +13 -12
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/click-dispatch.js +6 -25
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result.js +2 -3
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/index.js +10 -3
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/managed-session-daemon-policy.js +3 -0
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/wait-timeouts.js +1 -1
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare.js +104 -117
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output.js +61 -35
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state.js +100 -135
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/index.js +3 -1
- package/dist/extensions/agent-browser/lib/orchestration/input-plan.js +3 -1
- package/dist/extensions/agent-browser/lib/page-target-validation.js +10 -10
- package/dist/extensions/agent-browser/lib/parsing.js +7 -0
- package/dist/extensions/agent-browser/lib/playbook.js +5 -8
- package/dist/extensions/agent-browser/lib/process.js +23 -7
- package/dist/extensions/agent-browser/lib/recording-reservations.js +3 -1
- package/dist/extensions/agent-browser/lib/results/envelope.js +4 -1
- package/dist/extensions/agent-browser/lib/results/presentation/artifacts.js +45 -43
- package/dist/extensions/agent-browser/lib/results/recovery-actions.js +4 -4
- package/dist/extensions/agent-browser/lib/runtime.js +18 -2
- package/dist/extensions/agent-browser/lib/session-page-state.js +29 -10
- package/docs/ARCHITECTURE.md +7 -4
- package/docs/COMMAND_REFERENCE.md +22 -15
- package/docs/ELECTRON.md +6 -6
- package/docs/RELEASE.md +14 -5
- package/docs/REQUIREMENTS.md +1 -1
- package/docs/SUPPORT_MATRIX.md +13 -3
- package/docs/TOOL_CONTRACT.md +33 -22
- package/package.json +1 -1
package/docs/TOOL_CONTRACT.md
CHANGED
|
@@ -155,6 +155,7 @@ Upstream 0.35.2 adds `dashboard start --allowed-origins <origins>` and `AGENT_BR
|
|
|
155
155
|
## Headed and local fixture limits
|
|
156
156
|
|
|
157
157
|
- Upstream 0.35.0 and newer require separate `args` entries for global flag values. The wrapper rejects `--flag=value` global tokens before normal command dispatch, including trailing tokens; `--restore=<key>` is the explicit upstream-supported exception. Plain help/version inspection preserves exact caller argv, matching upstream. These exceptions are top-level only: global flags for `batch` belong before `batch`, and row-local equals forms fail validation before dispatch.
|
|
158
|
+
- A bare `--no-sandbox` in the command slot, or an option position after `open` / `goto` / `navigate`, fails validation before dispatch with Chromium launch-argument guidance. The former is an unknown upstream command; navigation ignores the latter. Pass the switch as a separate `--args` value and use `sessionMode: "fresh"` for launch changes. For native batches, `--args` belongs on the outer call before `batch`, not in a row. Other commands keep their native literal text, script, select, path and option-value handling; top-level help/version inspection remains unchanged.
|
|
158
159
|
- `--headed` is an upstream global flag passed through `args` (for example `{ "args": ["--headed", "open", "https://example.com"], "sessionMode": "fresh" }`). Use it on the first launch for demos, human-observed QA, or a user-completed login. If a managed browser session already exists, use `sessionMode: "fresh"` so the launch-scoped headed/headless choice is not ignored. Wrapper-owned headed launches default `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` to `0` because upstream 0.33.2's multi-origin restore collector opens visible temporary tabs and can delay session policy inspection. The wrapper records the effective launch-time interval and reapplies it to every helper and follow-up subprocess, still-owned off-current session, Electron cleanup close, and transcript reload/resume so daemon configuration remains stable. Native close still saves, but direct window close can lose newer state because upstream exempts headed browsers from idle shutdown; set an explicit interval before launch when periodic preservation matters. Because upstream reads it when the daemon starts, changing the recorded effective interval in either direction on a running wrapper-owned headed session is rejected until close plus a fresh launch.
|
|
159
160
|
- `--profile <name|path>` is upstream Chrome profile selection. `profiles` lists Chrome profile directory names from Chrome's user data directory; `Default` is common but not guaranteed. On profile/user-data-dir failures, use `details.nextActions` or run `profiles` / `doctor`, then tell the user which profile name/path to configure before retrying.
|
|
160
161
|
- `--executable-path <path>` selects a custom Chromium-compatible browser executable when upstream can launch it. Use it with `sessionMode: "fresh"` when switching from an already-active implicit session. For non-Chrome Chromium login state, use a full profile/user-data directory path only when upstream accepts it, or attach to a debug-enabled running browser with `--auto-connect` / `connect` when appropriate.
|
|
@@ -193,11 +194,11 @@ Upstream 0.35.2 adds `dashboard start --allowed-origins <origins>` and `AGENT_BR
|
|
|
193
194
|
- For Electron desktop apps, prefer top-level electron for wrapper-owned discovery, isolated launch, status, compact probe, and cleanup: list first, treat likely-sensitive annotations as hints rather than enforcement, launch with the default snapshot handoff unless handoff: "tabs" is the safer diagnostic starting point, use electron.probe or snapshot -i/qa.attached for current-session state, and always cleanup the returned launchId when done. electron.launch uses an isolated temporary profile; it does not reuse the app's normal signed-in profile or attach to an already-running authenticated app. For signed-in local app state, host-launch the normal app with --remote-debugging-port when appropriate, then use raw args connect <port|url>; after connect, run get url to verify the active target before page-content reads, inspect tab list, select the stable tab id such as tab t2, verify it again with get url, then run a condition wait or snapshot -i before using refs. close commands (`close`, `quit`, or `exit`) only close the browser/CDP session; leave manually launched app shutdown, profile cleanup, and explicit artifacts to the host owner.
|
|
194
195
|
- For provider or specialized app workflows, load version-matched upstream guidance with skills get agentcore|electron|slack|dogfood|vercel-sandbox|derive-client through the native tool; add --full when you need references/templates, and use skills get --all only for broad skill audits. Use derive-client when recording HAR traffic to generate a standalone API client; prefer network har start (text bodies by default) or network har start --content all|none before multi-step capture. For accessibility audits use a11y or a11y --tags wcag2a,wcag2aa (CDP browsers only). Hosted sandbox workflows should use upstream @agent-browser/sandbox helpers outside this wrapper. Provider launches such as -p ios, --provider browserbase/kernel/browseruse/browserless/agentcore, and iOS --device are upstream-owned setup paths; use sessionMode fresh when switching providers and expect external credentials or local Appium/Xcode setup to be required.
|
|
195
196
|
- For dialogs and frames, use dialog status/accept/dismiss and frame <selector|main> through native args; dialog commands and eval snippets that look like alert/confirm/prompt/dialog triggers are shorter-bounded than normal browser calls, and timed-out dialog-like interactions may add inspect-dialog-after-timeout, dismiss-dialog-after-timeout, or recover-fresh-session-after-dialog-timeout nextActions. When --confirm-actions produces a pending confirmation, use details.nextActions or exact confirm <id> / deny <id> calls instead of inventing ids.
|
|
196
|
-
- If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab <tab-id-or-label> / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load <state>, wait --download, or qa.attached; for disappearance checks, use wait --fn predicates instead of stale upstream-help examples like wait <selector> --state hidden. Use electron.probe/status for wrapper-owned launch health or target mismatch. Fixed waits are a last resort:
|
|
197
|
+
- If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab <tab-id-or-label> / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load <state>, wait --download, or qa.attached; for disappearance checks, use wait --fn predicates instead of stale upstream-help examples like wait <selector> --state hidden. Use electron.probe/status for wrapper-owned launch health or target mismatch. Fixed waits are a last resort: their duration is positional (wait <ms>, not wait --time <ms>). Use explicit --timeout or top-level timeoutMs for legitimately slow waits, and treat a successful payload like "waited":"timeout" as elapsed time only—verify completion with an observed condition, fresh snapshot, or screenshot.
|
|
197
198
|
- For feed, timeline, or inbox reading tasks, focus on the main timeline/list region and read the first item there rather than unrelated composer or sidebar content.
|
|
198
199
|
- For read-only browsing tasks, use read <url> for documentation or other unstructured text without requiring a Chrome page, or read with no URL for rendered active-tab DOM. Prefer the current snapshot, structured ref labels, getters, or scoped eval --stdin when you need interactive structure or targeted page state. Only click into media viewers, detail routes, or new pages when the current view does not contain the needed information.
|
|
199
200
|
- For downloads, prefer download <selector> <path> when an element click should save a file; simple loopback anchor downloads are saved to the requested path when the wrapper can resolve an HTTP(S) href. Do not rely on click alone when you need the downloaded file on disk.
|
|
200
|
-
- On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll <selector> <dir> [px|percent]. For native selects, use select <selector> <value...> (or semanticAction/job select) instead of clicking option refs; for custom comboboxes, a click/semanticAction may only focus the field, so re-snapshot and
|
|
201
|
+
- On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll <selector> <dir> [px|percent]. For native selects, use select <selector> <value...> (or semanticAction/job select) instead of clicking option refs; for custom comboboxes, a click/semanticAction may only focus the field, so re-snapshot and use keyboard type <text> for focused input, press ArrowDown or press Enter, or visible option refs. Raw type requires both <selector> and <text>.
|
|
201
202
|
- When using eval --stdin, scope checks and actions to the target element or route whenever possible instead of relying on broad page-wide text heuristics.
|
|
202
203
|
- When using eval --stdin for extraction, pass the JavaScript through the native tool stdin field, not as an extra args token after --stdin, and return the value you want instead of relying on console.log as the primary result channel. Prefer plain expressions like ({ title: document.title }) or explicitly invoked functions like (() => ({ title: document.title }))(); use outputPath when the eval/get/snapshot data should be saved as a durable local file, but never reuse a screenshot, download, recording, or other browser artifact destination as outputPath. If a function-shaped snippet returns {}, details.evalStdinHint may warn that the function was serialized instead of called. Local file pages and caller-selected output paths are supported when upstream allows them. If get text on a broad CSS selector surfaces details.selectorTextVisibility or selectorTextVisibilityAll, prefer a visible @ref, a more specific selector, or the inspect-visible-text-candidates nextAction over hidden tab content.
|
|
203
204
|
- When details.pageChangeSummary is present, use changeType and summary as a compact signal for navigation, DOM mutation, confirmations, or artifacts; when nextActionIds is set, match those ids to entries in details.nextActions (or per-step nextActions inside batch) for concrete follow-up payloads instead of inferring from prose alone. If details.clickDispatch reports a click-dispatch miss, refresh/inspect/retry the real click first; for static local fixtures only, an explicit eval --stdin programmatic .click() can exercise app handlers, but treat it as an untrusted scripted workaround and never use it to bypass stop-before-submit/order/purchase boundaries. If a no-navigation click surfaces details.overlayBlockers, inspect the fresh snapshot evidence before using a close/dismiss candidate nextAction; ordinary page chrome without dialog/alertdialog evidence should not trigger this diagnostic.
|
|
@@ -288,7 +289,7 @@ Example with a conditional branch and aggregation:
|
|
|
288
289
|
- optional `selector` is accepted for direct `click`, `check`, and `fill` targets (including current `@refs`); do not combine it with `locator`, `value`, `role`, or `name`. For `fill`, `text` is still required.
|
|
289
290
|
- `semanticAction` does not expose `uncheck` because upstream `find` actions are only `click, fill, check, hover, text`; use raw `args: ["uncheck", <selector-or-ref>]` after a stable selector or current snapshot ref
|
|
290
291
|
- for locator actions, `value` is the locator argument (for example ARIA role token `"button"`, label text, or visible substring), must be a non-empty string after trim; for `locator: "role"`, callers may provide `role` instead of redundant `value`
|
|
291
|
-
- `fill` requires non-empty `text` (compiled as the trailing value argument to `find`)
|
|
292
|
+
- `fill` requires non-empty `text` (compiled as the trailing value argument to `find`); `values` is only available for `select` in the tool schema, matching runtime validation
|
|
292
293
|
- `select` accepts either a non-empty direct `selector` plus `value` (one option value) or `values` (one or more option values), or an active-session semantic locator that the wrapper resolves through a fresh `snapshot -i` to exactly one current visible native dropdown ref. For `locator: "role"`, set `role` to `combobox` or `listbox`, set `name` to the accessible name, and provide option `value` / `values`. For `locator: "label"`, `value` is the accessible label text and `values` contains the option value(s). Locator-based select fails before action when no active browser exists or the current snapshot is missing/ambiguous; no fuzzy target is selected.
|
|
293
294
|
- optional `name` is only valid with `locator: "role"` and compiles to `--name <name>` after the action (and after `text` for `fill` when present)
|
|
294
295
|
- optional `role` is accepted only when `locator` is `role`; it may replace `value`, and must equal `value` if both are set
|
|
@@ -315,7 +316,7 @@ If a compiled `semanticAction` fails with `failureCategory: "selector-not-found"
|
|
|
315
316
|
|
|
316
317
|
If a compiled `semanticAction` `find` action fails with `failureCategory: "stale-ref"`, `details.nextActions` includes `retry-semantic-action-after-stale-ref` with the same redacted compiled argv as `details.compiledSemanticAction` in `params.args` (any leading `--session` pair from `semanticAction.session`, then the `find` tokens). The wrapper appends that entry **after** any `refresh-interactive-refs` snapshot step from `buildAgentBrowserNextActions` in `extensions/agent-browser/lib/results/action-recommendations.ts` (see `extensions/agent-browser/index.ts` where `nextActions` is merged). That retry is only offered because the semantic target is stable and the stale-ref error proves the previous action did not execute; `select` shorthands with stale `@e…` selectors and direct stale `@e…` commands still return refresh guidance instead of an unsafe blind retry.
|
|
317
318
|
|
|
318
|
-
For direct page-scoped `@
|
|
319
|
+
For direct page-scoped refs (`@eN`, `eN`, or `ref=eN`), successful `snapshot` results record `details.refSnapshot` with the latest ref ids and page target for the session. A failed session `snapshot` whose upstream error says `No active page` clears that session’s prior ref snapshot and records `details.refSnapshotInvalidation.reason: "no-active-page"`; any upstream-executed `record start` attempt (direct or inside a batch, including one that fails with `Recording already active`, because upstream swaps to the fresh recording page before that check) or `record restart` with a URL operand clears it and records `details.refSnapshotInvalidation.reason: "page-transition"`; a plain `record restart <path>` keeps the current page and refs; mutation-prone `@e…` preflight then fails with `failureCategory: "stale-ref"` until a later successful `snapshot -i` records fresh refs. Before page-scoped ref commands such as `get text`/`html`/`value`/`attr`/`box`/`styles`, `click`, `fill`, `check`, `select`, `download`, drag/upload actions, upstream ref-resolving reads and captures (`is`, `screenshot`, `highlight`, `scroll`, `frame`, `diff screenshot`), or equivalent batch steps run, the wrapper rejects refs from an older page target, refs absent from the latest same-page snapshot, or refs from an invalidated snapshot state. Batch steps are scanned from the source upstream actually executes: raw batch argument strings exclusively when any exist (upstream filters only the exact `--bail` token, so `--bail=true` stays a raw command), stdin steps only otherwise, so `batch "click @e1"` is guarded and stdin refs are not falsely rejected when upstream would ignore that stdin. Tab recovery verifies/selects the intended tab before semantic/ref helpers, then dispatches the caller's original argv/stdin. It does not force a continue-on-error batch to fail fast or turn literal operands into outer flags. A missing target or failed selection stops before any page-dependent user step. Local commands and explicit `connect` / `state load` recovery do not need the old tab, including the first effective batch row; later content still needs the normal page verification, and replacement does not supply fresh refs. Local success does not clear restored-target protection for the next page action. Same-tab checks preserve upstream refs and frame scope. Both pinned and unpinned failures retain `batchSteps`, `batchFailure`, and the visible failure roll-up. Artifact/screenshot preflights also skip upstream-ignored stdin rows. Getter batches receive the same same-page freshness check so a recycled `@ref` cannot silently read a different control after an in-place rerender. Only ref-resolving selector operands are guarded: ref-looking fill/type text, select values, file paths, attributes, and non-selector flag values remain literal. `get count` uses CSS/XPath, and `diff snapshot --selector` uses CSS; a bare `e999` remains a tag selector in those positions, not a ref. Commands whose operands upstream never resolves as refs (`wait`, `a11y`, `find`, `press`/`key`, `keyboard`, `mouse`) are not ref-guarded. Selector flags and positional selectors after `--new-tab` / `--full` remain guarded. A `batch` that times out or returns unparseable output after executing is treated conservatively: when its planned steps include a recording page swap, the wrapper still records the `page-transition` invalidation. This is a best-effort wrapper guard against upstream ref-number recycling after navigation; it does not prove the DOM stayed unchanged after the snapshot. Refresh with the session-aware `refresh-interactive-refs` next action before retrying.
|
|
319
320
|
|
|
320
321
|
Examples:
|
|
321
322
|
|
|
@@ -349,12 +350,14 @@ Examples:
|
|
|
349
350
|
- `type` with `text`; optional `selector` focuses the target first, optional `delayMs` emits per-character `keyboard type` plus `wait` rows, and optional `press` emits a final `press <key>` row such as `Enter`. Delayed typing is capped at 200 characters per step. Model-visible batch prose compacts generated per-character rows; `details.batchSteps` and `details.compiledJob.steps` retain the full bounded row list.
|
|
350
351
|
- `select` with `selector` plus either `value` or `values` (one or more option values; compiled as `select <selector> <value...>`)
|
|
351
352
|
- `wait` with positive integer `milliseconds`
|
|
352
|
-
- `assertText` with `text` (compiled as passive `wait --text <text>`)
|
|
353
|
+
- `assertText` with `text` only, without selector or locator fields (compiled as passive `wait --text <text>`)
|
|
353
354
|
- `assertUrl` with exact URL or `*` / `**` glob-style `url` pattern (compiled as `wait --url <url-or-pattern>`)
|
|
354
355
|
- `waitForDownload` with `path` (compiled as `wait --download <path>`)
|
|
355
356
|
- `snapshot` (compiled as `snapshot -i`; useful between mutation-prone steps before reusing current refs)
|
|
356
357
|
- `screenshot` with `path`
|
|
357
358
|
|
|
359
|
+
Clicks can stale subsequent `@refs`; split the job and take a fresh snapshot before using those refs.
|
|
360
|
+
|
|
358
361
|
**Navigation assertions are explicit only.** `job` never treats a successful `click` (or a `select` / submit-style interaction that may navigate) as proof that the expected next page loaded. Top-level `click` may still surface optional `details.navigationSummary` or `pageChangeSummary` hints for operators, but compiled `job` / `batch` steps do **not** auto-insert `assertUrl` or `assertText` after clicks—there is no deterministic expected URL source without caller intent. Use `open.loadState` to wait for initial page readiness after an `open`; after any later navigation-prone step (link/submit clicks, checkout or form flows, tab-sensitive UI), add an explicit `assertUrl` with the exact destination URL or a `*` / `**` glob-style pattern that does not already match the starting page, `assertText` for new on-page copy, or both, **before** screenshots or steps that assume the new page state. A broad URL pattern that already matches the current route is a precondition, not proof of navigation.
|
|
359
362
|
|
|
360
363
|
Example (static landing page):
|
|
@@ -411,6 +414,7 @@ Because `job` still executes as upstream `batch` with generated stdin, the same
|
|
|
411
414
|
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `sourceLookup`, `networkSourceLookup`, and `electron`
|
|
412
415
|
- lightweight preset built on the same batch compiler path as `job`, using `batch --bail` so missing readiness/text/selector assertions stop before slower diagnostics can burn the wrapper watchdog
|
|
413
416
|
- URL form: clears enabled network/console diagnostic buffers first and snapshots any page-error residue after `errors --clear` (upstream 0.33.2 does not reliably clear page errors), then opens `url`, waits with `wait --load <state>` using the resolved `loadState`, adds a bounded 150 ms diagnostic settle when console or page-error checks are enabled so immediate post-load callbacks can report, optionally asserts `expectedText` (string or string array, compiled to bounded visible-text `wait --fn … --timeout 5000` predicates after load) and/or `expectedSelector` (each may be omitted for a load-plus-diagnostics-only smoke), then runs enabled diagnostics: `network requests`, `console`, and `errors` only if preceding batch steps pass. Successful reset-step rows are labeled as reset output. Only unchanged page-error residue left after the clear is ignored; failed reset commands still fail the batch, and a matching error that reappears after a successful clear or any other post-open diagnostic row still counts normally.
|
|
417
|
+
- URL QA uses its explicitly requested navigation without requiring the previous tab; diagnostic clears remain before `open`. This does not reopen remembered pages for ordinary reads or `qa.attached`.
|
|
414
418
|
- attached form: `qa: { attached: true, expectedText?, expectedSelector?, screenshotPath?, checkNetwork?, checkConsole?, checkErrors?, loadState? }` runs the same waits (including the bounded diagnostic settle when needed), optional assertions, diagnostics, and screenshot against the current attached managed session without opening a URL. It rejects `url` and cannot be used with `sessionMode: "fresh"`; attach first with `electron.launch` or raw `args: ["connect", "<port-or-url>"]`, then run `qa.attached`. Before spawning the diagnostic batch, the wrapper preflights the attached session: `get url` must succeed and return a non-empty page URL. Missing URLs and read failures fail fast with `failureCategory: "validation-error"`, `details.validationError`, and recovery `nextActions` such as `list-tabs-before-qa-attached` and `snapshot-before-qa-attached` instead of running the full QA batch. `file:`, custom-scheme, and other attached targets are accepted. Attached QA does **not** run `network requests --clear`, `console --clear`, or `errors --clear`; `details.compiledQaPreset.checks.diagnosticsResetAtStart` is `false`. Visible text warns that existing diagnostic buffers were preserved only when `checkNetwork`, `checkConsole`, or `checkErrors` is enabled, and those diagnostics may include events from before the QA check.
|
|
415
419
|
- `loadState` is optional and must be `domcontentloaded`, `load`, or `networkidle`; it defaults to `domcontentloaded` so analytics-heavy or long-polling pages do not hang routine QA. Use `networkidle` only when the site is expected to go fully quiet.
|
|
416
420
|
- `checkNetwork`, `checkConsole`, and `checkErrors` default to `true` for URL-opening QA; for `qa.attached` they default to `false` because preserved upstream buffers may predate the current check. Set a field to `true` on `qa.attached` to opt into preserved-buffer diagnostics.
|
|
@@ -449,14 +453,16 @@ Action schemas:
|
|
|
449
453
|
|
|
450
454
|
Validation and defaults:
|
|
451
455
|
|
|
452
|
-
- `launch` requires exactly one target field. `list` accepts only `query` and `maxResults
|
|
456
|
+
- `launch` requires exactly one target field. `list` accepts only `query` and `maxResults`, with no configurable timeout (neither top-level `timeoutMs` nor nested `electron.timeoutMs`); `probe` accepts only `launchId` and `timeoutMs` beyond `action`; `status` / `cleanup` accept only `launchId`, `all`, and `timeoutMs`.
|
|
453
457
|
- Host install discovery (`electron.list` and resolving `appName` / `bundleId` through `discoverElectronApps` in `extensions/agent-browser/lib/electron/discovery.ts`) runs on **macOS** and **Linux** only. On **Windows** (and any other platform), `list` returns `platform: "unsupported"` with an empty `apps` array, and `launch` cannot resolve purely name-based targets without a prior scan—use `executablePath` or a host `appPath` that resolves to a verifiable Electron binary (`inspectElectronExecutablePath` still gates Windows executables).
|
|
454
458
|
- `launch.handoff` defaults to `"snapshot"`; supported values are `"connect"`, `"tabs"`, and `"snapshot"`. `"connect"` stops after attach, `"tabs"` verifies the live URL before a session-scoped `tab list` and is the safer diagnostic starting point when you do not want to capture refs/content yet, and `"snapshot"` verifies the live URL before `tab list` plus `snapshot -i` so current refs are immediately available. If the first Electron snapshot returns no refs while the app is still settling, the wrapper retries briefly before reporting no refs and tells agents to run `snapshot -i` once more before treating the UI as unusable.
|
|
455
459
|
- `launch.targetType` defaults to `"page"`; supported values are `"page"`, `"webview"`, and `"any"`. When a matching CDP target exposes a WebSocket URL, launch connects to that target; otherwise it falls back to the browser port.
|
|
456
460
|
- `appArgs` are passed to the Electron app, but wrapper-owned lifecycle/debug flags are rejected (`--user-data-dir`, `--remote-debugging-port`, `--remote-debugging-address`, `--remote-debugging-pipe`, and `--`).
|
|
457
461
|
- `allow` and `deny` are optional caller-owned policy lists. Entries match app name, bundle id, desktop id, app path, or executable path by substring. If `allow` is set, the target must match it; `deny` wins on conflict. With neither list, launch is permitted.
|
|
458
|
-
- `electron.status` / `electron.cleanup` accept optional `all` only as the boolean literal `true` to include every wrapper-tracked launch; `all` and `launchId` cannot both be set. Status and cleanup use the same runtime wrapper-tracked scope: current branch-visible records plus still-owned off-branch records. Default no-argument status/cleanup is intentionally ambiguous when more than one active launch is in that merged scope; pass `launchId` or `all: true`.
|
|
459
|
-
- `electron.launch
|
|
462
|
+
- `electron.status` / `electron.cleanup` accept optional `all` only as the boolean literal `true` to include every active wrapper-tracked launch (including dead, failed, or partial records, but excluding cleaned records); `all` and `launchId` cannot both be set. Status and cleanup use the same runtime wrapper-tracked scope: current branch-visible records plus still-owned off-branch records. Default no-argument status/cleanup is intentionally ambiguous when more than one active launch is in that merged scope; pass `launchId` or `all: true`.
|
|
463
|
+
- `electron.launch.timeoutMs` sets the host CDP readiness polling budget: **15000 ms** by default, capped at **120000 ms** (`normalizeTimeoutMs` in `extensions/agent-browser/lib/electron/launch.ts`). Its clock starts after target discovery and policy checks, before creating the isolated profile. Discovery has no configurable deadline; upstream attach and handoff use separate subprocess budgets.
|
|
464
|
+
- `status.timeoutMs` applies to each managed-session `get url` / `get title` subprocess for mismatch diagnostics. `probe.timeoutMs` applies to each upstream read (`get url`, `get title`, `eval --stdin`, `tab list`, `snapshot -i`). Their default wrapper budget is **35000 ms**, overridden by `PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` (`getAgentBrowserProcessTimeoutMs` in `extensions/agent-browser/lib/process.ts`). Localhost CDP HTTP probes use a fixed **1000 ms** each (`ELECTRON_CDP_FETCH_TIMEOUT_MS` in `extensions/agent-browser/lib/electron/cdp.ts`). Profile-path inspection has no configurable timeout.
|
|
465
|
+
- `cleanup.timeoutMs` is applied separately to the managed-session `close` subprocess and the initial host process-exit wait, not one combined deadline. It defaults to `PI_AGENT_BROWSER_IMPLICIT_SESSION_CLOSE_TIMEOUT_MS` or **5000 ms** (`getImplicitSessionCloseTimeoutMs` in `extensions/agent-browser/lib/runtime.ts`). Restored-PID verification and the later force-kill wait each have separate **1000 ms** limits; debug-port checks use the fixed CDP fetch budget, and profile removal has no configurable deadline.
|
|
460
466
|
- Non-Electron targets are rejected as a correctness failure; the wrapper does not blindly launch arbitrary executables as Electron.
|
|
461
467
|
|
|
462
468
|
Safety defaults and ownership:
|
|
@@ -514,15 +520,15 @@ Action-specific `details.electron` fields:
|
|
|
514
520
|
|
|
515
521
|
- `list`: `{ action: "list", status: "succeeded", apps, platform, query?, maxResults, skippedCount, omittedCount?, sensitiveAppCount?, profileIsolation }`. Each app is platform-tagged and may include `name`, `bundleId`, `desktopId`, `appPath`, `executablePath`, `icon`, `packageSource`, and non-blocking `sensitivity` metadata depending on platform/discovery source.
|
|
516
522
|
- `launch`: `{ action: "launch", status, launch, targets?, version?, handoff?, cleanup?, identifiers?, profileIsolation }`. `profileIsolation` states that wrapper launches use a new temporary profile, do not reuse existing signed-in app state, and do not attach to already-running authenticated apps; it also includes host debug-launch guidance for the separate normal-app attach path. `identifiers` repeats the launch-scoped `launchId` and attached `sessionName` so agents distinguish Electron lifecycle actions from browser session/tab actions. `launch.cleanupState` is one of `"active"`, `"cleaned"`, `"dead"`, `"failed"`, or `"partial"`. Failed launches expose `details.electron.failure.diagnostics` when available, including `pid` / `pidAlive`, wrapper `userDataDir`, elapsed/timeout timing, `DevToolsActivePort` file state, discovered port, and whether CDP `/json/version` was reached.
|
|
517
|
-
- `status`: `{ action: "status", status: "succeeded", launches, statuses, targets, identifiers?, identifierList?, managedSession?, managedSessions?, sessionMismatch?, sessionMismatches? }`, where each status includes the tracked `launchId`, port/pid liveness,
|
|
523
|
+
- `status`: `{ action: "status", status: "succeeded", launches, statuses, targets, identifiers?, identifierList?, managedSession?, managedSessions?, sessionMismatch?, sessionMismatches? }`, where each status includes the tracked `launchId`, `cleanupState`, independently measured port/pid liveness, bounded CDP targets, and fresh `userDataDirState`: `"present"`, `"absent"`, or `"unknown"`. Native `lstat` success means present (including dangling symlinks); only ENOENT means absent, and other filesystem errors mean unknown. This measures the tracked profile path, not all app residue, and is not stored in `ElectronLaunchRecord`. Explicit-ID status labels cleaned records as historical; default and `all: true` exclude them. Mismatch fields explain when the current managed session or tab does not match a live wrapper launch target.
|
|
518
524
|
- `cleanup`: `{ action: "cleanup", status: "succeeded" | "partial", cleanup: { partial, records, results } }`. Partial cleanup is a failed tool result with `failureCategory: "cleanup-failed"` and retry next actions. Cleanup steps may include `managed-session`, `process`, `debug-port`, and `user-data-dir`; managed-session close failures are reported while host-owned process/profile cleanup still runs.
|
|
519
|
-
- `probe`: `{ action: "probe", status: "succeeded" | "partial", probe, probeContext, identifiers?, sessionMismatch?, statusTargets?, launchStatus? }`. `probeContext` records whether the probe inspected the current managed session or a specific `launchId`. `probe` includes bounded `title`, `url`, `focusedElement`, `activeTab`, `tabs`, compact `snapshot` metadata (`refCount`, `refIds`, optional text preview and omission counts), `errors?`, and `summary`. When launch status is known, visible probe output includes debug-port/pid liveness so `about:blank` plus a dead wrapper launch is unmistakable. It also updates the normal session target/ref tracking when a snapshot is collected.
|
|
525
|
+
- `probe`: `{ action: "probe", status: "succeeded" | "partial", probe, probeContext, identifiers?, sessionMismatch?, statusTargets?, launchStatus? }`. `probeContext` records whether the probe inspected the current managed session or a specific `launchId`. `probe` includes bounded `title`, `url`, `focusedElement`, `activeTab`, `tabs`, compact `snapshot` metadata (`refCount`, `refIds`, optional text preview and omission counts), `errors?`, and `summary`. When launch status is known, `launchStatus.userDataDirState` carries the same fresh profile-path measurement as `status`, and visible probe output includes debug-port/pid liveness so `about:blank` plus a dead wrapper launch is unmistakable. It also updates the normal session target/ref tracking when a snapshot is collected.
|
|
520
526
|
|
|
521
527
|
Failure categories and next actions:
|
|
522
528
|
|
|
523
529
|
- `policy-blocked` is used when `electron.launch` is blocked by caller-supplied `allow` / `deny`; inspect `details.electron.failure.policy` for the matched list and entry when present.
|
|
524
530
|
- `cleanup-failed` is used when `electron.cleanup` only partially cleans tracked resources; inspect `details.electron.cleanup.results[].steps` for remaining process, port, or profile cleanup state.
|
|
525
|
-
- Launch timeout maps to `timeout`; non-Electron targets and input issues map to `validation-error`; launch/attach/spawn/CDP failures map to `upstream-error` unless a more specific category applies. If a successful-looking Electron mutation is followed by a dead process/debug port or an unrecoverable `about:blank` target, the wrapper upgrades the result to `failureCategory: "tab-drift"` and sets `details.electronPostCommandHealth` with the launch status and recovery action ids.
|
|
531
|
+
- Launch timeout maps to `timeout`; non-Electron targets and input issues map to `validation-error`; launch/attach/spawn/CDP failures map to `upstream-error` unless a more specific category applies. If a successful-looking Electron mutation is followed by a dead process/debug port or an unrecoverable `about:blank` target, the wrapper upgrades the result to `failureCategory: "tab-drift"` and sets `details.electronPostCommandHealth` with the launch status (including fresh `userDataDirState`) and recovery action ids.
|
|
526
532
|
- Successful Electron `fill <selector> <text>` commands may run a read-only `get value <selector>` verification, and successful fills against refs whose latest snapshot metadata proves a contenteditable target may run `get text <ref>` verification. If the value/text still differs, the wrapper keeps the tool successful but adds `details.fillVerification`, visible guidance to use snapshot/focus/keyboard typing for custom quick-input or rich editor controls, and `inspect-after-fill-verification` / `verify-filled-value` next actions.
|
|
527
533
|
- Successful active launches/status/probe results may include exact `details.nextActions` with ids `status-electron-launch`, `probe-electron-launch`, `cleanup-electron-launch`, `list-electron-tabs`, and `snapshot-electron-session`. Electron status/probe mismatch diagnostics may also include `reattach-electron-launch` before fresh tab/snapshot inspection. Electron post-command health failures include status/probe/cleanup actions for the same `launchId`; Electron `@e…` mutations may add `refresh-electron-refs-after-rerender` because desktop apps often rerender without URL changes. Electron cleanup partial failures may include `status-electron-launch` and `retry-electron-cleanup`.
|
|
528
534
|
|
|
@@ -653,7 +659,7 @@ Example:
|
|
|
653
659
|
### `timeoutMs`
|
|
654
660
|
|
|
655
661
|
- type: positive integer milliseconds
|
|
656
|
-
- optional per-call wrapper subprocess watchdog for the requested browser CLI process (`args`, `job`, `qa`, `sourceLookup`, and `networkSourceLookup`); Electron actions use nested `electron.timeoutMs`
|
|
662
|
+
- optional per-call wrapper subprocess watchdog for the requested browser CLI process (`args`, `job`, `qa`, `sourceLookup`, and `networkSourceLookup`); `electron.list` has no configurable timeout; other Electron actions use nested `electron.timeoutMs` for their action-specific budgets
|
|
657
663
|
- managed-session daemon-policy inspection has its own fixed budget of up to 35 seconds before that process and is intentionally not shortened by `timeoutMs`, so a busy valid daemon does not become an unsafe false negative
|
|
658
664
|
- use for long opens, large snapshots, paced `job` typing, or captures that legitimately need more than the default watchdog
|
|
659
665
|
- explicit long `wait` steps are forwarded to upstream; top-level `timeoutMs` only controls the wrapper subprocess watchdog and should be at least the wait duration plus a small grace window when supplied manually
|
|
@@ -674,6 +680,7 @@ Example:
|
|
|
674
680
|
Behavior:
|
|
675
681
|
- if `args` already include `--session` (including argv compiled from optional `semanticAction.session`), upstream session choice wins
|
|
676
682
|
- `"auto"` prepends the current extension-managed active session when appropriate
|
|
683
|
+
- after resume, a confirmed inactive wrapper-owned daemon with automatic managed restore enabled reopens its complete recorded URL, including its fragment, before the first current-page operation, including `get url` and `reload`. Non-page calls such as `tab list` or explicit HTTP `read <url>` can start a daemon without consuming that pending reopen, even across branch/reload replay. The wrapper verifies the observed tab and invalidates old refs; native `open` resets frame scope. This reloads the page with restored cookies/storage, not unsaved forms, JavaScript memory, or history. Live wrong-tab recovery does not navigate. Explicit URL reads, URL `a11y`/`vitals`, `diff url`, `window new`, URL-bearing recording commands, and explicit navigation/context changes do not require the old tab. Caller-owned/attached and restore-disabled sessions are not auto-reopened
|
|
677
684
|
- `"fresh"` rotates that managed session to a fresh upstream launch so startup-scoped flags like `--profile`, `--executable-path`, `--ca-cert`, `--no-ca-cert`, `--webgpu`, `--no-webmcp`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--headed`, `--enable`, `-p` / `--provider`, or iOS `--device` apply and later default calls follow the new browser; `--idle-timeout` must equal the Pi process's configured managed idle timeout or the wrapper rejects it with restart guidance
|
|
678
685
|
- upstream `--webgpu` is a launch-scoped optional boolean: both enabled and explicit `false` values require a fresh managed launch once an implicit session exists; enabled WebGPU is local-launch-only and upstream rejects combinations with CDP, auto-connect, or providers
|
|
679
686
|
- upstream `--no-webmcp` is also launch-scoped for bare/`true` and explicit `false` values because it selects whether locally managed Chrome enables the experimental feature
|
|
@@ -694,7 +701,7 @@ The extension should:
|
|
|
694
701
|
- invoke `agent-browser` directly, not through a shell
|
|
695
702
|
- parse JSON output into tool details
|
|
696
703
|
- handle observed JSON result shapes, including the array returned by `batch --json`
|
|
697
|
-
- allow plain-text
|
|
704
|
+
- allow plain-text output for native inspection calls and valid sessionless `upgrade` commands; all other commands retain JSON envelope validation
|
|
698
705
|
- support those inspection calls unconditionally so the tool contract stays local and predictable
|
|
699
706
|
|
|
700
707
|
<!-- agent-browser-playbook:start inspection -->
|
|
@@ -757,7 +764,7 @@ Recommended details:
|
|
|
757
764
|
Stable category fields are part of the machine-readable contract:
|
|
758
765
|
|
|
759
766
|
- `resultCategory`: always either `"success"` or `"failure"`.
|
|
760
|
-
- `successCategory`: present on successful results. Current values are `"completed"`, `"artifact-pending"`, `"artifact-saved"`, `"artifact-unverified"`, and `"inspection"`. `artifact-pending` means a recording started but its file is not expected until `record stop`; use the exact `stop-pending-recording` next action and verify the resulting file.
|
|
767
|
+
- `successCategory`: present on successful results. Current values are `"completed"`, `"artifact-pending"`, `"artifact-saved"`, `"artifact-unverified"`, and `"inspection"`. `artifact-pending` means a recording started but its file is not expected until `record stop`; use the exact `stop-pending-recording` next action and verify the resulting file. Dispatched `record start` and URL-bearing `record restart` attempts append one proactive `Page state:` warning, even on failure, advising a fresh snapshot; caller-requested `--json` carries it in `warnings`. Batch warnings require a reached result row. Preflight failures, missing binaries, help, plain restarts and unconfirmed planned rows do not claim a recording page change; the wrapper also invalidates the session’s prior ref snapshot (`refSnapshotInvalidation.reason: "page-transition"`) so old `@e…` refs fail as `stale-ref` until a fresh `snapshot -i`; that invalidation is attempt-scoped (a start that fails as `Recording already active` still swapped the page) and also covers `record restart` with a URL operand, while a plain `record restart` keeps the page and refs. Failed results also retain that action whenever their artifact rollup still contains a pending recording, except when the live daemon policy permits only cleanup: those results offer `close-pending-recording` instead, explicitly abandoning the unverified recording. `artifact-unverified` means upstream reported success but the merged `artifactVerification` summary still has unverified non-missing rows; inspect its counts and per-entry `state` / optional `limitation` before treating artifacts as durable evidence.
|
|
761
768
|
- `failureCategory`: present on failed results. Current values are `"aborted"`, `"artifact-missing"`, `"cleanup-failed"`, `"confirmation-required"`, `"download-not-verified"`, `"missing-binary"`, `"parse-failure"`, `"policy-blocked"`, `"qa-failure"`, `"script-error"`, `"selector-not-found"`, `"selector-unsupported"`, `"stale-ref"`, `"tab-drift"`, `"tab-gone"`, `"timeout"`, `"upstream-error"`, and `"validation-error"`. `artifact-missing` means upstream reported a saved/completed artifact path, but the wrapper verified the non-pending file is absent and failed closed.
|
|
762
769
|
|
|
763
770
|
For `script`, the top-level category describes the whole orchestration and cleanup, not the last inner call. `details.scriptRun` reports `callCount`, `successfulCallCount`, `failedCallCount`, `preDispatchRejectedCallCount`, `emitCount`, and timeout/abort flags when applicable. `script-error` means the caller's script source threw or rejected; browser subprocess and wrapper protocol failures remain `upstream-error` unless a more specific category applies. `details.scriptSteps[]` preserves bounded redacted per-call category/summary rows rather than replaying full inner tool results. `details.scriptSession` reports the exact isolated-session cleanup lease state only after the first accepted inner call; successful no-browser scripts omit it. A cleanup failure always wins over an otherwise successful or failed script so the leaked browser identity is not hidden.
|
|
@@ -776,6 +783,7 @@ Top-level `details.data` on `batch` is a compact per-step roll-up (not a verbati
|
|
|
776
783
|
|
|
777
784
|
Ref preflight details (command taxonomy in `extensions/agent-browser/lib/command-taxonomy.ts`, orchestration in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`):
|
|
778
785
|
|
|
786
|
+
- **Spelling and operands:** `@eN`, `eN`, and `ref=eN` share the same stale-ref checks, including inside raw and stdin batches. Only upstream ref-resolving selector slots are considered; `get count` and `diff snapshot --selector` keep their CSS/XPath or CSS semantics, while literal text and key/mouse data are not refs.
|
|
779
787
|
- **URL alignment:** `refSnapshot.target.url` and the session’s current tab URL are compared via `targetsMatch` / `normalizeComparableUrl` in `extensions/agent-browser/index.ts`: values are trimmed, parsed as URLs when possible, compared **after dropping the `#fragment`**, and the query string remains significant. If either side lacks a `url`, `targetsMatch` treats the pair as matching so early-session calls are not blocked.
|
|
780
788
|
- **Batch stdin ordering:** user `batch` JSON is scanned in order. Any step whose first token satisfies `isRefInvalidatingBatchCommand` sets a latch that blocks later steps whose first token satisfies `isRefGuardedCommand` and that mention `@e…` refs, except for same-snapshot native form-control steps whose current snapshot role metadata identifies all refs as safe controls (`check`/`uncheck` or direct `click`/`tap` on checkbox or radio refs, and `select` on combobox refs). A step whose first token is `snapshot` clears that latch for subsequent steps (pre-spawn intent only; it does not wait for upstream success). These predicates read explicit command capability flags from `command-taxonomy.ts`: navigation/mutation verbs such as `open` / `goto`, `reload`, non-form `click`, and related upstream commands have `invalidatesBatchRefs`, and `record start` steps (any outcome), `record restart` steps with a URL operand, plus WebMCP `invoke` / `result` / `cancel` steps also set the latch because upstream swaps or navigates the active page; same-snapshot `fill` rows and the role-checked native form-control rows stay guarded against missing/stale refs but do not set the latch, allowing ordinary form batches before a final click/submit step. Direct `click`/`tap @e…` is only treated as a safe form-control row when every ref in that step is a latest-snapshot checkbox or radio; other click/tap refs remain invalidating. Ref-guarded commands accept page-scoped refs for interaction (`click`, `fill`, `download`, `scrollintoview` / `scrollinto`, and others centralized in the command taxonomy). Changing either capability requires updating this contract, [`docs/SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) `RQ-0072`/`RQ-0087` notes, README and command-reference pitfalls, and `test/agent-browser.extension-validation.test.ts`.
|
|
781
789
|
|
|
@@ -791,7 +799,7 @@ For `batch`, each `batchSteps[]` entry can carry its own `nextActions` for that
|
|
|
791
799
|
|
|
792
800
|
`pageChangeSummary` is an optional compact summary for mutation-prone and artifact-producing commands. It includes `changeType` (`"navigation"`, `"mutation"`, `"artifact"`, or `"confirmation"`), `observed`, `command`, a readable `summary`, optional `title`/`url`, optional `artifactCount` or `savedFilePath`, and `nextActionIds`. `observed: false` means upstream dispatched a mutation-capable action but the wrapper did not observe an application change; standalone results also append a visible `Action dispatched; application change unverified` warning. The wrapper maintains explicit command/subcommand capability checks through `isPageChangeSummaryCommand` in `extensions/agent-browser/lib/command-taxonomy.ts`: those commands still emit a `mutation`-typed summary when upstream JSON lacks navigation metadata, as long as no stronger signal (artifact, saved path, navigation fields, or pending confirmation) applies. That capability is independent from `invalidatesBatchRefs` and `triggersPostMutationSnapshot`, so artifact summaries like `download` / `screenshot` and guarded-but-non-invalidating `fill` are documented directly in the capability table instead of implied by broad set spreading. Commands outside that set omit `pageChangeSummary` unless the parsed payload shows navigation, a confirmation prompt, saved files, or artifacts—including read-only inspection commands, which normally have no summary unless one of those signals appears. For `batch`, the top-level summary favors artifact rollups when any step produced artifacts; otherwise it synthesizes an observed-or-unverified summary from step evidence. Visible batch output promotes dispatch-only mutation evidence before step details and states that fixed waits are not postconditions. Agents should verify URL/text/state or an external receipt for important mutations before continuing.
|
|
793
801
|
|
|
794
|
-
`clickDispatch` may appear after a **top-level non-Electron** direct `click` when the wrapper installed a target-specific DOM-event probe, upstream reported success, and the post-click probe found no trusted DOM event reached the resolved target. Target-specific probes cover `xpath=` targets and role-gated `@e…` refs when the latest wrapper-tracked snapshot has role/name metadata; eligible ref roles are `button`, `checkbox`, `menuitem`, `radio`, `switch`, and `tab`, and
|
|
802
|
+
`clickDispatch` may appear after a **top-level non-Electron** direct `click` when the wrapper installed a target-specific DOM-event probe, upstream reported success, and the post-click probe found no trusted DOM event reached the resolved target. Target-specific probes cover `xpath=` targets and role-gated `@e…` refs when the latest wrapper-tracked snapshot has role/name metadata; eligible ref roles are `button`, `checkbox`, `menuitem`, `radio`, `switch`, and `tab`, and the role/name must be unique in both the saved snapshot and the live candidates. Duplicate-name refs pass through without a probe because snapshot order can change and is not proof of target identity. Raw `find … click` locator calls, including compiled `semanticAction` clicks that still execute as upstream `find`, are not probed because the wrapper has no concrete element before upstream resolves the locator, and document-level probes can falsely fail frame-scoped clicks. It does **not** take a fresh pre-click snapshot because that could recycle upstream refs before the intended click. The wrapper does **not** replay clicks in-page. On a miss it marks the tool failed, appends `Click dispatch diagnostic: …`, and sets `clickDispatch.status` to `"no-native-event-observed"` with `reason: "native-click-produced-no-target-dom-event"`, `nativeEventCount`, and a redacted `target` descriptor (`kind: "xpath"` plus `selector`, or `kind: "accessible"` plus `refId`, `role`, and redacted `name`). `details.nextActions` gains `inspect-click-dispatch-miss` (`snapshot -i`) and `retry-click-after-dispatch-miss` (same upstream click argv, session-prefixed when applicable). If a local static fixture must be exercised despite this diagnostic, a caller may explicitly run a programmatic activation via `eval --stdin` such as `document.querySelector(...).click()`, but that emits an untrusted scripted event and is only a debugging/workaround path; it must not be used as proof that real user-like clicking works or to bypass prompt stop boundaries. This diagnostic is only for standalone top-level direct click calls; `find` locator clicks and `batch`/`job`/`qa` click steps remain upstream-owned behavior.
|
|
795
803
|
|
|
796
804
|
`promptGuard` may appear on wrapper-blocked calls only for concrete, machine-checkable prompt requirements. `reason: "requested-artifacts-missing-before-close"` blocks `close` / `quit` / `exit` when the prompt used a direct screenshot/recording creation phrase with a destination such as `here`, `at`, `as`, or `to` and the session artifact manifest has not verified that exact path; a destination heading can carry that intent across contiguous plain or Markdown-bulleted path-only list lines. Bare, review-only, fenced-reference, conditional, permissive/uncertain, directly negated, and Pi clipboard/attachment image/video paths are treated as input, not output requirements, while subordinate requirements such as “do not close until you save” remain output intent. A recording-availability qualifier may precede, appear within, or follow its path/list and is scoped through the next path boundary; if the same path appears more than once, any required occurrence takes precedence. Markdown-link destinations resolve to the destination path. Explicitly optional artifacts are not close requirements. The classifier is deliberately conservative, so use wording such as `Save a screenshot here: <path>` when machine-enforced close blocking matters. Optional recording paths are only required when recording appears available. The wrapper does **not** parse broad user/business intent such as “do not place the order” or “do not post anything” into click/key blocks; agents must follow those instructions themselves. Prompt guards return `failureCategory: "policy-blocked"` and `validationError` text instead of invoking upstream.
|
|
797
805
|
|
|
@@ -853,11 +861,12 @@ Implementation and precedence:
|
|
|
853
861
|
- The main tool implementation merges these fields into Pi-facing `details` from `extensions/agent-browser/index.ts` and from `extensions/agent-browser/lib/results/presentation.ts` for presentation-time failures.
|
|
854
862
|
|
|
855
863
|
Additional structured fields can appear when relevant:
|
|
864
|
+
- `sessionTabReopenPending: boolean` persists a confirmed-cold managed session's outstanding URL reopen through non-page commands and branch/reload replay. `true` means a daemon may have started but the remembered page has not been reopened; `false` means a reopen attempt or an executed explicit context/navigation command consumed the obligation, not that navigation succeeded. Cancellation after the reopen CLI starts returns `failureCategory: "aborted"`, the exact `sessionName` / `namespace`, the consumed `false` marker and ref invalidation through the normal result path, so replay cannot repeat the navigation. Cancellation before the CLI starts leaves the obligation pending. Successful close clears it with the rest of that session's page state. Internal remembered URLs retain their complete fragment; comparison remains fragment-insensitive and presentation redaction is unchanged. Old persisted targets without a fragment cannot reconstruct it.
|
|
856
865
|
- `closeAllApplied: true` when a successful direct or nested `close` / `quit` / `exit --all` reached upstream. The marker makes live state and transcript replay clear every managed/attached/page/ref/route/trace/recording identity in the effective canonical namespace; a later batch row that proves browser reactivation may rebuild only the effective session.
|
|
857
866
|
- `attachedBrowserSession: true` on successful calls that establish or reuse a wrapper-tracked CDP/auto-connect/Electron attachment, and on a failed fresh attachment only when `managedSessionOutcome.activeAfter` proves its daemon remained active for cleanup. The marker restores attachment continuity from the active transcript branch, including that active-after-failure case; live state and transcript replay remove it after a terminal successful close/cleanup even when aggregate verification failed; a close followed by a later step whose lifecycle reports a browser launch preserves it, while a successful non-launching diagnostic leaves the close terminal. Caller config, environment, paths, and file-access settings remain upstream-owned; the marker only adds live-URL verification and lifecycle continuity.
|
|
858
867
|
- `lifecycle: { effectiveLaunch: { browserLaunched } }` when upstream returned that boolean. It separates starting the requested `agent-browser` CLI process from the effective Chrome session context. `readSource` exposes upstream's string `data.source` for direct `read` calls and identifies the raw HTTP fetch path; its lifecycle boolean can be `false` before a browser launch or `true` when the same managed session already has an active browser. Direct reads also append one visible `Read execution` line with the source, CLI-start result, managed browser lifecycle, and managed-session outcome so Pi models do not have to infer model-invisible details.
|
|
859
868
|
- `browserWindow: { mode: "headed", ownership: "wrapper-managed", sessionName, visibility: "unverified" }` only after a successful first/fresh local wrapper-managed headed call (including `batch`) that is not an attachment and whose lifecycle proves a browser launched and whose managed-session outcome is `created` or `replaced`. One visible handoff sentence tells the user to complete the login in that window if they can see it, then continue with `sessionMode: "auto"`; the field never claims OS desktop visibility.
|
|
860
|
-
- `sessionTabTargetUnknown: true` after a spawned `connect`, `state load`, history navigation,
|
|
869
|
+
- `sessionTabTargetUnknown: true` after a spawned `connect`, `state load`, history navigation, tab-selection/close, `window new`, or `diff url` call changes the active page without a trustworthy observed target. Direct `window new` / `diff url` and reached native batch rows retire the old target and refs; an intentional new blank window or an observed blank URL-diff destination never triggers old-tab recovery. URL diff inputs do not prove the final URL after redirects. Successful standalone tab selection/close now live-probes URL and a fresh non-blank title before state is committed, even when the new tab shares the prior URL; explicit selection of an existing `about:blank` tab and a post-close blank target are retained, so this marker remains only when the probe cannot verify the target. It is persisted and restored across branch/reload replay, clears stale refs and tab pinning, and blocks page inspection until `get url` or explicit navigation observes a target; `tab list`, tab selection, close, and blocking-dialog `status` / `accept` / `dismiss` remain available. A timeout against an unknown target removes standalone snapshot actions and returns `verify-page-target-after-timeout`, a session-scoped `batch --bail` whose stdin runs `get url` before `snapshot -i`.
|
|
861
870
|
- `compiledSemanticAction` when the call used `semanticAction` and the result includes the unified `details` merge: `{ action, locator, args }` for `find` actions or `{ action: "select", selector?, locator?, values, args }` for `select`, with the same redaction rules as `args` / `effectiveArgs`; omitted for plain `args`/`job` calls and omitted on some early error returns that omit this field (see the `semanticAction` section above)
|
|
862
871
|
- `compiledJob` when the call used `job` or the job-backed `qa` preset: by default `{ args: ["batch", "--bail"], failFast: true, stdin, steps: [{ action, args }] }`; with `failFast: false`, `{ args: ["batch"], failFast: false, stdin, steps: [{ action, args }] }`. Step args are redacted the same way as other invocation details. Semantic `job` click/fill steps appear here as their compiled upstream `find … click|fill …` argv, not as the input object.
|
|
863
872
|
- `compiledQaPreset` when the call used `qa`: the compiled job fields plus the QA `checks` object. `args` is `batch --bail` and `failFast` is `true` for QA presets. `checks.attached` is `true` for current-session QA, `checks.url` is present only for URL-opening QA, and `checks.diagnosticsResetAtStart` is `true` only for URL-opening QA because `qa.attached` preserves existing session diagnostics.
|
|
@@ -869,13 +878,13 @@ Additional structured fields can appear when relevant:
|
|
|
869
878
|
- `compiledElectron` when the call used `electron`: redacted action plan for `list`, `launch`, `status`, `cleanup`, or `probe`.
|
|
870
879
|
- `electron` when the call used `electron`: action-specific lifecycle, discovery, probe, and cleanup data; see the `electron` section below.
|
|
871
880
|
- `batchFailure` and `batchSteps` for `batch` rendering, including mixed-success runs
|
|
872
|
-
- `navigationSummary` for navigation-style commands like `click`, `back`, `forward`, `reload`, and successful standalone tab selection/close; `urlChanged` records whether the live URL differs from a known pinned pre-command URL, so same-URL clicks and clicks without a comparison baseline remain dispatch-only rather than being mislabeled as observed navigation. Helper probes run `get url` first and run `get title` for any verified non-`about:blank` URL. The title read is skipped when the probed URL already carries a wrapper-observed title for this session, except after a tab selection/close, which always refreshes a non-blank title even when the URL is unchanged: titles are last-observed labels for that URL, while the URL itself is live-probed on every call. Href-less CSS selector clicks use this same post-command helper so `sessionTabTarget` cannot stay on the pre-click page; any click-dispatch check still runs first. A failed non-batch `eval`, `back`, `forward`, `reload`, `connect`, `state load`, or `tab` selection also runs this helper (browser started, not aborted, not watchdog-timed-out), so an observed page stays verified instead of forcing a manual `get url` round trip; a failed or empty probe keeps the prior unverified-page behavior. Because a failed transition can still have mutated or replaced the document, a successful probe also invalidates the prior page-scoped ref snapshot (matching the previous unknown-target behavior, which dropped refs), so the next `@e…` use requires a fresh `snapshot -i`.
|
|
881
|
+
- `navigationSummary` for navigation-style commands like `click`, `back`, `forward`, `reload`, `window new`, `diff url`, and successful standalone tab selection/close; reached window-new/URL-diff rows also probe the final page after native batches, including failed batches that may already have changed the page (not aborted or timed-out calls). Unreached rows do not change target or ref state; `urlChanged` records whether the live URL differs from a known pinned pre-command URL, so same-URL clicks and clicks without a comparison baseline remain dispatch-only rather than being mislabeled as observed navigation. Helper probes run `get url` first and run `get title` for any verified non-`about:blank` URL. The title read is skipped when the probed URL already carries a wrapper-observed title for this session, except after a tab selection/close, which always refreshes a non-blank title even when the URL is unchanged: titles are last-observed labels for that URL, while the URL itself is live-probed on every call. Href-less CSS selector clicks use this same post-command helper so `sessionTabTarget` cannot stay on the pre-click page; any click-dispatch check still runs first. A failed non-batch `eval`, `back`, `forward`, `reload`, `connect`, `state load`, or `tab` selection also runs this helper (browser started, not aborted, not watchdog-timed-out), so an observed page stays verified instead of forcing a manual `get url` round trip; a failed or empty probe keeps the prior unverified-page behavior. Because a failed transition can still have mutated or replaced the document, a successful probe also invalidates the prior page-scoped ref snapshot (matching the previous unknown-target behavior, which dropped refs), so the next `@e…` use requires a fresh `snapshot -i`.
|
|
873
882
|
- `pageChangeSummary` for compact mutation/artifact/navigation summaries on commands that can change browser state
|
|
874
883
|
- `clickDispatch` when a top-level non-Electron direct `click` reported upstream success but the target-specific probe found no trusted event reached the resolved XPath or accessible `@ref` target; shape follows `ClickDispatchDiagnostic` in `extensions/agent-browser/lib/orchestration/browser-run/types.ts`
|
|
875
884
|
- `promptGuard` when the requested-artifact-before-close guard blocks browser close before required prompt artifact paths are verified; implementation lives in `extensions/agent-browser/lib/orchestration/browser-run/prompt-guards.ts`
|
|
876
885
|
- `overlayBlockers` for conservative overlay/banner/dialog blocker candidates when a successful snapshot itself contains strong modal evidence, or after a qualifying top-level `@e…` / `ref=` click stays on the same URL, no `clickDispatch` diagnostic fired, and a fresh snapshot provides evidence (`candidates`, `summary`, and `snapshot` per `OverlayBlockerDiagnostic` in `extensions/agent-browser/index.ts`). CSS selector clicks do not run this overlay probe.
|
|
877
886
|
- `visibleRefFallback` after a raw `find` or compiled `semanticAction` fails with `selector-not-found` and a fresh snapshot finds exact role/name `@ref` matches. Shape follows `VisibleRefFallbackDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, snapshot, summary, target }`, where each candidate has `ref`, `role`, `name`, optional direct ref `args`, and `reason`; visible text appends `Current snapshot ref fallback`. Non-fill candidates with direct args add `try-current-visible-ref` or numbered `try-current-visible-ref-N` actions. Fill candidates omit direct args and target text so recovery details do not repeat potentially sensitive fill text.
|
|
878
|
-
- `refSnapshotInvalidation` after a session `snapshot` fails with `No active page`, any upstream-executed `record start` attempt swaps the session to a fresh active page (even one that then fails as `Recording already active`), a `record restart` with a URL operand navigates it, or a failed non-batch transition command (`eval`, `back`, `forward`, `reload`, `connect`, `state load`, `tab` selection) whose live URL re-verification probe observed the page (a failed transition can still have mutated the document, so the verified URL is kept but the prior refs are not). Shape follows `SessionRefSnapshotInvalidation` in `extensions/agent-browser/lib/session-page-state.ts`: `{ reason: "no-active-page" | "page-transition", summary }`; replay preserves the persisted summary. The wrapper deletes prior refs for that session, persists the invalidation for resume, and blocks mutation-prone `@e…` preflight with `failureCategory: "stale-ref"` until a successful fresh `snapshot -i` records refs again.
|
|
887
|
+
- `refSnapshotInvalidation` after a confirmed cold managed-session shutdown (`reason: "page-transition"`, including when reopening fails), a session `snapshot` fails with `No active page`, any upstream-executed `record start` attempt swaps the session to a fresh active page (even one that then fails as `Recording already active`), a `record restart` with a URL operand navigates it, a direct or reached batch `window new` / `diff url` attempt changes the page, or a failed non-batch transition command (`eval`, `back`, `forward`, `reload`, `connect`, `state load`, `tab` selection) whose live URL re-verification probe observed the page (a failed transition can still have mutated the document, so the verified URL is kept but the prior refs are not). Shape follows `SessionRefSnapshotInvalidation` in `extensions/agent-browser/lib/session-page-state.ts`: `{ reason: "no-active-page" | "page-transition", summary }`; replay preserves the persisted summary. The wrapper deletes prior refs for that session, persists the invalidation for resume, and blocks mutation-prone `@e…` preflight with `failureCategory: "stale-ref"` until a successful fresh `snapshot -i` records refs again.
|
|
879
888
|
- `snapshotFilter` after wrapper-side `snapshot -i --search <text>` or `snapshot -i --filter role=<role>`. Shape: `{ cleanArgs, search?, role?, matchedRefs, totalRefs, visibleLines, totalLines, renderedTextMatches?, renderedTextTotalMatches?, renderedTextTruncated? }`. Search runs one bounded read-only rendered-DOM probe across the full document; each visible match carries bounded `text`, `tagName`, `kind` (`text` or prioritized `validation`), `offscreen`, optional `role`/accessible `name`, and a unique mapped `ref` when the full snapshot supports it. Hidden elements are excluded. The filtered accessibility snapshot remains separate, while `details.refSnapshot` still records the full upstream ref map for later stale-ref checks.
|
|
880
889
|
- `snapshotViewport` after wrapper-side `snapshot --viewport` (with or without `-i`, `--search`, or `--filter`). Shape matches the scroll-position probe: viewport scroll offsets, inner/document dimensions, sampled scrollable-container count, and bounded container offsets. The wrapper strips `--viewport` before upstream spawn and gathers this with a read-only `eval --stdin` call.
|
|
881
890
|
- `snapshotDiff` after wrapper-side `snapshot --diff` (with or without `-i`, `--search`, `--filter`, or `--viewport`). Shape: `{ addedRefs, removedRefs, changedRefs, unchangedRefs, summary }`, comparing ref ids plus role/name metadata from the previous wrapper-tracked snapshot for the session with the newly returned full ref map. It is a quick ref-map delta, not a visual diff.
|
|
@@ -895,9 +904,9 @@ Additional structured fields can appear when relevant:
|
|
|
895
904
|
- `managedSessionHeadedAutosaveDisabled: true` is the narrower compatibility marker that the targeted session uses the wrapper's default interval `0`, rather than an explicit caller interval. It accompanies `managedSessionHeadedAutosaveInterval: "0"` on active rows and remains omitted for explicitly configured autosave.
|
|
896
905
|
- `managedSessionOutcome` after a managed-session plan reaches process execution (`buildManagedSessionOutcome` / `formatManagedSessionOutcomeText` in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`). Populated when `buildExecutionPlan` injects an extension-managed implicit or fresh `--session`, and also when a successful explicit `--session <current-wrapper-managed-session> close` closes the current managed session. It remains omitted for unrelated explicit user-managed sessions and for sessionless inspection/local paths that skip injection. Successful nested-batch lifecycle rows are evaluated in order: a terminal close reports and replays `status: "closed"` even when aggregate artifact verification makes the tool result fail; a later lifecycle-proven browser launch (including a post-close `record stop`) keeps the session active, an explicitly non-launching diagnostic leaves it closed, and an unknown row stays conservatively active even when the failed batch was the first managed call. Fields: `status` (`created`, `replaced`, `unchanged`, `closed`, `preserved`, or `abandoned`), `sessionMode`, `attemptedSessionName`, `previousSessionName`, `currentSessionName`, optional `currentSessionNamespace`, optional `replacedSessionName`, optional `replacedSessionNamespace`, optional `replacedSessionClosed` (false means automatic close failed and the previous session remains wrapper-owned/restorable for explicit cleanup), `activeBefore`, `activeAfter`, `succeeded`, and `summary` (machine-oriented; may include generated session names). Use `currentSessionNamespace` with `currentSessionName` when following preserved-session recovery actions; retry-fresh actions stay in the attempted namespace. Model-visible echo: when `sessionMode` is `"fresh"` **and** `succeeded` is false, or when `replacedSessionClosed` is false after a replacement, the wrapper appends action-oriented `Managed session outcome` and `Recovery` lines without repeating generated session ids in visible prose; session names remain in `details.managedSessionOutcome`. Failed fresh launches may also append `details.nextActions` such as `run-agent-browser-doctor`, `verify-current-managed-session`, `snapshot-current-managed-session`, or `retry-fresh-managed-session`. When other trailing diagnostic prose is also emitted in the same result, that block is concatenated **after** semantic-action candidate lines, overlay/selector-visibility tails, eval hints/warnings, and `Timeout partial progress` (see `rawAppendedDiagnosticText` in `extensions/agent-browser/lib/orchestration/browser-run/final-result.ts`). For `"auto"` failures the same struct may appear on `details` without that extra line. When post-upstream analysis (for example **`qa`** preset failure) flips the overall tool result after a successful batch, or a fresh `job`/batch opens the requested page and then a later step fails, the managed-session transition still reflects that the fresh browser became current. The visible recovery says the fresh launch became current and points to `failureCategory` / `qaPreset` / `batchFailure` for the post-launch failure instead of telling the agent that the old session was preserved.
|
|
897
906
|
- `imagePath` / `imagePaths` for Pi inline image attachments from the **`screenshot`** command (including batched screenshot steps). **`diff screenshot`** still records the diff output as an `image`-kind entry in `details.artifacts`, but it does **not** populate `imagePath` / `imagePaths` or attach an inline image: only plain `screenshot` is treated as a trusted live-capture path for automatic inlining (`isTrustedScreenshotOutput` in `extensions/agent-browser/lib/results/presentation/artifacts.ts`).
|
|
898
|
-
- `artifacts` for saved files such as screenshots, `state save` outputs, `diff screenshot` diff images, PDFs, downloads, `wait --download` / `wait -d` files, traces, CPU profiles, completed WebM recordings, path-bearing HAR captures, and future recording output paths reported by `record start` / `record restart`. Non-file URL payloads such as `data:` / `blob:` / `http(s):` values are not treated as verified local artifacts. For direct artifact commands and batch artifact steps, the wrapper creates parent directories for requested paths before spawning upstream. Each artifact includes the original saved or requested `path`, resolved `absolutePath`, `kind`/`artifactType`, optional `mediaType`, optional `extension`, best-effort disk metadata such as `exists`, `sizeBytes`, and `updatedAtMs`, plus `requestedPath`, `status`, `cwd`, `session`, `namespace`, and `tempPath` when applicable. For commands that create/update artifacts, a path that existed but was not updated during this command uses `status: "stale"`; observational `wait --download` may accept a file completed just before the wait began. Pending `record start` / `record restart` artifacts use `status: "pending"`, omit `exists` rather than reporting false, and include `recordingState: "openRecording"` / `willExistOnStop: true`. Within one Pi extension process, the wrapper keeps an unbounded transcript-backed active-recording reservation index separate from the bounded artifact manifest, keyed by canonical namespace plus session; still-live process-owned reservations survive branch switches, while
|
|
907
|
+
- `artifacts` for saved files such as screenshots, `state save` outputs, `diff screenshot` diff images, PDFs, downloads, `wait --download` / `wait -d` files, traces, CPU profiles, completed WebM recordings, path-bearing HAR captures, and future recording output paths reported by `record start` / `record restart`. Non-file URL payloads such as `data:` / `blob:` / `http(s):` values are not treated as verified local artifacts. For direct artifact commands and batch artifact steps, the wrapper creates parent directories for requested paths before spawning upstream. Filesystem `mkdir` failures at this shared preparation boundary return `validation-error`, `agentBrowserStarted: false`, the attempted directory and `verify-artifact-path` guidance. Raw batch strings are never rewritten; use absolute artifact paths because the daemon's cwd may differ from Pi's. Each artifact includes the original saved or requested `path`, resolved `absolutePath`, `kind`/`artifactType`, optional `mediaType`, optional `extension`, best-effort disk metadata such as `exists`, `sizeBytes`, and `updatedAtMs`, plus `requestedPath`, `status`, `cwd`, `session`, `namespace`, and `tempPath` when applicable. `requestedPath` is retained only when known from the caller, separately from reported/resolved locations; a differing screenshot report remains in `tempPath` and is displayed as `Reported path`, whether it is a temporary file or a canonical path alias. Ordinary file `mediaType` values come from bounded PNG/JPEG/GIF/WebP header recognition, not suffixes; unknown, missing, unreadable or truncated headers leave it undefined. Header recognition is not full-file format validation. Inline screenshot attachments use the same byte classifier and existing size limit. Direct-anchor downloads retain their response Content-Type metadata. For commands that create/update artifacts, a path that existed but was not updated during this command uses `status: "stale"`; observational `wait --download` may accept a file completed just before the wait began. Pending `record start` / `record restart` artifacts use `status: "pending"`, omit `exists` rather than reporting false, and include `recordingState: "openRecording"` / `willExistOnStop: true`. Within one Pi extension process, the wrapper keeps an unbounded transcript-backed active-recording reservation index separate from the bounded artifact manifest, keyed by canonical namespace plus session; still-live process-owned reservations survive branch switches, while known closures are appended after tree navigation and during shutdown/reload so a close on one branch cannot be resurrected after returning to an older branch. Persisted active reservations require absolute storage paths and cwd; their display paths may remain relative. If a journal append fails, the next serialized browser boundary, tree navigation, or shutdown retries all current reservations and known closures. `recordingPersistenceWarning` and visible warning text remain present while restart protection is not durable; successful recovery is quiet and cleanup still runs. Artifact lifecycle calls, explicit `wait --download <path>` / `wait -d <path>` destinations, and result `outputPath` writes serialize around the global destination check/update, every successful direct, ordered nested-batch, fresh-replacement, script, Electron, or shutdown close retires only its exact identity at that lifecycle point, and destination reuse is rejected through lexical, existing or dangling symlink, hardlink, full Unicode-fold, or macOS/Windows case aliases. Batch preflight rejects `record start` / `record restart` after a close row because upstream can report a recording that did not start; split those operations into separate calls. A definitive `No recording in progress` stop failure, direct or nested in a batch, retires stale reservation state at that ordered step instead of recommending the same stop again; a later successful batch recording row opens its new pending path normally. Batch preflight applies the same distinct-destination rule to the steps upstream will execute: raw argument command strings exclusively when any exist, stdin arrays only otherwise; upstream-ignored stdin rows cannot fail artifact preflight, add pending recordings, or create parent directories. Parent directories are prepared for the effective steps in both modes; raw argument strings are never rewritten, so the screenshot absolute-path normalization and tracked path request apply to stdin rows only.
|
|
899
908
|
- `savedFilePath` / `savedFile` for direct `download`, `pdf`, and `wait --download` / `wait -d` saved-file workflows when a host file path is reported or wrapper-verified. Batch results preserve the same fields on the relevant `batchSteps` entry. These fields are metadata only until `artifactVerification` verifies the file. For simple loopback `download <selector> <path>` anchors with a non-ref selector, `details.downloadRecovery.method: "direct-anchor-fetch"` means the wrapper resolved the anchor URL in-session and saved the in-page HTTP(S) response directly to the requested path before using upstream's click/download fallback; non-loopback/profile downloads stay upstream-owned so external provider behavior is preserved.
|
|
900
|
-
- `batchSteps[].artifacts` for per-step artifacts in `batch` output; top-level `artifacts` and `artifactManifest` coalesce an earlier pending recording into the later saved, missing, or stale terminal result for the same namespace/session identity. `record restart` includes both the previous recording it finalized (or an explicit missing/stale failure) and the new pending recording; missing/stale terminal rows retire the prior pending manifest row. A successful later `close` / `quit` / `exit` represents an earlier unfinalized pending recording as `status: "missing"` / `subcommand: "close-abandoned"`, clears its stop action, and updates aggregate verification/manifest state consistently; a later successful `record stop` replaces that intermediate abandoned row with its saved artifact. Close also resets ref/page/network-route state produced by earlier rows; later lifecycle-proven browser launches, including `record stop`, can rebuild that state without triggering stale pre-close `about:blank` recovery, explicitly non-launching diagnostics cannot, and unknown later rows stay conservatively active. Per-step history remains unchanged. When any later call on the same namespace/session fails while a recording remains pending, `nextActions` combines its normal recovery with exact `stop-pending-recording` args and visible cleanup guidance; the same applies at top level when a later batch step fails.
|
|
909
|
+
- `batchSteps[].artifacts` for per-step artifacts in `batch` output; top-level `artifacts` and `artifactManifest` coalesce an earlier pending recording into the later saved, missing, or stale terminal result for the same namespace/session identity. `record restart` includes both the previous recording it finalized (or an explicit missing/stale failure) and the new pending recording; missing/stale terminal rows retire the prior pending manifest row. A successful later `close` / `quit` / `exit` represents an earlier unfinalized pending recording as `status: "missing"` / `subcommand: "close-abandoned"`, clears its stop action, and updates aggregate verification/manifest state consistently; a later successful `record stop` replaces that intermediate abandoned row with its saved artifact. Close also resets ref/page/network-route state produced by earlier rows; later lifecycle-proven browser launches, including `record stop`, can rebuild that state without triggering stale pre-close `about:blank` recovery, explicitly non-launching diagnostics cannot, and unknown later rows stay conservatively active. Per-step history remains unchanged. When any later call on the same namespace/session fails while a recording remains pending, `nextActions` combines its normal recovery with exact `stop-pending-recording` args and visible cleanup guidance; the same applies at top level when a later batch step fails. After reload in a non-Git checkout or with managed restore disabled, a live daemon without current-instance provenance cannot accept a stop. That policy refusal includes `managedSessionCleanupOnlyReason: "restore-disabled-daemon-without-provenance"` plus the exact `sessionName`/`namespace`, including on implicit calls. It replaces the impossible stop with `close-pending-recording`, an exact close without `sessionMode: "fresh"`. Close retires the recording as `close-abandoned`; any file it leaves is unverified. Same-instance recordings and supported durable-Git reloads still use stop and normal WebM verification.
|
|
901
910
|
- `artifactVerification` for a normalized verification summary on the unified result and on each successful `batchSteps[]` row (failed batch steps omit artifact rows). Top-level `batch` verification rolls up all step file artifacts; each step’s summary reflects that step’s nested tool presentation (including its spill paths and manifest slice). It reports `verified`, `verifiedCount`, `missingCount`, `pendingCount`, `unverifiedCount`, and `artifacts[]` entries with `path`, optional `absolutePath`, optional `requestedPath`, `kind` (a normal file artifact kind or `"spill"` for manifest-backed rows), optional `mediaType`, optional `exists`, optional `sizeBytes`, optional `updatedAtMs`, optional `status`, optional `retentionState` / `storageScope` on manifest-derived rows, `state` (`verified`, `missing`, `pending`, or `unverified`), and optional `limitation` (human-readable lifecycle or retention context, for example pending `record start` / `record restart`, missing, stale, or otherwise unverified files, ephemeral spill files, or evicted persisted spills). The summary `verified` boolean is true only when every entry is `verified`. `record start` / `record restart` are `pending` until `record stop`; `state load` may mention a path in command output but is not a saved artifact row.
|
|
902
911
|
- `fullOutputPath` / `fullOutputPaths` when parse-valid large snapshot output or other oversized tool output is compacted and spilled to a private file; persisted sessions keep that path under a private session-scoped artifact directory with a bounded per-session budget so it survives reload/resume without unbounded growth. Malformed oversized upstream output is discarded after parsing, is omitted from `details.stdout`, and reports `fullOutputUnavailable` instead of creating a parse-failure spill.
|
|
903
912
|
- `artifactManifest` for a bounded, metadata-only inventory of recent session artifacts. Entries include path metadata, optional canonical `namespace` plus `session` lifecycle identity, artifact `kind`, source `command`/`subcommand` when safe, `storageScope` (`persistent-session`, `process-temp`, or `explicit-path`), and `retentionState` (`live`, `ephemeral`, `missing`, or `evicted`). The default recent window is 100 entries and can be configured with `PI_AGENT_BROWSER_SESSION_ARTIFACT_MANIFEST_MAX_ENTRIES`. A successful session close retires only that exact namespace/session identity's pending recording rows; the separate active reservation index remains authoritative even if this bounded display inventory evicts them. Only the newest pending recording row per namespace/session identity remains live in the manifest. The manifest must not store command args, output contents, headers, DOM snapshots, or downloaded file contents.
|
|
@@ -906,6 +915,7 @@ Additional structured fields can appear when relevant:
|
|
|
906
915
|
- compact **snapshot** metadata on successful presentation when `details.data.compacted` is true (oversized trees): `previewMode` (`"structured"` vs outline `"outline"`), `structuredPreviewUsed`, `previewRefIds`, `previewSections` (per-section `linesShown` / `omittedLines` / root `role` / `title`), `additionalSectionsOmitted`, counts such as `refCount`, `snapshotLineCount`, and `roleCounts`, optional `highValueControlRefIds` aligned with the visible bounded `Omitted high-value controls` lines, and optional `spillError` when the wrapper could not write the redacted spill file; the model text still ends with `Full redacted snapshot path:` or an explicit unavailable reason plus `details.fullOutputPath` when a path exists
|
|
907
916
|
- `sessionRecoveryHint` when startup-scoped flags need `sessionMode: "fresh"` while an implicit session is already active: includes `reason`, `recommendedSessionMode` (`"fresh"`), redacted `exampleArgs`, and `exampleParams` where `sessionMode` is `"fresh"` and `args` is the same redacted argv as `exampleArgs` (from `buildExecutionPlan` in `extensions/agent-browser/lib/runtime.ts`, merged through `redactRecoveryHint` in `extensions/agent-browser/index.ts`)
|
|
908
917
|
- `inspection: true` plus `stdout` for successful plain-text inspection commands like `--help` and `--version`
|
|
918
|
+
- valid sessionless `upgrade` commands accept native text output with surrounding whitespace trimmed and normal redaction, using the existing scalar `details.data` result. They are not inspection calls and do not claim or replace a managed browser session. Caller-requested `--json` stays a parseable `{ success, data, error? }` result. Nonzero exits, structured upstream errors, spawn failures, timeout and cancellation remain failures even when text looks successful or a terminated child exits zero; failed native text remains diagnostic data alongside the error and stderr. Caller argv and session planning are unchanged.
|
|
909
919
|
- `versionValidation` on a browser-backed preflight failure when installed upstream output is not a stable version at or above the supported floor; it includes `expected` and optional parsed `observed`, while top-level `expectedVersion` remains the recommended current baseline, `minimumSupportedVersion` reports the floor, and `observedVersion` reports the installed version. The extension caches a successful `agent-browser --version` probe per cwd/PATH for the Pi process; plain help/version, close recovery, and sessionless local commands remain available without this browser-backed gate.
|
|
910
920
|
- `agentBrowserStarted` on results that reached browser-run processing: `false` proves the requested main subprocess never started (for example a socket-path, policy, or spawn preflight failure); `true` proves only that the CLI started, not that Chrome launched. Use `details.lifecycle.effectiveLaunch.browserLaunched` for the latter. Preparation helpers may already have touched the isolated session, so script leases always take the normal fail-closed cleanup path.
|
|
911
921
|
|
|
@@ -966,12 +976,13 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
966
976
|
<!-- agent-browser-playbook:start wrapper-tab-recovery -->
|
|
967
977
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
968
978
|
- After open/goto/navigate calls with --profile, --restore, --session-name, or --state, agent_browser best-effort re-selects the tab whose URL matches the returned page when restored tabs steal focus during launch or reconnect.
|
|
969
|
-
- After
|
|
979
|
+
- After confirmed shutdown of an automatically restored managed session, the wrapper retains its complete recorded URL, including the fragment, until the first current-page operation (including get url and reload). Non-page calls such as tab list or read <url> may start a daemon without fulfilling that reopen. The wrapper uses native open once, verifies the observed tab, and discards old refs/frame scope; it does not restore unsaved forms, JavaScript memory, or history. Explicit navigation, caller-owned/attached sessions, and restore-disabled sessions are not auto-reopened.
|
|
980
|
+
- For a still-live browser after tab drift or resume, the wrapper verifies/selects the intended tab before ref/semantic helpers and page commands; failed selection stops the call without navigating. Local commands, read <url>, URL a11y/vitals, diff url, window new, and explicit tab/navigation/connection/state recovery do not require the prior tab. Batch checks follow effective rows past non-page prefixes and stop at explicit context changes, preserving caller argv/stdin and continue-on-error behavior. Same-tab reselection is avoided because it clears refs. Use exact batch --bail for fail-fast, not --bail=<value>. Routine same-session calls skip tab-list preflights.
|
|
970
981
|
- For sessions with observed tab-drift risk, after a successful command on a known target tab, agent_browser also best-effort restores that intended tab if a restored/background tab steals focus after the command completes. Routine same-session commands skip this post-command tab-list probe.
|
|
971
982
|
- If a known session target unexpectedly reports about:blank, agent_browser best-effort re-selects the prior intended target when it still exists; if recovery fails, it records the observed about:blank target and reports exact recovery guidance instead of treating the prior page as active.
|
|
972
983
|
- If upstream reports tab_gone, the pinned bound tab is gone; use details.nextActions (tab list / tab new) instead of assuming another tab is yours.
|
|
973
984
|
<!-- agent-browser-playbook:end wrapper-tab-recovery -->
|
|
974
|
-
- on local Unix launches, set a short private socket directory for wrapper-spawned `agent-browser` processes so extension-generated session names do not fail the upstream Unix socket-path length limit in longer cwd/session-name combinations; require an absolute non-symlink directory owned by the current uid with mode `0700`, otherwise fail before spawn. Android/Termux uses a short directory under the owner-only `/data/data/<package>` app sandbox, compacts generated managed identities to one 80-bit digest so ordinary namespace plus fresh-session paths remain within the limit, stores policy-lock coordination under `os.tmpdir()`, and probes process identity with Termux's `ps` beside Node instead of unavailable `/bin/ps`
|
|
985
|
+
- on local Unix launches, set a short private socket directory for wrapper-spawned `agent-browser` processes so extension-generated session names do not fail the upstream Unix socket-path length limit in longer cwd/session-name combinations; require an absolute non-symlink directory owned by the current uid with mode `0700`, otherwise fail before spawn. Socket checks trust the operating environment's actual `/`, not its reported UID; all non-root ownership, permission and alias-destination checks remain in force. This is not protection from the controller of the root filesystem; see [socket trust](ARCHITECTURE.md#ownership). Android/Termux uses a short directory under the owner-only `/data/data/<package>` app sandbox, compacts generated managed identities to one 80-bit digest so ordinary namespace plus fresh-session paths remain within the limit, stores policy-lock coordination under `os.tmpdir()`, and probes process identity with Termux's `ps` beside Node instead of unavailable `/bin/ps`
|
|
975
986
|
- keep wrapper-spawned commands bounded by clamping `AGENT_BROWSER_DEFAULT_TIMEOUT` to the upstream documented default of 25 seconds while the default wrapper child-process watchdog is 35 seconds (`PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides it, and top-level `timeoutMs` overrides it per call for browser CLI subprocesses). Explicit `wait <ms>`, `wait --timeout <ms>`, and WebMCP `invoke` / `result --timeout <ms>` calls can exceed that default; when top-level `timeoutMs` is omitted, the wrapper derives a per-call subprocess watchdog from the requested command duration plus a small grace window. Dialog commands use `PI_AGENT_BROWSER_DIALOG_PROCESS_TIMEOUT_MS` (default 5000 ms), and click/tap/find refs or tokens plus `eval --stdin` snippets whose text looks like alert/confirm/prompt/dialog triggers use `PI_AGENT_BROWSER_DIALOG_TRIGGER_PROCESS_TIMEOUT_MS` (default 8000 ms). Timed-out compiled `job` / `qa` or caller `batch` calls may add `details.timeoutPartialProgress` and visible `Timeout partial progress` evidence with per-step status, retry payloads, current page title/URL, and declared artifact path checks; timed-out dialog-like commands may add dialog status/dismiss/fresh-session recovery next actions
|
|
976
987
|
- interactive or long-running upstream families such as `chat` without a prompt, `dashboard start`, `stream enable`, `trace start`, `profiler start`, `record start`, `inspect`, `install`, `upgrade`, `doctor --fix`, and `confirm-interactive` are passed through thinly but remain bounded by the same wrapper timeout/session planning rules; prefer explicit arguments, single-shot `chat <message>`, non-interactive flags like `doctor --offline --quick` or `doctor --json`, and cleanup pairs such as `dashboard stop`, `stream disable`, `trace stop`, `profiler stop`, and `record stop`
|
|
977
988
|
- treat successful plain-text inspection commands like `--help` and `--version` as stateless: do not inject the implicit managed session and do not let those calls claim the managed-session slot
|
package/package.json
CHANGED