pi-agent-browser-native 0.3.0 → 0.6.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +265 -0
- package/README.md +130 -54
- package/dist/extensions/agent-browser/index.js +781 -169
- package/dist/extensions/agent-browser/lib/argv-descriptor.js +35 -3
- package/dist/extensions/agent-browser/lib/argv-grammar.js +50 -2
- package/dist/extensions/agent-browser/lib/batch-lifecycle.js +71 -0
- package/dist/extensions/agent-browser/lib/command-policy.js +5 -8
- package/dist/extensions/agent-browser/lib/command-taxonomy.js +53 -12
- package/dist/extensions/agent-browser/lib/config-policy.js +25 -1
- package/dist/extensions/agent-browser/lib/config.js +1 -1
- package/dist/extensions/agent-browser/lib/input-modes/job.js +61 -13
- package/dist/extensions/agent-browser/lib/input-modes/lookups.js +2 -2
- package/dist/extensions/agent-browser/lib/input-modes/params.js +23 -24
- package/dist/extensions/agent-browser/lib/input-modes/script.js +462 -0
- package/dist/extensions/agent-browser/lib/input-modes/semantic-action.js +51 -12
- package/dist/extensions/agent-browser/lib/launch-scoped-flags.js +26 -4
- package/dist/extensions/agent-browser/lib/managed-session-policy-lock.js +6 -139
- package/dist/extensions/agent-browser/lib/managed-session-restore.js +26 -116
- package/dist/extensions/agent-browser/lib/managed-session-snapshots.js +2 -4
- package/dist/extensions/agent-browser/lib/managed-session-storage.js +54 -25
- package/dist/extensions/agent-browser/lib/orchestration/batch-stdin.js +26 -5
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/artifact-paths.js +110 -30
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/click-dispatch.js +2 -1
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/diagnostics.js +54 -48
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/final-result.js +71 -5
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/index.js +2 -1
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/managed-session-daemon-policy.js +6 -7
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/snapshot-filter.js +119 -2
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare/wait-timeouts.js +7 -6
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/prepare.js +152 -64
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/process-output.js +244 -102
- package/dist/extensions/agent-browser/lib/orchestration/browser-run/session-state.js +63 -37
- package/dist/extensions/agent-browser/lib/orchestration/electron-host/index.js +20 -21
- package/dist/extensions/agent-browser/lib/orchestration/input-plan.js +36 -18
- package/dist/extensions/agent-browser/lib/orchestration/output-file.js +41 -21
- package/dist/extensions/agent-browser/lib/orchestration/script-mode.js +299 -0
- package/dist/extensions/agent-browser/lib/page-target-validation.js +270 -0
- package/dist/extensions/agent-browser/lib/pi-tool-rendering.js +32 -10
- package/dist/extensions/agent-browser/lib/playbook.js +29 -25
- package/dist/extensions/agent-browser/lib/process-environment.js +14 -0
- package/dist/extensions/agent-browser/lib/process-identity.js +5 -12
- package/dist/extensions/agent-browser/lib/process.js +130 -104
- package/dist/extensions/agent-browser/lib/recording-reservations.js +116 -0
- package/dist/extensions/agent-browser/lib/results/action-recommendations.js +63 -6
- package/dist/extensions/agent-browser/lib/results/artifact-manifest.js +62 -4
- package/dist/extensions/agent-browser/lib/results/categories.js +6 -1
- package/dist/extensions/agent-browser/lib/results/next-actions.js +19 -5
- package/dist/extensions/agent-browser/lib/results/presentation/artifacts.js +85 -38
- package/dist/extensions/agent-browser/lib/results/presentation/batch.js +86 -18
- package/dist/extensions/agent-browser/lib/results/presentation/common.js +38 -2
- package/dist/extensions/agent-browser/lib/results/presentation/diagnostics.js +18 -17
- package/dist/extensions/agent-browser/lib/results/presentation/errors.js +2 -1
- package/dist/extensions/agent-browser/lib/results/presentation/navigation.js +38 -20
- package/dist/extensions/agent-browser/lib/results/presentation/registry.js +60 -15
- package/dist/extensions/agent-browser/lib/results/presentation/semantic-action.js +1 -10
- package/dist/extensions/agent-browser/lib/results/presentation.js +36 -6
- package/dist/extensions/agent-browser/lib/results/recovery-actions.js +3 -1
- package/dist/extensions/agent-browser/lib/results/recovery-next-actions.js +9 -0
- package/dist/extensions/agent-browser/lib/results/selector-recovery.js +54 -11
- package/dist/extensions/agent-browser/lib/results/snapshot-high-value-controls.js +13 -7
- package/dist/extensions/agent-browser/lib/results/snapshot-spill.js +2 -1
- package/dist/extensions/agent-browser/lib/results/snapshot.js +4 -4
- package/dist/extensions/agent-browser/lib/runtime.js +186 -108
- package/dist/extensions/agent-browser/lib/session-page-state.js +71 -10
- package/dist/extensions/agent-browser/lib/temp.js +1 -2
- package/dist/extensions/agent-browser/lib/upstream-version.js +14 -0
- package/dist/extensions/agent-browser/lib/web-search.js +108 -24
- package/dist/extensions/agent-browser/script-worker.js +169 -0
- package/dist/scripts/agent-browser-target.mjs +21 -0
- package/docs/ARCHITECTURE.md +57 -34
- package/docs/COMMAND_REFERENCE.md +255 -68
- package/docs/ELECTRON.md +2 -2
- package/docs/RELEASE.md +12 -10
- package/docs/REQUIREMENTS.md +11 -8
- package/docs/SUPPORT_MATRIX.md +36 -24
- package/docs/TOOL_CONTRACT.md +169 -95
- package/package.json +3 -1
- package/platform-smoke.config.mjs +2 -2
- package/scripts/agent-browser-capability-baseline.mjs +87 -9
- package/scripts/agent-browser-target.mjs +21 -0
- package/scripts/build.mjs +41 -0
- package/scripts/config.mjs +1 -0
- package/scripts/doctor.mjs +16 -9
- package/scripts/platform-smoke/browser-dogfood-windows.ps1 +9 -3
- package/scripts/platform-smoke/targets.mjs +12 -6
- package/dist/extensions/agent-browser/lib/managed-session-capabilities.js +0 -20
- package/dist/extensions/agent-browser/lib/managed-session-state-policy.js +0 -583
- package/dist/extensions/agent-browser/lib/navigation-policy.js +0 -78
- package/dist/extensions/agent-browser/lib/results/presentation/managed-list-filter.js +0 -37
package/docs/TOOL_CONTRACT.md
CHANGED
|
@@ -26,17 +26,17 @@ This keeps the integration:
|
|
|
26
26
|
|
|
27
27
|
It also keeps the main UX where it belongs: the agent invokes the tool directly instead of relying on bash or a large manual command surface.
|
|
28
28
|
|
|
29
|
-
The tool guidance should be written for task discovery first, not wrapper implementation first. That means the description should emphasize browser use cases like
|
|
29
|
+
The tool guidance should be written for task discovery first, not wrapper implementation first. That means the description should emphasize browser use cases like reading live pages, clicking, filling, screenshots, extraction, and authenticated/profile-based workflows. Live/current external facts belong on `agent_browser_web_search` when that companion tool is registered. Low-level wrapper details like `stdin` and exact CLI args belong in the schema and guidelines, not the lead description.
|
|
30
30
|
|
|
31
31
|
The tool also needs an operating playbook, not just a capability list. The model should not have to rediscover basics each session, but always-on guidance must stay concise. The canonical agent-facing playbook lives in `extensions/agent-browser/lib/playbook.ts`; it provides compact runtime rules plus absolute installed-package paths to `README.md`, `docs/COMMAND_REFERENCE.md`, and this contract so agents with file tools can read targeted guidance on demand instead of receiving the full docs in prompt context. Generated Markdown fragments are updated by `npm run docs -- playbook write`, and `npm run docs -- playbook check` fails when checked-in documentation drifts.
|
|
32
32
|
|
|
33
|
-
The native command reference in `docs/COMMAND_REFERENCE.md` is driven by the same pattern:
|
|
33
|
+
The native command reference in `docs/COMMAND_REFERENCE.md` is driven by the same pattern: `scripts/agent-browser-target.mjs` owns the runtime version and `scripts/agent-browser-capability-baseline.mjs` imports it alongside help/doc inventory; selected regions are generated into the Markdown by `npm run docs -- command-reference write`, and `npm run docs` plus `npm run verify -- command-reference` catch drift (the latter also samples the installed `agent-browser` on `PATH`). Maintainer workflow details live in `AGENTS.md` under upstream capability baseline.
|
|
34
34
|
|
|
35
35
|
## Optional companion web search
|
|
36
36
|
|
|
37
37
|
`agent_browser_web_search` is a separate custom tool, not an `agent_browser` input mode. It is available when the extension can see at least one configured/resolvable Exa or Brave credential source from `~/.pi/config/pi-agent-browser-native/config.json`, `.pi/config/pi-agent-browser-native/config.json`, `PI_AGENT_BROWSER_CONFIG`, or the `EXA_API_KEY` / `BRAVE_API_KEY` environment fallbacks, and runtime execution still checks that the final available merged config has not set `webSearch.enabled` to `false`. Config layers merge global → project → `PI_AGENT_BROWSER_CONFIG` override; under Pi 0.84.0+, globally installed and CLI-loaded copies read `.pi/config/...` when Pi trust allows that project layer, and they skip the project layer when Pi reports the project is untrusted or when launched with `--no-approve`. Disable scope is explicit: a global disable is a normal user default, a project disable applies to one repo, and an override file with `webSearch.enabled: false` is the highest-priority hard disable for that run. Credential sources may be plaintext, `$ENV_VAR` / `${ENV_VAR}` interpolation, escaped literals, or command sources such as `"!op read 'op://Private/Exa/API Key'"` from any loaded config layer; they make the tool available without exposing the value in status text, and command values resolve when the tool executes. Browser profile/executable config uses the same paths and emits prompt guidance from the highest-priority loaded layer, including project config when that layer is loaded.
|
|
38
38
|
|
|
39
|
-
|
|
39
|
+
Prefer it for live/current external web facts, current docs/news, and candidate URLs. Prefer it over browser-driving public search-engine forms such as Google: headless `job`/`type` flows may be redirected to anti-bot or CAPTCHA pages, and agents should use search API results, then `agent_browser` on a target URL, instead of attempting CAPTCHA bypass. Use `agent_browser` when the task needs browser interaction, screenshots, authenticated/profile content, page inspection, or DOM work. The search tool is namespaced to avoid colliding with generic `web_search`, chooses Exa or Brave automatically from available credentials, defaults to Exa when both are available (unless `webSearch.preferredProvider` is set), and must not expose resolved API keys in content, details, errors, status output, docs examples, logs, or PR artifacts.
|
|
40
40
|
|
|
41
41
|
Config shape:
|
|
42
42
|
|
|
@@ -45,6 +45,7 @@ Config shape:
|
|
|
45
45
|
"webSearch": {
|
|
46
46
|
"enabled": true,
|
|
47
47
|
"preferredProvider": "exa",
|
|
48
|
+
"defaultSearchType": "deep-lite",
|
|
48
49
|
"exaApiKey": "$EXA_API_KEY",
|
|
49
50
|
"braveApiKey": "$BRAVE_API_KEY"
|
|
50
51
|
}
|
|
@@ -57,7 +58,12 @@ Schema:
|
|
|
57
58
|
{
|
|
58
59
|
"query": "search text",
|
|
59
60
|
"provider": "auto",
|
|
60
|
-
"searchType": "
|
|
61
|
+
"searchType": "deep-lite",
|
|
62
|
+
"includeDomains": ["docs.exa.ai"],
|
|
63
|
+
"excludeDomains": ["archive.example"],
|
|
64
|
+
"category": "publication",
|
|
65
|
+
"additionalQueries": ["Exa search API defaults"],
|
|
66
|
+
"highlightsDynamic": false,
|
|
61
67
|
"count": 5,
|
|
62
68
|
"offset": 0,
|
|
63
69
|
"country": "US",
|
|
@@ -69,9 +75,13 @@ Schema:
|
|
|
69
75
|
|
|
70
76
|
Provider notes:
|
|
71
77
|
- `provider` is optional; `auto` uses available keys plus `webSearch.preferredProvider`.
|
|
72
|
-
- `searchType` applies to Exa only and supports `auto`, `fast`, `instant`, `deep-lite`, `deep`, and `deep-reasoning`.
|
|
73
|
-
- Exa
|
|
74
|
-
-
|
|
78
|
+
- `searchType` applies to Exa only and supports `auto`, `fast`, `instant`, `deep-lite`, `deep`, and `deep-reasoning`. Effective precedence is the per-call field, then `webSearch.defaultSearchType`, then `auto`. Use `deep-lite` for implementation research, `deep` for hard multi-source work, and `deep-reasoning` only for the hardest or exhaustive work.
|
|
79
|
+
- `includeDomains` and `excludeDomains` accept 1–20 Exa hostname, path-prefix, or wildcard-subdomain strings. `category` accepts `company`, `people`, `publication`, `news`, `personal site`, or `financial report`. `company` and `people` reject `freshness` and `excludeDomains` before a request is sent.
|
|
80
|
+
- `additionalQueries` accepts 1–10 strings only when the effective type is `deep-lite`, `deep`, or `deep-reasoning`. HTTP timeouts are 15 seconds for non-deep types, 45 seconds for `deep-lite`, 60 seconds for `deep`, and 90 seconds for `deep-reasoning`.
|
|
81
|
+
- Exa requests use `/search` with `contents.highlights: true` for compact excerpts and a fixed `systemPrompt` asking for primary official sources, requested versions/dates, and distinct results. `highlightsDynamic: true` switches to `{ "dynamic": true }` and sends Exa's required beta header. The wrapper intentionally does not expose full page text or structured-output schemas.
|
|
82
|
+
- Brave-specific `searchLang` is ignored by Exa. Exa maps `country` to `userLocation`, `safesearch` moderate/strict to `moderation: true`, and `freshness` to `startPublishedDate`. Brave keeps ignoring `searchType`; explicitly requested newer Exa-only filters fail before a Brave request.
|
|
83
|
+
- Requests are serialized. Agents should run one focused query, inspect it, and make at most one follow-up rather than parallel searches. HTTP 429 means stop and report that the provider plan or limit needs time or a change.
|
|
84
|
+
- After normalization, the wrapper removes later results whose normalized URL is exactly equal to an earlier result, preserves first-result order, does not collapse distinct paths/query URLs by guesswork, and does not overfetch replacements. `details.duplicatesRemoved` is present only when rows were removed, so `details.results.length` can be smaller than the requested `count`.
|
|
75
85
|
|
|
76
86
|
Result details:
|
|
77
87
|
|
|
@@ -82,8 +92,9 @@ Result details:
|
|
|
82
92
|
"returnedQuery": "search text",
|
|
83
93
|
"count": 5,
|
|
84
94
|
"offset": 0,
|
|
85
|
-
"searchType": "
|
|
95
|
+
"searchType": "deep-lite",
|
|
86
96
|
"requestId": "request-id-when-provider-returns-one",
|
|
97
|
+
"duplicatesRemoved": 1,
|
|
87
98
|
"fetchedAt": "2026-06-02T00:00:00.000Z",
|
|
88
99
|
"results": [
|
|
89
100
|
{
|
|
@@ -92,19 +103,22 @@ Result details:
|
|
|
92
103
|
"description": "Compact summary or first highlight",
|
|
93
104
|
"highlights": ["Relevant excerpt"],
|
|
94
105
|
"source": "Example",
|
|
95
|
-
"
|
|
106
|
+
"pageDate": "2026-06-02",
|
|
96
107
|
"language": "en"
|
|
97
108
|
}
|
|
98
109
|
]
|
|
99
110
|
}
|
|
100
111
|
```
|
|
101
112
|
|
|
113
|
+
For Exa, `details.searchType` is the effective requested type even when the provider response does not echo it. `requestId` is included when Exa returns one. Brave omits both fields. `pageDate` comes from Exa `publishedDate` (an estimated page creation date) or Brave `page_age`; Brave may separately return `age`. These provider fields are not crawl/retrieval age or proof that a result matches a requested version. For version-sensitive work, inspect the result, constrain one follow-up to the primary domain (`includeDomains` for Exa or `site:` in a Brave query), and read the primary page.
|
|
114
|
+
|
|
102
115
|
## Input mode chooser
|
|
103
116
|
|
|
104
117
|
Use exactly one top-level input per call:
|
|
105
118
|
|
|
106
119
|
| When you need | Use | Notes |
|
|
107
120
|
| --- | --- | --- |
|
|
121
|
+
| One-shot loops, conditional page branches, or multi-page aggregation | `script` | Sandboxed async JavaScript with `browser()` and `emit()`; unique isolated session, bounded calls/output/time, always closed. Not for profiles, attachments, persistent auth, or reusable named recipes. |
|
|
108
122
|
| Routine browse, click, fill, screenshots, upstream commands | `args` | Default path: `open` → `snapshot -i` → `click`/`fill` `@eN` → `snapshot -i` after navigation or DOM changes. Do not pass `--json`; the wrapper injects it (see [Wrapper `--json`](#wrapper-json)). |
|
|
109
123
|
| Stable visible role/text/label/placeholder targets | `semanticAction` | Compiles to upstream `find` or `select`; optional `session` for a named upstream browser. |
|
|
110
124
|
| Short multi-step smoke or evidence flows | `job` or `qa` | Both compile to `batch`; `qa` may reclassify diagnostics as failure. |
|
|
@@ -124,39 +138,55 @@ For link and button text, use the **exact** visible label from the latest `snaps
|
|
|
124
138
|
|
|
125
139
|
## Wrapper `--json`
|
|
126
140
|
|
|
127
|
-
The extension always plans normal browser commands with `--json` prepended in `effectiveArgs` so upstream returns structured JSON for presentation and `details`. **Do not** include `--json` in caller `args`; it is unnecessary and can confuse planning or transcript hooks that treat caller-requested JSON differently. Plain-text inspection (`--help`, `--version`) keeps its own output shape. Read-only skills and local/setup commands such as
|
|
141
|
+
The extension always plans normal browser commands with `--json` prepended in `effectiveArgs` so upstream returns structured JSON for presentation and `details`. **Do not** include `--json` in caller `args`; it is unnecessary and can confuse planning or transcript hooks that treat caller-requested JSON differently. Plain-text inspection (`--help`, `--version`) keeps its own output shape. Read-only skills and local/setup commands such as auth/profile/setup, `session list`, and syntactically local state lifecycle operations skip implicit session injection as documented under `sessionMode`. Upstream session/state rows and targets remain visible, and state/config/path operations pass through unchanged.
|
|
142
|
+
|
|
143
|
+
## Experimental WebMCP
|
|
144
|
+
|
|
145
|
+
Upstream 0.36.0 exposes page-registered tools through ordinary `args`: `webmcp list`, `webmcp invoke <tool> [--params <json|@file>] [--frame <frame-id>] [--detach] [--timeout <ms>]`, `webmcp result <id>`, and `webmcp cancel <id>`. Locally managed Chrome enables the experimental CDP feature by default. `--no-webmcp` / `AGENT_BROWSER_NO_WEBMCP` / upstream config `noWebmcp` disables it; attached browsers, providers, Lightpanda, Safari/iOS, and older Chrome builds may return upstream `webmcp_unsupported`.
|
|
146
|
+
|
|
147
|
+
The wrapper keeps this as thin CLI pass-through. `webmcp list` is read-only. `invoke`, `result`, and `cancel` may run page code that mutates, rerenders, or navigates, so the wrapper rechecks the live target, emits the normal `pageChangeSummary` and `inspect-after-mutation` follow-up when applicable, and stores `refSnapshotInvalidation.reason: "page-transition"`; old page-scoped refs remain blocked until a fresh `snapshot -i`. A direct or batched call whose result is still `pending`, or a failed `result` / `cancel` attempt made while that target is unknown, does not treat the immediate URL probe or a same-batch snapshot as stable: `details.sessionTabTargetUnknown` stays true until a successful settlement, `get url`, or explicit navigation verifies the page. Its `details.nextActions` replaces the blocked snapshot suggestion with `verify-page-target-after-pending-webmcp` (`get url`); the action warns that URL inspection does not settle the detached page tool. Inside one `batch --bail`, put `get url` after a completed WebMCP mutation and before `snapshot -i`; a snapshot directly against the unknown target remains blocked. Detached invocation ids and page-returned data remain in `details.data`. When top-level `timeoutMs` is omitted, `webmcp invoke` and `webmcp result` extend the wrapper subprocess watchdog to the upstream `--timeout` value plus a small grace window, including effective raw-argument batch rows (which take precedence over stdin exactly as upstream does).
|
|
148
|
+
|
|
149
|
+
`--no-webmcp` is launch-scoped for both bare/`true` and explicit `false` values. Put it on the first call for a session or use `sessionMode: "fresh"` after an implicit managed session exists. The upstream `webmcp-gen` skill is available through stateless `skills get webmcp-gen`; an external MCP server can opt in with `mcp --tools core,webmcp`, but bare long-running `mcp` remains unsuitable for a one-shot Pi tool call.
|
|
150
|
+
|
|
151
|
+
## Dashboard reverse-proxy origins
|
|
152
|
+
|
|
153
|
+
Upstream 0.35.2 adds `dashboard start --allowed-origins <origins>` and `AGENT_BROWSER_DASHBOARD_ALLOWED_ORIGINS` for comma-separated exact HTTPS reverse-proxy origins. The wrapper treats both explicit `dashboard start` and the no-subcommand `dashboard` equivalent as sessionless local lifecycle commands when they use `--port`, `--allowed-origins`, or `--json`; it does not add a browser session or its own dashboard access layer.
|
|
128
154
|
|
|
129
155
|
## Headed and local fixture limits
|
|
130
156
|
|
|
131
|
-
-
|
|
157
|
+
- Upstream 0.35.0 and newer require separate `args` entries for global flag values. The wrapper rejects `--flag=value` global tokens before normal command dispatch, including trailing tokens; `--restore=<key>` is the explicit upstream-supported exception. Plain help/version inspection preserves exact caller argv, matching upstream. These exceptions are top-level only: global flags for `batch` belong before `batch`, and row-local equals forms fail validation before dispatch.
|
|
158
|
+
- `--headed` is an upstream global flag passed through `args` (for example `{ "args": ["--headed", "open", "https://example.com"], "sessionMode": "fresh" }`). Use it on the first launch for demos, human-observed QA, or a user-completed login. If a managed browser session already exists, use `sessionMode: "fresh"` so the launch-scoped headed/headless choice is not ignored. Wrapper-owned headed launches default `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` to `0` because upstream 0.33.2's multi-origin restore collector opens visible temporary tabs and can delay session policy inspection. The wrapper records the effective launch-time interval and reapplies it to every helper and follow-up subprocess, still-owned off-current session, Electron cleanup close, and transcript reload/resume so daemon configuration remains stable. Native close still saves, but direct window close can lose newer state because upstream exempts headed browsers from idle shutdown; set an explicit interval before launch when periodic preservation matters. Because upstream reads it when the daemon starts, changing the recorded effective interval in either direction on a running wrapper-owned headed session is rejected until close plus a fresh launch.
|
|
132
159
|
- `--profile <name|path>` is upstream Chrome profile selection. `profiles` lists Chrome profile directory names from Chrome's user data directory; `Default` is common but not guaranteed. On profile/user-data-dir failures, use `details.nextActions` or run `profiles` / `doctor`, then tell the user which profile name/path to configure before retrying.
|
|
133
160
|
- `--executable-path <path>` selects a custom Chromium-compatible browser executable when upstream can launch it. Use it with `sessionMode: "fresh"` when switching from an already-active implicit session. For non-Chrome Chromium login state, use a full profile/user-data directory path only when upstream accepts it, or attach to a debug-enabled running browser with `--auto-connect` / `connect` when appropriate.
|
|
161
|
+
- `--ca-cert <path>` and `--no-ca-cert` are launch-scoped under the wrapper and require a fresh managed session. Upstream 0.35.0 imports PEM/DER CA material into an isolated NSS store for locally launched Linux Chromium; the wrapper disables automatic managed restore when CA trust is enabled and passes caller-selected paths through unchanged.
|
|
134
162
|
- `--allowed-domains <list>` is launch-scoped under the wrapper and requires a fresh local Chrome context. As of upstream 0.32.0, it contains workers and popups and disables Chromium `RTCPeerConnection`; upstream rejects combinations with CDP/auto-connect, profiles, restore/state replay, direct-page providers, iOS/Safari, and startup/profile Chrome args because those paths cannot guarantee containment.
|
|
135
|
-
-
|
|
136
|
-
- `localhost` / `127.0.0.1` URLs are resolved by the browser host, which may differ from the shell or Pi process that started a temporary server. Errors such as `net::ERR_EMPTY_RESPONSE` on local ports are not reliable page-render evidence; they can mean the browser cannot reach the host loopback. Use an environment-specific host-reachable HTTP(S) address.
|
|
163
|
+
- On a successful first/fresh local wrapper-managed headed launch whose upstream lifecycle says a browser launched, `details.browserWindow` is `{ mode: "headed", ownership: "wrapper-managed", sessionName, visibility: "unverified" }` and model-visible output adds one headed-login handoff sentence. It proves the requested local headed launch path, not OS desktop visibility; CDP, auto-connect, provider, and Electron attachments do not receive this field or handoff. If the user can see it, they can finish the login and the agent should continue with `sessionMode: "auto"`; otherwise inspect `screenshot`, `tab list`, `get url`, or `snapshot -i` and treat the problem as display/provider/session setup.
|
|
164
|
+
- `localhost` / `127.0.0.1` URLs are resolved by the browser host, which may differ from the shell or Pi process that started a temporary server. Errors such as `net::ERR_EMPTY_RESPONSE` on local ports are not reliable page-render evidence; they can mean the browser cannot reach the host loopback. Use an environment-specific host-reachable HTTP(S) address. Caller-selected `file://` navigation and follow-up inspection pass through unchanged.
|
|
137
165
|
- `file://` pages do not provide HTTP headers and can differ from HTTP pages for MIME handling, CORS, storage, and debugger/script behavior. If `eval --stdin` returns `null` or otherwise fails to prove DOM state on a `file://` page, first confirm the script was passed through the native tool `stdin` field (not as a third `args` item after `--stdin`), then treat that verification as inconclusive and use `snapshot -i`, `get text` from current refs, screenshots, or a reachable HTTP fixture instead.
|
|
138
166
|
- Temporary HTTP servers launched outside the tool are host-owned. The native tool does not allocate ports, track background server PIDs, or clean them up; use a harness or shell cleanup for those processes.
|
|
139
167
|
|
|
140
168
|
<!-- agent-browser-playbook:start shared-guidelines -->
|
|
141
169
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
170
|
+
- Use top-level script only for one-shot loops, conditional page branches, or multi-page aggregation that would otherwise require several calls: await browser({ args, stdin?, timeoutMs? }), branch on its ok field, and emit one bounded JSON value. Script gets an isolated non-profile browser session that is always closed, cannot use caller session/namespace/lifecycle/attachment controls, inherited agent-browser launch/proxy settings, or host APIs, and is not a reusable named recipe. One top-level approval can authorize up to 25 inner calls, so inspect the full source before approval. Use args/job/qa for ordinary linear flows.
|
|
142
171
|
- Standard workflow: open the page, snapshot -i, interact using current @refs from that snapshot, and re-snapshot after navigation, scrolling, rerendering, or other major DOM changes because refs are page-scoped; the wrapper fails mutation-prone stale/recycled refs before upstream can silently target a different current-page element. On dense pages, use wrapper-side snapshot -i --search <text> or snapshot -i --filter role=<role> to render matching refs while preserving the full ref map in details.refSnapshot, add snapshot --viewport when scroll position or above/below-fold context matters, and add snapshot --diff when a quick before/after ref-map delta would prevent reading a full spill file.
|
|
143
172
|
- For ordinary forms from one snapshot, batch multiple fill @refs before the submit/click step to avoid serial tool calls; if a fill may autosubmit, navigate, or rerender later fields, split the flow and refresh refs first.
|
|
144
|
-
- Do not use browser automation to drive public search-engine forms such as Google for discovery; headless jobs that type a query and press Enter can be redirected to anti-bot or CAPTCHA pages.
|
|
145
|
-
- Snapshot choice: prefer snapshot -i for routine clicks/fills (interactive @refs, main-content-first). Use snapshot --compact when you need a denser same-page tree without full spill; use full snapshot (no -i) only when you need the complete accessibility tree. Re-snapshot after navigation or major DOM changes. When snapshot -i compacts because the tree is oversized, scan visible output for Omitted high-value controls and optional details.data.highValueControlRefIds before opening the spill file: those list bounded searchboxes, textboxes, comboboxes, buttons, tabs, checkboxes, radios, options, and menuitems that did not fit the key/other ref previews.
|
|
173
|
+
- Do not use browser automation to drive public search-engine forms such as Google for discovery; headless jobs that type a query and press Enter can be redirected to anti-bot or CAPTCHA pages. Prefer agent_browser_web_search for live discovery, then agent_browser on a target URL. Do not attempt CAPTCHA bypass.
|
|
174
|
+
- Snapshot choice: prefer snapshot -i for routine clicks/fills (interactive @refs, main-content-first). Use snapshot --compact when you need a denser same-page tree without full spill; use full snapshot (no -i) only when you need the complete accessibility tree. Re-snapshot after navigation or major DOM changes. When snapshot -i compacts because the tree is oversized, scan visible output for Omitted high-value controls and optional details.data.highValueControlRefIds before opening the spill file: those list bounded searchboxes, textboxes, comboboxes, buttons, named action links, tabs, checkboxes, radios, options, and menuitems that did not fit the key/other ref previews.
|
|
146
175
|
- When a visible text or accessible-name target should survive ref churn, prefer find locators such as role, text, label, placeholder, alt, title, or testid with the intended action instead of guessing a CSS selector.
|
|
147
|
-
- For desktop or host-controlled rich inputs, if semanticAction fill misses, refresh refs and prefer a current editable @ref from details.richInputRecovery or the latest snapshot; focus or click that ref, then use keyboard
|
|
176
|
+
- For desktop or host-controlled rich inputs, if semanticAction fill misses, refresh refs and prefer a current editable @ref from details.richInputRecovery or the latest snapshot; focus or click that ref, then use keyboard type for framework-controlled editors that require real key events. keyboard inserttext is paste-like and can change a DOM value without updating application state, so use it only when later application-state evidence proves the edit was accepted. Do not auto-submit with Enter or a submit button unless the user flow explicitly calls for it.
|
|
148
177
|
- Do not assume Playwright selector dialects such as text=Close or button:has-text('Close') are supported wrapper syntax unless current upstream agent-browser behavior has been verified.
|
|
149
|
-
- For authenticated or user-specific content explicitly requested by the user, such as feeds, inboxes, account pages, or private dashboards, use a real profile only when the user/config asks for it or profiles have been inspected; do not assume --profile Default exists on every machine. Do not use a real profile for public pages just because they are dashboards. Treat visible page content from real profiles as model-visible transcript data; use --auto-connect only if profile-based reuse is unavailable or the task is specifically about attaching to a running debug-enabled browser. If profile/user-data-dir resolution fails, stop retrying opens, run profiles and/or doctor through agent_browser, then report what the user needs to configure.
|
|
178
|
+
- For authenticated or user-specific content explicitly requested by the user, such as feeds, inboxes, account pages, or private dashboards, use a real profile only when the user/config asks for it or profiles have been inspected; do not assume --profile Default exists on every machine. Do not use a real profile for public pages just because they are dashboards. Treat visible page content from real profiles as model-visible transcript data. On macOS, copied Chrome profiles may omit encrypted cookies, so profile selection alone is not proof of authentication; verify the target page and use a user-approved headed login once when needed. Use --auto-connect only if profile-based reuse is unavailable or the task is specifically about attaching to a running debug-enabled browser. If profile/user-data-dir resolution fails, stop retrying opens, run profiles and/or doctor through agent_browser, then report what the user needs to configure.
|
|
150
179
|
- Do not invent fixed explicit session names for routine tasks. Use the implicit session unless you truly need multiple isolated browser sessions in the same conversation.
|
|
151
|
-
- When using launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --enable, --executable-path, --webgpu, --init-script, --idle-timeout, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), put them on the first command for that session. If you intentionally use an explicit --session, keep using that same explicit session for follow-ups.
|
|
180
|
+
- When using launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --ca-cert, --no-ca-cert, --enable, --executable-path, --webgpu, --no-webmcp, --init-script, --idle-timeout, --args, --user-agent, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), put them on the first command for that session. If you intentionally use an explicit --session, keep using that same explicit session for follow-ups.
|
|
152
181
|
- Caller-owned explicit sessions are serialized per effective canonical namespace/session inside this extension while live URL checks, semantic-action snapshots, and the requested command run. For raw batches whose later content step depends on navigation, use exact batch --bail or split the calls; unsafe continue-after-navigation-failure shapes are rejected before the batch runs.
|
|
153
|
-
- After a successful `connect`, `--cdp`, or enabled `--auto-connect` call, verify with get url and keep using the resulting session without repeating the attach flag. The wrapper remembers that attachment across active-branch reload/resume
|
|
154
|
-
- If you already used the implicit session and now need launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --enable, --executable-path, --webgpu, --init-script, --idle-timeout, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), retry with top-level sessionMode set to fresh or pass an explicit --session for the new launch; never pass --session-mode inside args. After a successful unnamed fresh launch, later auto calls follow that new session.
|
|
182
|
+
- After a successful `connect`, `--cdp`, or enabled `--auto-connect` call, verify with get url and keep using the resulting session without repeating the attach flag. The wrapper remembers that attachment across active-branch reload/resume and live-checks the URL before later page reads/interactions because an attached browser can drift externally; caller config, file access, launch arguments, and environment pass through unchanged. A successful close clears the marker. When several named sessions share one Chrome, pass --pin-tab once (AGENT_BROWSER_PIN_TAB) so a closed bound tab fails as tab_gone instead of acting on a neighbor; recover with tab new or tab list. --no-pin-tab turns the sticky pin off. tab list includes each tab's CDP targetId, accepted as a tab ref.
|
|
183
|
+
- If you already used the implicit session and now need launch-scoped flags (--auto-connect, --allowed-domains, --namespace, --cdp, --ca-cert, --no-ca-cert, --enable, --executable-path, --webgpu, --no-webmcp, --init-script, --idle-timeout, --args, --user-agent, --headed, --device, --profile, --provider, -p, --session-name, --restore, --restore-save, --restore-check-url, --restore-check-text, --restore-check-fn, --state), retry with top-level sessionMode set to fresh or pass an explicit --session for the new launch; never pass --session-mode inside args. After a successful unnamed fresh launch, later auto calls follow that new session.
|
|
155
184
|
- For WebGPU pages, use args ["--webgpu", "open", "<url>"] on a fresh local browser launch; use doctor --webgpu (or --headed on Linux/Windows capture paths) to prove rendering before trusting a non-black screenshot. WebGPU cannot be combined with --cdp, --auto-connect, or provider launches unless --webgpu false overrides an enabled config/environment default.
|
|
185
|
+
- For experimental WebMCP page tools, use webmcp list, then webmcp invoke <tool> with --params and optional --frame/--detach/--timeout; use webmcp result or cancel for detached calls. Locally managed Chrome enables WebMCP by default. --no-webmcp is launch-scoped and requires a fresh session; invoke/result/cancel can mutate or navigate, so refresh snapshot refs afterward.
|
|
156
186
|
- For --allowed-domains, use a fresh local Chrome context. Upstream rejects CDP/auto-connect, profiles, restore/state replay, direct-page providers, iOS/Safari, and startup/profile Chrome args because they cannot guarantee containment; Chromium also disables RTCPeerConnection while the allowlist is active.
|
|
157
187
|
- For React introspection, launch the page with --enable react-devtools before first navigation, then use react tree, react inspect <fiberId>, sourceLookup candidates for local UI source hints, react renders start/stop, or react suspense; sourceLookup is experimental and reports confidence/evidence instead of guaranteed DOM-to-file mappings. For failed fetches and APIs, networkSourceLookup (experimental) correlates failed network requests with initiator metadata and bounded workspace URL literals—candidates only, not definitive blame. Use vitals [url] for Core Web Vitals and hydration timing, and pushstate <url> for client-side SPA navigation.
|
|
158
188
|
- For first-navigation setup, use open without a URL plus network route --resource-type <csv>, cookies set --curl <file>, or --init-script/--enable before navigate/opening the target page.
|
|
159
|
-
- For stateful browser context work,
|
|
189
|
+
- For stateful browser context work, use auth save --password-stdin with the tool stdin field for credentials, auth list/show/delete/remove for local auth-profile maintenance, auth login when you need the browser to fill a saved profile, and state save/load/list/show/rename/clear/clean for upstream saved-state lifecycle. State paths, restore identifiers, wrapper-prefixed sessions, and all upstream list rows remain available; credential values inside cookie/storage/auth payloads are still redacted from presentation.
|
|
160
190
|
- Upstream restore sessions periodically autosave cookies and localStorage while the browser stays open, including page-driven background changes; AGENT_BROWSER_AUTOSAVE_INTERVAL_MS controls the interval (30000 by default; 0 disables periodic saves but keeps save-on-close), while the never value for --restore-save disables automatic saves for that restore session. For wrapper-owned headed launches, the wrapper defaults the interval to 0 because upstream 0.33.2 collects multi-origin storage through visible temporary tabs, then records and reapplies the effective launch-time value to helpers and follow-ups so daemon configuration remains stable. Native close still saves, but direct window close can lose newer state because upstream exempts headed browsers from idle shutdown; set AGENT_BROWSER_AUTOSAVE_INTERVAL_MS before launch when periodic preservation matters, because changing it on a running wrapper-owned headed session requires close plus a fresh launch.
|
|
161
191
|
- For batch chains that touch cookies, storage, auth, or other secret-bearing commands, use details.batchSteps for per-step artifacts, categories, spill paths, and full structured errors; top-level details.data on batch is only a compact redacted step matrix (success, argv-redacted command, redacted result or scrubbed error text) built from the same presentation rules as standalone calls.
|
|
162
192
|
- For non-core families, pass current upstream commands through the native tool directly: network requests, network route <url>, network har start/stop [path], diff snapshot, diff screenshot --baseline <file>, diff url <u1> <u2>, trace start, trace stop [path], profiler start, profiler stop [path], record start <path>, record stop, console/errors [--clear], highlight <selector>, inspect, clipboard read, clipboard write <text>, clipboard copy/paste, stream enable/disable/status, dashboard start/stop, device list for iOS simulator inventory, and chat <message>. For compact network requests output, prefer details.nextActions for request detail, route-mock diagnostics, actionable failed-request networkSourceLookup, filtering, clearing the aggregate buffer before repro, or HAR capture follow-ups instead of guessing request-id syntax. Artifact-producing commands report details.artifacts and verification state; long-running starts such as stream, dashboard, trace/profiler, and record should be paired with the matching stop/disable command when the task is done; stream enable already-enabled outcomes are treated as idempotent success with status/disable follow-ups.
|
|
@@ -165,22 +195,26 @@ The extension always plans normal browser commands with `--json` prepended in `e
|
|
|
165
195
|
- For dialogs and frames, use dialog status/accept/dismiss and frame <selector|main> through native args; dialog commands and eval snippets that look like alert/confirm/prompt/dialog triggers are shorter-bounded than normal browser calls, and timed-out dialog-like interactions may add inspect-dialog-after-timeout, dismiss-dialog-after-timeout, or recover-fresh-session-after-dialog-timeout nextActions. When --confirm-actions produces a pending confirmation, use details.nextActions or exact confirm <id> / deny <id> calls instead of inventing ids.
|
|
166
196
|
- If a session lands on the wrong page or tab, an interaction changes origin unexpectedly, or an open call returns blocked, blank, or otherwise unexpected results, use tab list / tab <tab-id-or-label> / snapshot -i to recover state before retrying different URLs or fallback strategies. For headed demos, put --headed on the first launch with sessionMode=fresh and verify with screenshot/tab/get-url evidence because tool success cannot prove the OS window is visible to the user. For desktop readiness, prefer real conditions first: wait --text, wait --url, wait --fn, wait --load <state>, wait --download, or qa.attached; for disappearance checks, use wait --fn predicates instead of stale upstream-help examples like wait <selector> --state hidden. Use electron.probe/status for wrapper-owned launch health or target mismatch. Fixed waits are a last resort: use explicit --timeout or top-level timeoutMs for legitimately slow waits, and treat a successful payload like "waited":"timeout" as elapsed time only—verify completion with an observed condition, fresh snapshot, or screenshot.
|
|
167
197
|
- For feed, timeline, or inbox reading tasks, focus on the main timeline/list region and read the first item there rather than unrelated composer or sidebar content.
|
|
168
|
-
- For read-only browsing tasks, use read <url> for documentation or other unstructured text without
|
|
198
|
+
- For read-only browsing tasks, use read <url> for documentation or other unstructured text without requiring a Chrome page, or read with no URL for rendered active-tab DOM. Prefer the current snapshot, structured ref labels, getters, or scoped eval --stdin when you need interactive structure or targeted page state. Only click into media viewers, detail routes, or new pages when the current view does not contain the needed information.
|
|
169
199
|
- For downloads, prefer download <selector> <path> when an element click should save a file; simple loopback anchor downloads are saved to the requested path when the wrapper can resolve an HTTP(S) href. Do not rely on click alone when you need the downloaded file on disk.
|
|
170
200
|
- On dashboards with nested scroll containers, verify scroll with a screenshot or fresh snapshot -i; if the viewport did not move, details.data.scrolled may be false/noMovement true and you should prefer scrollintoview <@ref> or target the actual scrollable region with scroll <selector> <dir> [px|percent]. For native selects, use select <selector> <value...> (or semanticAction/job select) instead of clicking option refs; for custom comboboxes, a click/semanticAction may only focus the field, so re-snapshot and fall back to type, press Enter/arrow keys, or visible option refs.
|
|
171
201
|
- When using eval --stdin, scope checks and actions to the target element or route whenever possible instead of relying on broad page-wide text heuristics.
|
|
172
|
-
- When using eval --stdin for extraction, pass the JavaScript through the native tool stdin field, not as an extra args token after --stdin, and return the value you want instead of relying on console.log as the primary result channel. Prefer plain expressions like ({ title: document.title }) or explicitly invoked functions like (() => ({ title: document.title }))(); use outputPath when the eval/get/snapshot data should be saved as a durable local file, but never reuse a screenshot, download, recording, or other browser artifact destination as outputPath. If a function-shaped snippet returns {}, details.evalStdinHint may warn that the function was serialized instead of called.
|
|
202
|
+
- When using eval --stdin for extraction, pass the JavaScript through the native tool stdin field, not as an extra args token after --stdin, and return the value you want instead of relying on console.log as the primary result channel. Prefer plain expressions like ({ title: document.title }) or explicitly invoked functions like (() => ({ title: document.title }))(); use outputPath when the eval/get/snapshot data should be saved as a durable local file, but never reuse a screenshot, download, recording, or other browser artifact destination as outputPath. If a function-shaped snippet returns {}, details.evalStdinHint may warn that the function was serialized instead of called. Local file pages and caller-selected output paths are supported when upstream allows them. If get text on a broad CSS selector surfaces details.selectorTextVisibility or selectorTextVisibilityAll, prefer a visible @ref, a more specific selector, or the inspect-visible-text-candidates nextAction over hidden tab content.
|
|
173
203
|
- When details.pageChangeSummary is present, use changeType and summary as a compact signal for navigation, DOM mutation, confirmations, or artifacts; when nextActionIds is set, match those ids to entries in details.nextActions (or per-step nextActions inside batch) for concrete follow-up payloads instead of inferring from prose alone. If details.clickDispatch reports a click-dispatch miss, refresh/inspect/retry the real click first; for static local fixtures only, an explicit eval --stdin programmatic .click() can exercise app handlers, but treat it as an untrusted scripted workaround and never use it to bypass stop-before-submit/order/purchase boundaries. If a no-navigation click surfaces details.overlayBlockers, inspect the fresh snapshot evidence before using a close/dismiss candidate nextAction; ordinary page chrome without dialog/alertdialog evidence should not trigger this diagnostic.
|
|
174
204
|
- When commands save or spill files (screenshots, downloads, PDFs, traces, recordings, HAR, large snapshot spills), use the user's exact requested paths when given and treat paths as provisional until details.artifactVerification shows every row verified: branch on missingCount, pendingCount, unverifiedCount, per-entry state, and optional limitation before downstream file use or PASS/FAIL reporting.
|
|
175
205
|
- For evidence-only screenshots, QA captures, or other audit artifacts, save to an explicit path and branch on details.artifactVerification plus details.artifacts before reporting PASS/FAIL; do not require vision review of inline image attachments unless the user asked for visual inspection.
|
|
176
|
-
- Respect explicit user stop boundaries yourself
|
|
206
|
+
- Respect explicit user stop boundaries yourself. When the surrounding authenticated employee or automation context is explicitly unattended/auto-approved, ordinary non-destructive form submissions within the requested flow may proceed without separate confirmation. Still require explicit authorization for purchases, production-control actions, destructive or irreversible actions, and account, security, or privacy changes. The wrapper does not infer broad business intent from prompt text; details.promptGuard is reserved for concrete artifact-before-close checks.
|
|
177
207
|
- Successful record stop needs ffmpeg on PATH; the wrapper may warn after record start when ffmpeg is missing.
|
|
178
208
|
- Do not call --help or other exploratory inspection commands unless the user explicitly asks for them or debugging the browser integration is necessary.
|
|
179
209
|
<!-- agent-browser-playbook:end shared-guidelines -->
|
|
180
210
|
|
|
181
211
|
## Parameters
|
|
182
212
|
|
|
183
|
-
Illustrative shapes (each real call uses exactly one of `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, or `electron`):
|
|
213
|
+
Illustrative shapes (each real call uses exactly one of `script`, `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, or `electron`):
|
|
214
|
+
|
|
215
|
+
```json
|
|
216
|
+
{ "script": "const title = await browser({ args: ['get', 'title'] }); if (!title.ok) throw new Error(title.error); emit(title.data.title ?? title.data.result);" }
|
|
217
|
+
```
|
|
184
218
|
|
|
185
219
|
```json
|
|
186
220
|
{ "args": ["open", "https://example.com"], "stdin": "optional raw stdin content", "outputPath": "logs/result.json", "timeoutMs": 35000, "sessionMode": "auto" }
|
|
@@ -200,7 +234,7 @@ Illustrative shapes (each real call uses exactly one of `args`, `semanticAction`
|
|
|
200
234
|
### `args`
|
|
201
235
|
|
|
202
236
|
- type: `string[]`
|
|
203
|
-
- required unless `semanticAction`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, or `electron` is provided
|
|
237
|
+
- required unless `script`, `semanticAction`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, or `electron` is provided
|
|
204
238
|
- exact CLI args passed after `agent-browser`; this is the 1:1 upstream CLI coverage path for the targeted `agent-browser` version
|
|
205
239
|
- no shell operators
|
|
206
240
|
- do not include the binary name
|
|
@@ -218,10 +252,35 @@ Examples:
|
|
|
218
252
|
{ "args": ["quit"] }
|
|
219
253
|
```
|
|
220
254
|
|
|
255
|
+
### `script`
|
|
256
|
+
|
|
257
|
+
- type: string, maximum 65,536 characters at the public schema and 65,536 UTF-8 bytes at execution
|
|
258
|
+
- optional; mutually exclusive with `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, `electron`, `sessionMode`, and top-level `stdin`
|
|
259
|
+
- intended only for one-shot loops, conditional page branches, and multi-page aggregation where several top-level browser calls and host artifact reads would otherwise be required; use ordinary `args`, `job`, or `qa` for linear workflows
|
|
260
|
+
- executes as an async JavaScript body in a separate Node child with a 64 MiB V8 heap limit, Node permission mode, VM string/WebAssembly code generation disabled, an empty environment, no imports or dynamic imports, and no exposed host objects/functions; common host capabilities such as `process`, `require`, filesystem, network, and timers are absent, while context-native `eval` / `Function` code generation is disabled
|
|
261
|
+
- exposes only `browser(params)` and `emit(value)` as null-prototype task-specific functions. The ordinary context-native JavaScript intrinsics remain available for local computation.
|
|
262
|
+
- `browser({ args, stdin?, timeoutMs? })` runs each inner call through the same full native-tool executor used by a top-level `args` call. It returns a JSON clone shaped as `{ ok, data, details?, error?, failureCategory?, nextActions?, resultCategory, successCategory?, summary, text }`; callers must check `ok`. Wrapper-verified parse-valid compact spills may be rehydrated into `data` before the clone is returned, and ordinary structured/prose redaction still applies. `summary` and `text` are bounded before they enter IPC or `scriptSteps`; if the complete serialized envelope still exceeds the per-message limit, that inner call resolves to a bounded `{ ok: false, failureCategory: "upstream-error" }` envelope instead of terminating the sandbox protocol. Browser-bearing `nextActions` are exposed only when their params pass script policy after the wrapper removes the current isolated `--namespace "" --session piab-script-*` prefix; unsupported top-level modes and local commands are omitted, while artifact-path metadata may remain.
|
|
263
|
+
- `emit(value)` accepts JSON-compatible data. One emission becomes `data` directly; multiple emissions become a `data` array in emission order. With no emission, the async body’s return value becomes `data`; `data` is omitted when the body returns nothing. When any emission exists, the body return value is ignored.
|
|
264
|
+
- browser calls are serialized even when source uses `Promise.all`; the maximum is 25 attempted calls. Source and final serialized output are each capped at 64 KiB; cumulative child/parent IPC is bounded. Final data is redacted, serialized without pretty-print amplification, and byte-checked again before presentation; excessive depth or post-redaction growth becomes `failureCategory: "validation-error"` with data omitted.
|
|
265
|
+
- default top-level `timeoutMs` is 120,000 ms for script; the hard ceiling is 300,000 ms. Every inner timeout is clamped to the remaining outer deadline. Abort, timeout, Pi branch change, quit, reload, and child failure cascade to the active inner call, wait for isolated-session cleanup, then terminate and reap the sandbox child with a bounded SIGTERM/SIGKILL sequence.
|
|
266
|
+
- before the first accepted inner browser call, the wrapper appends a model-invisible Pi custom-entry lease containing only a strict wrapper-generated `piab-script-<uuid>` session name, exact close argv, `launchAttempted: true`, and cleanup state. Script mode therefore requires a persisted Pi session and fails validation under `--no-session`.
|
|
267
|
+
- every invocation uses a unique wrapper-owned session in the empty canonical namespace, with managed restore disabled. It never reads, replaces, or updates the extension-managed implicit conversation session. Inner identity/lifecycle/attachment/local commands, nested `batch`, nested top-level modes, `--session`, `--namespace`, profile/state/restore/provider/CDP/raw-args/init/extension launch controls, and local/sessionless commands are rejected before upstream spawn. Every script-owned helper and cleanup subprocess also clears ambient `AGENT_BROWSER_*` and standard proxy variables before wrapper-owned namespace, timeout, and compatibility values are applied.
|
|
268
|
+
- the wrapper closes the isolated session in `finally`, including after error, timeout, abort, or branch change. `session_tree` and `session_shutdown` abort active scripts and await their normal cleanup before branch restoration or shutdown cleanup continues. On session start or branch change, exact non-closed leases from the active branch are then retried; forged names or close argv are ignored.
|
|
269
|
+
- details include `scriptRun.callCount`, disjoint `successfulCallCount`, ordinary dispatched `failedCallCount`, `preDispatchRejectedCallCount`, and emission/timeout/abort fields plus bounded `scriptSteps` category/summary rows. Any pre-dispatch policy or validation rejection fails the top-level tool result even when source handles the returned `{ ok: false }` envelope. When a browser launch was attempted, `scriptSession: { sessionName, cleanup: "closed", closeCommandArgs, launchAttempted: true }` and the compact prose both confirm successful cleanup. Uncaught user-source exceptions use `failureCategory: "script-error"`. Cleanup failure overrides every script outcome with `failureCategory: "cleanup-failed"`, `scriptSession.cleanup: "failed"`, a redacted error, and `nextActions[0].id: "close-script-session-after-cleanup-failure"` carrying the exact close args.
|
|
270
|
+
- this is not a reusable recipe layer: there is no script name, registry, persistence, cross-call state, host module import, or workflow versioning surface. Pi approval applies to the one visible top-level tool input, which may issue up to 25 inner calls. The collapsed Pi tool row renders a bounded terminal-safe source preview with line breaks marked as `↵`; expand the row to inspect the full terminal-safe script before approving it. Rendering normalizes JavaScript CR, U+2028, and U+2029 line terminators to visible newlines and replaces stripped terminal/directional/zero-width controls with a visible marker so sanitization cannot join a line comment to executable source or silently hide removed characters.
|
|
271
|
+
|
|
272
|
+
Example with a conditional branch and aggregation:
|
|
273
|
+
|
|
274
|
+
```json
|
|
275
|
+
{
|
|
276
|
+
"script": "const out = []; for (const url of ['https://example.com/a', 'https://example.com/b']) { const opened = await browser({ args: ['open', url] }); if (!opened.ok) throw new Error(opened.error); const probe = await browser({ args: ['eval', '--stdin'], stdin: \"({ blocked: Boolean(document.querySelector('[role=dialog]')), title: document.title })\" }); if (!probe.ok) throw new Error(probe.error); if (probe.data.result.blocked) { const closed = await browser({ args: ['click', '[role=dialog] button'] }); if (!closed.ok) throw new Error(closed.error); } out.push({ url, title: probe.data.result.title }); } emit(out);"
|
|
277
|
+
}
|
|
278
|
+
```
|
|
279
|
+
|
|
221
280
|
### `semanticAction`
|
|
222
281
|
|
|
223
282
|
- type: object
|
|
224
|
-
- optional; mutually exclusive with `args`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, and `electron` (omit all of them when using this field)
|
|
283
|
+
- optional; mutually exclusive with `script`, `args`, `job`, `qa`, `sourceLookup`, `networkSourceLookup`, and `electron` (omit all of them when using this field)
|
|
225
284
|
- top-level tool input only: `batch` stdin remains upstream argv arrays; express find steps inside batch as string arrays such as `["find","role","button","click","--name","Export"]`, not nested `semanticAction` objects
|
|
226
285
|
- thin intent schema compiled by this wrapper into existing upstream commands; locator actions compile to `find`, direct selector/ref `click` / `check` / `fill` compile to the matching upstream command, and native dropdown selection compiles to `select <selector> <value...>`; behavior and locator/selector semantics stay upstream-owned
|
|
227
286
|
- supported actions: `click`, `fill`, `check`, `select`
|
|
@@ -230,7 +289,7 @@ Examples:
|
|
|
230
289
|
- `semanticAction` does not expose `uncheck` because upstream `find` actions are only `click, fill, check, hover, text`; use raw `args: ["uncheck", <selector-or-ref>]` after a stable selector or current snapshot ref
|
|
231
290
|
- for locator actions, `value` is the locator argument (for example ARIA role token `"button"`, label text, or visible substring), must be a non-empty string after trim; for `locator: "role"`, callers may provide `role` instead of redundant `value`
|
|
232
291
|
- `fill` requires non-empty `text` (compiled as the trailing value argument to `find`)
|
|
233
|
-
- `select`
|
|
292
|
+
- `select` accepts either a non-empty direct `selector` plus `value` (one option value) or `values` (one or more option values), or an active-session semantic locator that the wrapper resolves through a fresh `snapshot -i` to exactly one current visible native dropdown ref. For `locator: "role"`, set `role` to `combobox` or `listbox`, set `name` to the accessible name, and provide option `value` / `values`. For `locator: "label"`, `value` is the accessible label text and `values` contains the option value(s). Locator-based select fails before action when no active browser exists or the current snapshot is missing/ambiguous; no fuzzy target is selected.
|
|
234
293
|
- optional `name` is only valid with `locator: "role"` and compiles to `--name <name>` after the action (and after `text` for `fill` when present)
|
|
235
294
|
- optional `role` is accepted only when `locator` is `role`; it may replace `value`, and must equal `value` if both are set
|
|
236
295
|
- optional `session` is an upstream session name; when set, compilation prepends `--session <session>` before the compiled `find`, direct selector/ref command, or `select` command so the shorthand targets that named browser context instead of the managed default; this is independent of top-level `sessionMode`, which only injects or rotates the extension-managed implicit session when the planned argv does not already start with `--session` (see `buildExecutionPlan` in `extensions/agent-browser/lib/runtime.ts`). On successful unified results, `details.sessionName` matches that name and `usedImplicitSession` is `false` because the call named upstream directly rather than consuming the extension-managed implicit session slot.
|
|
@@ -244,17 +303,19 @@ Compilation (then `--json` and session handling apply like any other call):
|
|
|
244
303
|
| `fill` | `["find",<locator>,<value>,"fill",<text>]` plus optional `["--name",<name>]` after `text` when `locator` is `role` and `name` is set |
|
|
245
304
|
| `click` / `check` / `fill` + `selector` | `[<action>,<selector>]` (plus `<text>` for `fill`) |
|
|
246
305
|
| `select` + `selector` + `value` / `values` | `["select",<selector>,<value...>]` |
|
|
306
|
+
| `select` + `locator: "role"` + `role: "combobox"` or `"listbox"` + `name` + `value` / `values` | resolves one exact current visible ref, then executes `["select","@ref",<value...>]` |
|
|
307
|
+
| `select` + `locator: "label"` + label `value` + option `values` | resolves one exact current visible combobox/listbox ref, then executes `["select","@ref",<value...>]` |
|
|
247
308
|
| any supported action + `session` | prepends `["--session",<session>]` before the compiled argv |
|
|
248
309
|
|
|
249
|
-
When `semanticAction` compiles successfully, `details.compiledSemanticAction` echoes `{ action, locator, args }` for `find` actions, `{ action, selector, args }` for direct selector/ref click/check/fill actions, or `{ action: "select", selector
|
|
310
|
+
When `semanticAction` compiles successfully, `details.compiledSemanticAction` echoes `{ action, locator, args }` for `find` actions, `{ action, selector, args }` for direct selector/ref click/check/fill actions, or `{ action: "select", selector?, locator?, values, args }` for `select`, with `args` redacted the same way as other invocation details. Expect it on the initial wrapper validation return (when that path still builds the early `details` object) and on the unified result after `agent-browser` runs. It is omitted when the call used `args` only, when compilation never produced argv, and on some in-`execute` error returns that attach a slimmer `details` shape before the unified merge (for example certain session-plan, stdin-contract, tab-pinning, or missing-binary guard paths); compare `extensions/agent-browser/index.ts` where `compiledSemanticAction` is assigned. For active sessions, role/name `click`, `check`, guarded `fill`, and locator-based `select` semantic actions may be resolved through one fresh `snapshot -i` to a current visible `@ref` before execution; for a caller-owned explicit `semanticAction.session`, the wrapper first verifies the live URL, and the same process-local per-session critical section covers that probe, snapshot, and main action; fill only resolves when one exact editable `combobox`, `searchbox`, or `textbox` ref matches. This avoids hidden duplicate matches stealing an upstream `find` action. In that case `details.compiledSemanticAction` keeps the original semantic target fields; locator-based `select` reports the resolved `select @ref` in its `args`, while other resolved actions keep their original compiled `find` args and expose the executed ref action through `details.effectiveArgs`.
|
|
250
311
|
|
|
251
312
|
If a raw `find` or compiled `semanticAction` fails with `failureCategory: "selector-not-found"`, the wrapper may run one fresh session-scoped `snapshot -i` and add visible `Current snapshot ref fallback` plus `details.visibleRefFallback` when that snapshot contains exact role/name matches for the failed target. Non-fill matches can also add `try-current-visible-ref` / `try-current-visible-ref-N` next actions. The matcher is bounded to current snapshot refs and exact normalized role/name matches: role locators require `--name`, text-click falls back only to exact-name `button`/`link` refs, label-fill to exact-name `textbox`, and placeholder-fill to exact-name `searchbox`/`textbox`. It never fuzzy-matches names such as prefixes; when several exact refs match, each action carries safety copy telling agents to inspect the snapshot and choose only if unambiguous. For post-failure `fill` matches, `visibleRefFallback.candidates[].args` and `visibleRefFallback.target.text` are omitted so recovery details do not repeat the fill text.
|
|
252
313
|
|
|
253
|
-
If a compiled `semanticAction` fails with `failureCategory: "selector-not-found"`, visible content can also include an `Agent-browser candidate fallbacks` block when the wrapper has bounded role/name retries for that locator and action, and `details.nextActions` includes the normal `refresh-interactive-refs` snapshot step plus those entries. When `session` was provided, candidate retry args preserve the same `--session <session>` prefix. Today `buildSemanticActionCandidateActions` in `extensions/agent-browser/index.ts` only appends click candidates for `click` + `text` → `try-button-name-candidate` and `try-link-name-candidate`. Fill misses no longer emit `find … fill <text>` retry actions because those would repeat potentially sensitive text. Instead, when the same selector-miss snapshot finds exact current editable refs (`searchbox` or `textbox`), the wrapper emits `details.richInputRecovery`, visible `Rich input recovery`, and `focus-current-editable-ref` / `click-current-editable-ref` (numbered when ambiguous) next actions. Those actions carry only focus/click argv for the candidate ref; they do not copy fill text, press `Enter`, or submit. Use `keyboard
|
|
314
|
+
If a compiled `semanticAction` fails with `failureCategory: "selector-not-found"`, visible content can also include an `Agent-browser candidate fallbacks` block when the wrapper has bounded role/name retries for that locator and action, and `details.nextActions` includes the normal `refresh-interactive-refs` snapshot step plus those entries. When `session` was provided, candidate retry args preserve the same `--session <session>` prefix. Today `buildSemanticActionCandidateActions` in `extensions/agent-browser/index.ts` only appends click candidates for `click` + `text` → `try-button-name-candidate` and `try-link-name-candidate`. Fill misses no longer emit `find … fill <text>` retry actions because those would repeat potentially sensitive text. Instead, when the same selector-miss snapshot finds exact current editable refs (`searchbox` or `textbox`), the wrapper emits `details.richInputRecovery`, visible `Rich input recovery`, and `focus-current-editable-ref` / `click-current-editable-ref` (numbered when ambiguous) next actions. Those actions carry only focus/click argv for the candidate ref; they do not copy fill text, press `Enter`, or submit. Use `keyboard type` after focusing the right current ref when a framework-controlled editor needs real key events. `keyboard inserttext` is paste-like and can change the DOM value without updating application state; every successful `keyboard inserttext` result carries a visible warning to verify application state before saving. Submit only when the user flow explicitly calls for it. Candidate fallbacks are heuristics, not proof that an element exists; inspect the page when several controls could share the same name.
|
|
254
315
|
|
|
255
316
|
If a compiled `semanticAction` `find` action fails with `failureCategory: "stale-ref"`, `details.nextActions` includes `retry-semantic-action-after-stale-ref` with the same redacted compiled argv as `details.compiledSemanticAction` in `params.args` (any leading `--session` pair from `semanticAction.session`, then the `find` tokens). The wrapper appends that entry **after** any `refresh-interactive-refs` snapshot step from `buildAgentBrowserNextActions` in `extensions/agent-browser/lib/results/action-recommendations.ts` (see `extensions/agent-browser/index.ts` where `nextActions` is merged). That retry is only offered because the semantic target is stable and the stale-ref error proves the previous action did not execute; `select` shorthands with stale `@e…` selectors and direct stale `@e…` commands still return refresh guidance instead of an unsafe blind retry.
|
|
256
317
|
|
|
257
|
-
For direct page-scoped `@e…` refs, successful `snapshot` results record `details.refSnapshot` with the latest ref ids and page target for the session. A failed session `snapshot` whose upstream error says `No active page` clears that session’s prior ref snapshot and records `details.refSnapshotInvalidation.reason: "no-active-page"`; mutation-prone `@e…` preflight then fails with `failureCategory: "stale-ref"` until a later successful `snapshot -i` records fresh refs. Before
|
|
318
|
+
For direct page-scoped `@e…` refs, successful `snapshot` results record `details.refSnapshot` with the latest ref ids and page target for the session. A failed session `snapshot` whose upstream error says `No active page` clears that session’s prior ref snapshot and records `details.refSnapshotInvalidation.reason: "no-active-page"`; any upstream-executed `record start` attempt (direct or inside a batch, including one that fails with `Recording already active`, because upstream swaps to the fresh recording page before that check) or `record restart` with a URL operand clears it and records `details.refSnapshotInvalidation.reason: "page-transition"`; a plain `record restart <path>` keeps the current page and refs; mutation-prone `@e…` preflight then fails with `failureCategory: "stale-ref"` until a later successful `snapshot -i` records fresh refs. Before page-scoped ref commands such as `get`, `click`, `fill`, `check`, `select`, `download`, drag/upload/keyboard-style actions, upstream ref-resolving reads and captures (`is`, `screenshot`, `highlight`, `scroll`, `frame`, `diff`), or equivalent batch steps run, the wrapper rejects refs from an older page target, refs absent from the latest same-page snapshot, or refs from an invalidated snapshot state. Batch steps are scanned from the source upstream actually executes: raw batch argument strings exclusively when any exist (upstream filters only the exact `--bail` token, so `--bail=true` stays a raw command), stdin steps only otherwise, so `batch "click @e1"` is guarded and stdin refs are not falsely rejected when upstream would ignore that stdin. Tab-pinned batch rewrites dispatch those same upstream-effective steps and re-emit the caller's exact `--bail` token so a validated fail-fast batch stays fail-fast, so pinning cannot resurrect ignored stdin the guard never scanned, and artifact/screenshot preflights (parent-directory creation, recording lifecycle rules) also skip upstream-ignored stdin rows. Getter batches receive the same same-page freshness check so a recycled `@ref` cannot silently read a different control after an in-place rerender. Commands whose operands upstream never resolves as refs (`wait`, `a11y`, `find`) are not guarded, and `@e…`-looking values of value-taking flags other than `--selector`/`-s` (for example `wait --text @e1` or `diff screenshot --baseline @e1.png`) pass through as literals; boolean flags such as `--new-tab` or `--full` do not consume the following token, so a ref after them is a positional selector and stays guarded. A `batch` that times out or returns unparseable output after executing is treated conservatively: when its planned steps include a recording page swap, the wrapper still records the `page-transition` invalidation. This is a best-effort wrapper guard against upstream ref-number recycling after navigation; it does not prove the DOM stayed unchanged after the snapshot. Refresh with the session-aware `refresh-interactive-refs` next action before retrying.
|
|
258
319
|
|
|
259
320
|
Examples:
|
|
260
321
|
|
|
@@ -267,6 +328,8 @@ Examples:
|
|
|
267
328
|
{ "semanticAction": { "action": "click", "selector": "#submit" } }
|
|
268
329
|
{ "semanticAction": { "action": "select", "selector": "#flavor", "value": "chocolate" } }
|
|
269
330
|
{ "semanticAction": { "action": "select", "selector": "#multi", "values": ["dark", "compact"] } }
|
|
331
|
+
{ "semanticAction": { "action": "select", "locator": "role", "role": "combobox", "name": "Flavor", "value": "chocolate" } }
|
|
332
|
+
{ "semanticAction": { "action": "select", "locator": "label", "value": "Flavor", "values": ["chocolate"] } }
|
|
270
333
|
{ "semanticAction": { "action": "check", "locator": "label", "value": "Remember me" } }
|
|
271
334
|
{ "semanticAction": { "action": "click", "locator": "text", "value": "Close", "session": "named-browser" } }
|
|
272
335
|
```
|
|
@@ -274,7 +337,7 @@ Examples:
|
|
|
274
337
|
### `job`
|
|
275
338
|
|
|
276
339
|
- type: object with a non-empty `steps` array
|
|
277
|
-
- optional; mutually exclusive with `args`, `semanticAction`, `qa`, `sourceLookup`, `networkSourceLookup`, and `electron`
|
|
340
|
+
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `qa`, `sourceLookup`, `networkSourceLookup`, and `electron`
|
|
278
341
|
- top-level tool input only; do not nest `job` inside `batch` stdin
|
|
279
342
|
- constrained orchestration only: every step compiles to existing upstream `batch` argv and the compiled plan is echoed as `details.compiledJob`
|
|
280
343
|
- optional `failFast` boolean; defaults to `true`, compiling to upstream `batch --bail` so later mutating job steps do not run after an earlier required step fails. Set `failFast: false` only when you explicitly want upstream batch's continue-after-error behavior and every later step remains safe if an earlier navigation fails; otherwise the wrapper may require fail-fast behavior or split calls.
|
|
@@ -292,7 +355,7 @@ Examples:
|
|
|
292
355
|
- `snapshot` (compiled as `snapshot -i`; useful between mutation-prone steps before reusing current refs)
|
|
293
356
|
- `screenshot` with `path`
|
|
294
357
|
|
|
295
|
-
**Navigation assertions are explicit only.** `job` never treats a successful `click` (or a `select` / submit-style interaction that may navigate) as proof that the expected next page loaded. Top-level `click` may still surface optional `details.navigationSummary` or `pageChangeSummary` hints for operators, but compiled `job` / `batch` steps do **not** auto-insert `assertUrl` or `assertText` after clicks—there is no deterministic expected URL source without caller intent. Use `open.loadState` to wait for initial page readiness after an `open`; after any later navigation-prone step (link/submit clicks, checkout or form flows, tab-sensitive UI), add an explicit `assertUrl` with the exact destination URL or `*` / `**` glob-style pattern
|
|
358
|
+
**Navigation assertions are explicit only.** `job` never treats a successful `click` (or a `select` / submit-style interaction that may navigate) as proof that the expected next page loaded. Top-level `click` may still surface optional `details.navigationSummary` or `pageChangeSummary` hints for operators, but compiled `job` / `batch` steps do **not** auto-insert `assertUrl` or `assertText` after clicks—there is no deterministic expected URL source without caller intent. Use `open.loadState` to wait for initial page readiness after an `open`; after any later navigation-prone step (link/submit clicks, checkout or form flows, tab-sensitive UI), add an explicit `assertUrl` with the exact destination URL or a `*` / `**` glob-style pattern that does not already match the starting page, `assertText` for new on-page copy, or both, **before** screenshots or steps that assume the new page state. A broad URL pattern that already matches the current route is a precondition, not proof of navigation.
|
|
296
359
|
|
|
297
360
|
Example (static landing page):
|
|
298
361
|
|
|
@@ -338,17 +401,17 @@ On app pages that expose a native dropdown, add a `select` step such as `{ "acti
|
|
|
338
401
|
|
|
339
402
|
`assertUrl` compiles to upstream `wait --url <url-or-pattern>` for exact URLs and glob-style patterns, preserving query strings and literal `?` in exact URLs and delegating `*` / `**` matching to the current upstream matcher.
|
|
340
403
|
|
|
341
|
-
Use raw `args` plus `stdin` for upstream `batch` when a flow needs commands, flags, stdin forms, or failure policies outside this constrained schema. `job.failFast: false` keeps the constrained schema but removes `--bail` for continue-after-error diagnostics. If a later content step assumes navigation succeeded, keep the default fail-fast behavior or split the flow: the wrapper rejects non-bail batches when any failed transition could leave
|
|
404
|
+
Use raw `args` plus `stdin` for upstream `batch` when a flow needs commands, flags, stdin forms, or failure policies outside this constrained schema. `job.failFast: false` keeps the constrained schema but removes `--bail` for continue-after-error diagnostics. If a later content step assumes navigation succeeded, keep the default fail-fast behavior or split the flow: the wrapper rejects non-bail batches when any failed transition could leave an unverified page active.
|
|
342
405
|
|
|
343
406
|
Because `job` still executes as upstream `batch` with generated stdin, the same wrapper page-scoped `@e…` preflight applies: if you pass `@refs` in `click`/`fill`/`select` selectors after an `open`, non-form `click`, or another step that can navigate or mutate the page, split the work across tool calls or switch to raw `batch` and insert your own `snapshot -i` rows between steps—the constrained `job` vocabulary emits a `snapshot` step only when you include `{ action: "snapshot" }` explicitly. Multiple same-snapshot `fill @e…` rows may run before the first click/submit-style step. Raw `args:["batch"]` stdin can also batch native form-control rows (`check`/`uncheck` checkbox or radio refs, checkbox/radio `click`/`tap` refs, and `select` combobox refs) before that click.
|
|
344
407
|
|
|
345
408
|
### `qa`
|
|
346
409
|
|
|
347
410
|
- type: object with either required `url` (normal URL-opening QA) or `attached: true` (current attached-session QA)
|
|
348
|
-
- optional; mutually exclusive with `args`, `semanticAction`, `job`, `sourceLookup`, `networkSourceLookup`, and `electron`
|
|
411
|
+
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `sourceLookup`, `networkSourceLookup`, and `electron`
|
|
349
412
|
- lightweight preset built on the same batch compiler path as `job`, using `batch --bail` so missing readiness/text/selector assertions stop before slower diagnostics can burn the wrapper watchdog
|
|
350
|
-
- URL form: clears enabled diagnostic buffers first
|
|
351
|
-
- attached form: `qa: { attached: true, expectedText?, expectedSelector?, screenshotPath?, checkNetwork?, checkConsole?, checkErrors?, loadState? }` runs the same waits, optional assertions, diagnostics, and screenshot against the current attached managed session without opening a URL. It rejects `url` and cannot be used with `sessionMode: "fresh"`; attach first with `electron.launch` or raw `args: ["connect", "<port-or-url>"]`, then run `qa.attached`. Before spawning the diagnostic batch, the wrapper preflights the attached session: `get url` must succeed and return
|
|
413
|
+
- URL form: clears enabled network/console diagnostic buffers first and snapshots any page-error residue after `errors --clear` (upstream 0.33.2 does not reliably clear page errors), then opens `url`, waits with `wait --load <state>` using the resolved `loadState`, adds a bounded 150 ms diagnostic settle when console or page-error checks are enabled so immediate post-load callbacks can report, optionally asserts `expectedText` (string or string array, compiled to bounded visible-text `wait --fn … --timeout 5000` predicates after load) and/or `expectedSelector` (each may be omitted for a load-plus-diagnostics-only smoke), then runs enabled diagnostics: `network requests`, `console`, and `errors` only if preceding batch steps pass. Successful reset-step rows are labeled as reset output. Only unchanged page-error residue left after the clear is ignored; failed reset commands still fail the batch, and a matching error that reappears after a successful clear or any other post-open diagnostic row still counts normally.
|
|
414
|
+
- attached form: `qa: { attached: true, expectedText?, expectedSelector?, screenshotPath?, checkNetwork?, checkConsole?, checkErrors?, loadState? }` runs the same waits (including the bounded diagnostic settle when needed), optional assertions, diagnostics, and screenshot against the current attached managed session without opening a URL. It rejects `url` and cannot be used with `sessionMode: "fresh"`; attach first with `electron.launch` or raw `args: ["connect", "<port-or-url>"]`, then run `qa.attached`. Before spawning the diagnostic batch, the wrapper preflights the attached session: `get url` must succeed and return a non-empty page URL. Missing URLs and read failures fail fast with `failureCategory: "validation-error"`, `details.validationError`, and recovery `nextActions` such as `list-tabs-before-qa-attached` and `snapshot-before-qa-attached` instead of running the full QA batch. `file:`, custom-scheme, and other attached targets are accepted. Attached QA does **not** run `network requests --clear`, `console --clear`, or `errors --clear`; `details.compiledQaPreset.checks.diagnosticsResetAtStart` is `false`. Visible text warns that existing diagnostic buffers were preserved only when `checkNetwork`, `checkConsole`, or `checkErrors` is enabled, and those diagnostics may include events from before the QA check.
|
|
352
415
|
- `loadState` is optional and must be `domcontentloaded`, `load`, or `networkidle`; it defaults to `domcontentloaded` so analytics-heavy or long-polling pages do not hang routine QA. Use `networkidle` only when the site is expected to go fully quiet.
|
|
353
416
|
- `checkNetwork`, `checkConsole`, and `checkErrors` default to `true` for URL-opening QA; for `qa.attached` they default to `false` because preserved upstream buffers may predate the current check. Set a field to `true` on `qa.attached` to opt into preserved-buffer diagnostics.
|
|
354
417
|
- optional `screenshotPath` adds an evidence screenshot step
|
|
@@ -369,7 +432,7 @@ Use custom `job` or raw `batch` for QA flows that need custom commands, flags, a
|
|
|
369
432
|
Workflow-oriented public guide: [`ELECTRON.md`](ELECTRON.md). This section remains the canonical field contract; the guide covers when and how to use these actions in practice.
|
|
370
433
|
|
|
371
434
|
- type: object with required `action`
|
|
372
|
-
- optional; mutually exclusive with `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, and `networkSourceLookup`
|
|
435
|
+
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, and `networkSourceLookup`
|
|
373
436
|
- top-level wrapper shorthand for Electron desktop apps; do not nest it inside `batch` stdin
|
|
374
437
|
- `stdin` is rejected with `electron`; host-only actions manage their own local work and `launch` manages its own upstream `connect`
|
|
375
438
|
- supported actions: `list`, `launch`, `status`, `cleanup`, and `probe`
|
|
@@ -381,8 +444,8 @@ Action schemas:
|
|
|
381
444
|
| `list` | `query?`, `maxResults?` | Scans supported platform app locations for Electron evidence and returns bounded app metadata in `details.electron.apps`. Likely-sensitive app annotations are advisory metadata only. Does not spawn upstream `agent-browser`; list output also warns that later wrapper launches are isolated and will not read existing signed-in desktop state. |
|
|
382
445
|
| `launch` | exactly one of `appPath`, `appName`, `bundleId`, or `executablePath`; optional `appArgs`, `handoff`, `targetType`, `timeoutMs`, `allow`, `deny` | Resolves and verifies an Electron target, launches it with a wrapper-owned isolated profile and OS-chosen CDP port, attaches through upstream `connect` using `sessionMode: "fresh"`, and records `details.electron.launch`. It does not reuse the app's normal signed-in profile or attach to an already-running authenticated app; when signed-in local app state is the goal, use a host debug-port launch plus raw `connect` instead. Pi cancellation before spawn returns `failureCategory: "aborted"` without opening the app; cancellation during readiness polling or post-attach handoff closes the managed session, stops the process, and removes its isolated profile. If any wrapper validation, policy, or live-URL handoff check fails after launch, the wrapper immediately cleans the new process/profile and retains a partial record only when cleanup could not finish. |
|
|
383
446
|
| `status` | optional `launchId` or `all`, optional `timeoutMs` | Inspects wrapper-tracked launches, debug-port liveness, and current CDP targets without mutating the app. With neither `launchId` nor `all`, selects the single active wrapper launch when unambiguous. Runtime-owned launches remain visible by `launchId` after a Pi `session_tree` branch switch. Current branch-visible launches survive `/reload`; off-branch owned launches are cleaned on reload. |
|
|
384
|
-
| `cleanup` | optional `launchId` or `all`, optional `timeoutMs` | Closes the tracked upstream session when present, stops only the wrapper-tracked process, verifies debug-port shutdown, removes the wrapper-created `userDataDir`, and marks records cleaned or partial. Cleanup is serialized with managed-session browser work, and a successful managed-session close step clears live/restore
|
|
385
|
-
| `probe` | optional `launchId`, optional `timeoutMs` | Runs bounded current-session or launch-scoped state reads (`get url`, then `get title`, focused-element `eval --stdin`, `tab list`, compact `snapshot -i`) and reports `details.electron.probe`; current-managed probes also persist top-level `details.namespace`, `sessionTabTarget`, and `refSnapshot` so reload/branch replay keeps the same namespaced page identity. Without `launchId`, requires an active attached managed session; with `launchId`, it resolves the tracked launch session, including runtime-owned off-branch launch records, and can report mismatch guidance. If every read fails, the tool fails as `upstream-error`;
|
|
447
|
+
| `cleanup` | optional `launchId` or `all`, optional `timeoutMs` | Closes the tracked upstream session when present, stops only the wrapper-tracked process, verifies debug-port shutdown, removes the wrapper-created `userDataDir`, and marks records cleaned or partial. Cleanup is serialized with managed-session and artifact-lifecycle browser work, reuses the launch record's canonical namespace/session identity, and a successful managed-session close step clears live/restore and recording-reservation state even when host process/profile cleanup remains partial. |
|
|
448
|
+
| `probe` | optional `launchId`, optional `timeoutMs` | Runs bounded current-session or launch-scoped state reads (`get url`, then `get title`, focused-element `eval --stdin`, `tab list`, compact `snapshot -i`) and reports `details.electron.probe`; current-managed probes also persist top-level `details.namespace`, `sessionTabTarget`, and `refSnapshot` so reload/branch replay keeps the same namespaced page identity. Without `launchId`, requires an active attached managed session; with `launchId`, it resolves the tracked launch session, including runtime-owned off-branch launch records, and can report mismatch guidance. If every read fails, the tool fails as `upstream-error`; the live URL is rechecked before any title/content helper. |
|
|
386
449
|
|
|
387
450
|
Validation and defaults:
|
|
388
451
|
|
|
@@ -504,7 +567,7 @@ For an app you launched manually with remote debugging enabled, skip `electron.c
|
|
|
504
567
|
### `sourceLookup`
|
|
505
568
|
|
|
506
569
|
- type: object with at least one of `selector`, `reactFiberId`, or `componentName`
|
|
507
|
-
- optional; mutually exclusive with `args`, `semanticAction`, `job`, `qa`, `networkSourceLookup`, and `electron`
|
|
570
|
+
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `qa`, `networkSourceLookup`, and `electron`
|
|
508
571
|
- **EXPERIMENTAL — candidates only:** opt-in helper for local app debugging; it reports candidate source locations with confidence and evidence instead of claiming a guaranteed DOM-to-file mapping. Do not treat output as authoritative file ownership or edit targets without verification.
|
|
509
572
|
- compiles to existing upstream `batch` commands only:
|
|
510
573
|
- `selector` adds `is visible <selector>` and, unless `includeDomHints: false`, adds `get html <selector>` for source-like DOM attributes (`data-source-file`, `data-file`, `data-component-file`, `data-source`, plus optional `data-source-line` / `data-line` and `data-source-column` / `data-column`) and for `.ts`/`.tsx`/`.js`/`.jsx` paths embedded in HTML text
|
|
@@ -529,9 +592,9 @@ Use raw `args` for direct upstream React inspection when you already know the ex
|
|
|
529
592
|
### `networkSourceLookup`
|
|
530
593
|
|
|
531
594
|
- type: object with at least one of `requestId`, `filter`, or `url`, plus optional `maxWorkspaceFiles`
|
|
532
|
-
- optional; mutually exclusive with `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, and `electron`
|
|
595
|
+
- optional; mutually exclusive with `script`, `args`, `semanticAction`, `job`, `qa`, `sourceLookup`, and `electron`
|
|
533
596
|
- **EXPERIMENTAL — candidates only:** failed-request source-hint helper; it reports failed network requests and candidate source hints with evidence instead of assigning blame or proving root cause
|
|
534
|
-
- compiles to existing upstream `batch` commands only: `network request <requestId>` when provided plus `network requests` with `--filter <filter-or-url>` when a filter or URL is provided (if both are set, `filter` wins; when only `url` is set, it becomes the `--filter` argument); optional `namespace` / `session` prepends `--namespace <name>` / `--session <name>` before that generated `batch`
|
|
597
|
+
- compiles to existing upstream `batch` commands only: `network request <requestId>` when provided plus `network requests` with `--filter <filter-or-url>` when a filter or URL is provided (if both are set, `filter` wins; when only `url` is set, it becomes the `--filter` argument); optional `namespace` / `session` prepends `--namespace <name>` / `--session <name>` before that generated `batch`; exact `namespace: ""` selects the default namespace and remains explicit so an ambient namespace cannot redirect the lookup
|
|
535
598
|
- detects failed requests from `status >= 400`, `failed: true`, or an `error` field
|
|
536
599
|
- candidate sources come from source-like initiator/stack metadata in upstream network results and bounded local workspace search for URL/path literals under the Pi session cwd
|
|
537
600
|
- optional `maxWorkspaceFiles` defaults to 2000 and cannot exceed 5000; workspace-search candidates are capped at ten
|
|
@@ -577,8 +640,8 @@ For `eval --stdin`, put the script in the top-level `stdin` field. The wrapper n
|
|
|
577
640
|
- type: `string`
|
|
578
641
|
- optional; can be used with any successful browser result path, most often `eval --stdin`, `get text`, `get html`, `snapshot`, or diagnostic captures whose result should become a durable local file
|
|
579
642
|
- workspace-relative paths resolve against the Pi session cwd; absolute paths are used as-is; a leading `@` is stripped for consistency with Pi file arguments
|
|
580
|
-
- after the upstream command completes, the wrapper writes `details.data` when present, otherwise the model-facing text content; objects/arrays are written as pretty JSON with a trailing newline and strings are written as-is
|
|
581
|
-
- `outputPath` must not resolve to the same file as a screenshot, download, recording, or other browser artifact produced by that result, including
|
|
643
|
+
- after the upstream command completes, the wrapper writes `details.data` when present, otherwise the model-facing text content; objects/arrays are written as pretty JSON with a trailing newline and strings are written as-is. If a direct `details.data` value, a `batch` row's `result`, or the whole batch is compacted, the wrapper instead reads and serializes each full command-redacted pre-compaction payload only from the corresponding live `spill` entry in its own `details.artifactManifest` (`persistent-session` or `process-temp`). It never writes a compact metadata object as a substitute; any missing, evicted, malformed, or untrusted required spill makes the result fail and leaves `outputPath` unwritten.
|
|
644
|
+
- `outputPath` must not resolve to the same file as a screenshot, download, recording, or other browser artifact produced by that result, including dangling/existing symlink, hardlink, Unicode-fold, and platform-case aliases. When both destinations are known before launch, preflight rejects the call as `validation-error` without browser activity; if an alias becomes apparent only from the completed result, the writer preserves the browser artifact, rejects the result-data write, and reports `details.outputFile.status: "failed"`
|
|
582
645
|
- successful writes append `details.outputFile = { status: "saved", path, absolutePath, source, bytes }`; they also append a visible `Output file: …` line except when the caller explicitly passed upstream `--json`, where parseable JSON content is preserved and the saved-file notice lives only in `details.outputFile`. Write failures append `details.outputFile.status: "failed"`, remove success-only category fields, and mark the tool result failed without rolling back browser session state.
|
|
583
646
|
|
|
584
647
|
Example:
|
|
@@ -594,7 +657,7 @@ Example:
|
|
|
594
657
|
- managed-session daemon-policy inspection has its own fixed budget of up to 35 seconds before that process and is intentionally not shortened by `timeoutMs`, so a busy valid daemon does not become an unsafe false negative
|
|
595
658
|
- use for long opens, large snapshots, paced `job` typing, or captures that legitimately need more than the default watchdog
|
|
596
659
|
- explicit long `wait` steps are forwarded to upstream; top-level `timeoutMs` only controls the wrapper subprocess watchdog and should be at least the wait duration plus a small grace window when supplied manually
|
|
597
|
-
- when the watchdog fires, `details.timeoutMs`, `details.timedOut`, and possibly `details.timeoutPartialProgress` explain what was recovered
|
|
660
|
+
- when the watchdog fires, `details.timeoutMs`, `details.timedOut`, and possibly `details.timeoutPartialProgress` explain what was recovered. If the page target is unknown, standalone snapshot suggestions are removed and one session-scoped `verify-page-target-after-timeout` fail-fast batch (`get url`, then `snapshot -i`) appears in visible failure text and `details.nextActions`, so the returned recovery is executable under the same page-target guard.
|
|
598
661
|
|
|
599
662
|
Example:
|
|
600
663
|
|
|
@@ -611,10 +674,11 @@ Example:
|
|
|
611
674
|
Behavior:
|
|
612
675
|
- if `args` already include `--session` (including argv compiled from optional `semanticAction.session`), upstream session choice wins
|
|
613
676
|
- `"auto"` prepends the current extension-managed active session when appropriate
|
|
614
|
-
- `"fresh"` rotates that managed session to a fresh upstream launch so startup-scoped flags like `--profile`, `--executable-path`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--headed`, `--enable`, `-p` / `--provider`, or iOS `--device` apply and later default calls follow the new browser; `--idle-timeout` must equal the Pi process's configured managed idle timeout or the wrapper rejects it with restart guidance
|
|
677
|
+
- `"fresh"` rotates that managed session to a fresh upstream launch so startup-scoped flags like `--profile`, `--executable-path`, `--ca-cert`, `--no-ca-cert`, `--webgpu`, `--no-webmcp`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--headed`, `--enable`, `-p` / `--provider`, or iOS `--device` apply and later default calls follow the new browser; `--idle-timeout` must equal the Pi process's configured managed idle timeout or the wrapper rejects it with restart guidance
|
|
615
678
|
- upstream `--webgpu` is a launch-scoped optional boolean: both enabled and explicit `false` values require a fresh managed launch once an implicit session exists; enabled WebGPU is local-launch-only and upstream rejects combinations with CDP, auto-connect, or providers
|
|
679
|
+
- upstream `--no-webmcp` is also launch-scoped for bare/`true` and explicit `false` values because it selects whether locally managed Chrome enables the experimental feature
|
|
616
680
|
- upstream restore sessions may periodically save cookies/localStorage while open; an explicit `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` passes through unchanged when a daemon launches (`30000` upstream default, `0` disables periodic saves but keeps native close saves), while wrapper-owned headed launches default it to `0` when unset to avoid upstream 0.33.2's visible temporary collector tabs; direct window close can lose newer state because headed browsers are exempt from idle shutdown, the effective interval persists across resume and changing it in either direction on a running wrapper-owned headed daemon requires close plus a fresh launch, and those restore files remain upstream-owned rather than wrapper artifacts
|
|
617
|
-
- sessionless paths skip that injection even under `"auto"`: plain-text
|
|
681
|
+
- sessionless paths skip that injection even under `"auto"`: plain-text help/version, read-only skills, local auth/profile/setup commands, `session list`, and syntactically local state lifecycle operations keep `effectiveArgs` free of the implicit managed `--session` unless the caller supplied one. All upstream rows and supported targets remain available. Browser-backed or context-dependent commands such as `auth login` and `state save/load` keep normal managed-session injection when the caller did not choose an explicit session (`extensions/agent-browser/lib/command-policy.ts`, `needsManagedSession`; `extensions/agent-browser/lib/runtime.ts`, `buildExecutionPlan`)
|
|
618
682
|
|
|
619
683
|
Recommended use:
|
|
620
684
|
- use `"auto"` for the common browse/snapshot/click flow inside one `pi` session
|
|
@@ -655,12 +719,12 @@ Primary content should be:
|
|
|
655
719
|
- an image attachment when relevant
|
|
656
720
|
- browser-aware compacting for oversized snapshots so the model gets a concise actionable view before raw page noise
|
|
657
721
|
- compact snapshots should be main-content-first: prefer the primary content block and nearby sections over top-of-page chrome, ads, or unrelated sidebars when those can be distinguished from the snapshot tree. They are DOM/signal-prioritized, not guaranteed viewport-first after scroll; compact output may include `details.snapshotCompaction.viewportOrdering: "dom-signal-prioritized"` and a visible viewport note when viewport context matters.
|
|
658
|
-
- when compacting hides actionable controls, snapshot output should add an `Omitted high-value controls` section for bounded editable/searchbox/textbox/combobox controls, named tab/surface controls, primary action buttons,
|
|
659
|
-
- wrapper-side `snapshot -i --search <text>` and `snapshot -i --filter role=<role>` filters should strip those wrapper-only flags before upstream spawn, preserve the full latest ref map in `details.refSnapshot`, and render matching direct refs plus surrounding snapshot context with `details.snapshotFilter` counts so dense-page agents can find controls without opening raw spill files; the visible summary should distinguish direct ref matches from contextual
|
|
722
|
+
- when compacting hides actionable controls, snapshot output should add an `Omitted high-value controls` section for bounded editable/searchbox/textbox/combobox controls, named tab/surface controls, primary action buttons, named action links such as row/navigation links and repository-style result links, and other useful controls such as checkboxes, radios, options, and menuitems that were not already shown in key refs
|
|
723
|
+
- wrapper-side `snapshot -i --search <text>` and `snapshot -i --filter role=<role>` filters should strip those wrapper-only flags before upstream spawn, preserve the full latest ref map in `details.refSnapshot`, and render matching direct refs plus surrounding snapshot context with `details.snapshotFilter` counts so dense-page agents can find controls without opening raw spill files; search should also run one bounded read-only rendered-DOM probe across the full document so visible below-fold warnings and accessible labels omitted from the accessibility snapshot remain discoverable while hidden nodes stay excluded; the visible summary should distinguish direct ref matches from rendered/contextual matches to avoid apparent count mismatches; wrapper-side `--viewport` should also strip before upstream spawn, run one read-only viewport/scroll probe, and report `details.snapshotViewport`; wrapper-side `--diff` should strip before upstream spawn and report `details.snapshotDiff` against the previous wrapper-tracked ref map for that session
|
|
660
724
|
|
|
661
725
|
Examples:
|
|
662
726
|
- small `snapshot` results should include the actual snapshot text
|
|
663
|
-
- oversized `snapshot` results should switch to a compact view that preserves the primary content, nearby sections, a trimmed set of high-value refs, and a separate bounded list of omitted high-value controls when dense pages or desktop host screens would otherwise hide editable inputs, named surfaces/tabs, or primary action buttons, while exposing the full
|
|
727
|
+
- oversized `snapshot` results should switch to a compact view that preserves the primary content, nearby sections, a trimmed set of high-value refs, and a separate bounded list of omitted high-value controls when dense pages or desktop host screens would otherwise hide editable inputs, named surfaces/tabs, or primary action buttons, while exposing the full redacted snapshot path directly in the rendered tool text and via `details.fullOutputPath`
|
|
664
728
|
- successful navigation actions like `click`, `back`, `forward`, and `reload` should include a lightweight post-action title/url summary when the wrapper can address the active session
|
|
665
729
|
- `tab list` should include a readable tab summary
|
|
666
730
|
- `screenshot` should include the saved-path summary plus the inline image attachment when available
|
|
@@ -693,16 +757,18 @@ Recommended details:
|
|
|
693
757
|
Stable category fields are part of the machine-readable contract:
|
|
694
758
|
|
|
695
759
|
- `resultCategory`: always either `"success"` or `"failure"`.
|
|
696
|
-
- `successCategory`: present on successful results. Current values are `"completed"`, `"artifact-saved"`, `"artifact-unverified"`, and `"inspection"`. `artifact-
|
|
697
|
-
- `failureCategory`: present on failed results. Current values are `"aborted"`, `"artifact-missing"`, `"cleanup-failed"`, `"confirmation-required"`, `"download-not-verified"`, `"missing-binary"`, `"parse-failure"`, `"policy-blocked"`, `"qa-failure"`, `"selector-not-found"`, `"selector-unsupported"`, `"stale-ref"`, `"tab-drift"`, `"timeout"`, `"upstream-error"`, and `"validation-error"`. `artifact-missing` means upstream reported a saved/completed artifact path, but the wrapper verified the non-pending file is absent and failed closed.
|
|
760
|
+
- `successCategory`: present on successful results. Current values are `"completed"`, `"artifact-pending"`, `"artifact-saved"`, `"artifact-unverified"`, and `"inspection"`. `artifact-pending` means a recording started but its file is not expected until `record stop`; use the exact `stop-pending-recording` next action and verify the resulting file. For `record start`, visible output also states that upstream switches to a fresh active page for video capture, prior in-page DOM and JavaScript state does not carry over, and the next interaction should follow a fresh snapshot; the wrapper also invalidates the session’s prior ref snapshot (`refSnapshotInvalidation.reason: "page-transition"`) so old `@e…` refs fail as `stale-ref` until a fresh `snapshot -i`; that invalidation is attempt-scoped (a start that fails as `Recording already active` still swapped the page) and also covers `record restart` with a URL operand, while a plain `record restart` keeps the page and refs. Failed results also retain that action whenever their artifact rollup still contains a pending recording. `artifact-unverified` means upstream reported success but the merged `artifactVerification` summary still has unverified non-missing rows; inspect its counts and per-entry `state` / optional `limitation` before treating artifacts as durable evidence.
|
|
761
|
+
- `failureCategory`: present on failed results. Current values are `"aborted"`, `"artifact-missing"`, `"cleanup-failed"`, `"confirmation-required"`, `"download-not-verified"`, `"missing-binary"`, `"parse-failure"`, `"policy-blocked"`, `"qa-failure"`, `"script-error"`, `"selector-not-found"`, `"selector-unsupported"`, `"stale-ref"`, `"tab-drift"`, `"tab-gone"`, `"timeout"`, `"upstream-error"`, and `"validation-error"`. `artifact-missing` means upstream reported a saved/completed artifact path, but the wrapper verified the non-pending file is absent and failed closed.
|
|
762
|
+
|
|
763
|
+
For `script`, the top-level category describes the whole orchestration and cleanup, not the last inner call. `details.scriptRun` reports `callCount`, `successfulCallCount`, `failedCallCount`, `preDispatchRejectedCallCount`, `emitCount`, and timeout/abort flags when applicable. `script-error` means the caller's script source threw or rejected; browser subprocess and wrapper protocol failures remain `upstream-error` unless a more specific category applies. `details.scriptSteps[]` preserves bounded redacted per-call category/summary rows rather than replaying full inner tool results. `details.scriptSession` reports the exact isolated-session cleanup lease state only after the first accepted inner call; successful no-browser scripts omit it. A cleanup failure always wins over an otherwise successful or failed script so the leaked browser identity is not hidden.
|
|
698
764
|
|
|
699
765
|
These categories are intentionally bounded and stable so agents can branch on them instead of parsing prose. They do not replace raw diagnostics: `details.error`, `details.stderr`, `details.parseError`, `details.validationError`, and visible content still preserve the specific upstream or wrapper message after normal redaction.
|
|
700
766
|
|
|
701
|
-
For argv-supplied `--allowed-domains`, the wrapper treats domain containment as launch-scoped
|
|
767
|
+
For argv-supplied `--allowed-domains`, the wrapper treats domain containment as launch-scoped. Upstream owns request, worker, popup, and WebRTC containment plus incompatible-mode rejection; the wrapper passes the setting and upstream result through unchanged.
|
|
702
768
|
|
|
703
769
|
Real Pi custom tools only mark a tool result failed when the tool throws during `execute`; returned `isError` fields are not authoritative. The extension therefore also registers a `tool_result` handler that treats any `agent_browser` result with `details.resultCategory: "failure"` as a real Pi tool error. For normal prose output, it appends `Result category: failure; failureCategory: …; Pi tool isError: true.` to model-visible text. For caller-requested `--json` output, it only patches `isError` and preserves visible parseable JSON content unchanged. The TUI renderer also repeats that category line at the top of failed rendered results so collapsed failed rows keep the outcome visible. The hook treats `--json` as requested when echoed `details.args` or the original tool `input.args` includes that flag; it skips appending the prose notice when any non-empty text content item is parseable JSON, even if other text items are not parseable. Invalid or non-JSON text still gets the visible prose notice. Implementation: `buildAgentBrowserToolResultPatch` in `extensions/agent-browser/lib/pi-tool-rendering.ts`; `extensions/agent-browser/index.ts` registers the handler. This keeps Pi transcript semantics aligned with the machine-readable result contract, including wrapper-side reclassifications such as `qa-failure` after an upstream-successful batch and `artifact-missing` after an upstream-successful artifact command whose requested file is absent.
|
|
704
770
|
|
|
705
|
-
For `batch`, top-level `details` still carries `resultCategory` plus `successCategory` or `failureCategory` for the **aggregate** tool outcome: if any step fails, the overall result is a failure (`resultCategory: "failure"`) even when later steps succeed—inspect `batchSteps[]` for per-step outcomes. Each `batchSteps[]` entry includes its own `resultCategory` and either `successCategory` or `failureCategory` for that step. `batchFailure.failedStep` duplicates the first failing step’s details, including its `failureCategory
|
|
771
|
+
For `batch`, top-level `details` still carries `resultCategory` plus `successCategory` or `failureCategory` for the **aggregate** tool outcome: if any step fails, the overall result is a failure (`resultCategory: "failure"`) even when later steps succeed—inspect `batchSteps[]` for per-step outcomes. Each `batchSteps[]` entry includes its own `resultCategory` and either `successCategory` or `failureCategory` for that step. When upstream reports it, successful and failed rows also expose a dedicated `lifecycle` field containing only the bounded `effectiveLaunch.browserLaunched` boolean; on failed rows this preserves launch evidence even though the ordinary result payload is omitted. Live state and transcript replay use that evidence to distinguish a terminal nested close from a post-close browser launch. `batchFailure.failedStep` duplicates the first failing step’s details, including its `failureCategory`, bounded lifecycle evidence, and any `nextActions`.
|
|
706
772
|
|
|
707
773
|
Top-level `details.data` on `batch` is a compact per-step roll-up (not a verbatim replay of raw upstream batch JSON): each element is `{ success, command, result? | error? }` where `command` is argv-redacted the same way as echoed invocation args (including `clipboard write` text, `cookies set` cookie values, `storage local|session set` values, and other sensitive flags/positionals), `result` is the presentation-layer data for that step after the same structured redaction as non-batch commands, and `error` is failure text with clipboard-write/cookie/storage/password literals stripped when those values appeared in argv. Prefer `batchSteps[]` for full per-step `details` (artifacts, categories, spill paths); use the roll-up when you only need a redacted matrix of what ran. If a large batch/job/qa result is compacted and spilled, the inline compacted text still includes bounded failed-step context (first failing step, failure category, failure detail, and any failed-step spill path) before the preview and top-level `Full output path:`.
|
|
708
774
|
|
|
@@ -711,11 +777,11 @@ Top-level `details.data` on `batch` is a compact per-step roll-up (not a verbati
|
|
|
711
777
|
Ref preflight details (command taxonomy in `extensions/agent-browser/lib/command-taxonomy.ts`, orchestration in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`):
|
|
712
778
|
|
|
713
779
|
- **URL alignment:** `refSnapshot.target.url` and the session’s current tab URL are compared via `targetsMatch` / `normalizeComparableUrl` in `extensions/agent-browser/index.ts`: values are trimmed, parsed as URLs when possible, compared **after dropping the `#fragment`**, and the query string remains significant. If either side lacks a `url`, `targetsMatch` treats the pair as matching so early-session calls are not blocked.
|
|
714
|
-
- **Batch stdin ordering:** user `batch` JSON is scanned in order. Any step whose first token satisfies `isRefInvalidatingBatchCommand` sets a latch that blocks later steps whose first token satisfies `isRefGuardedCommand` and that mention `@e…` refs, except for same-snapshot native form-control steps whose current snapshot role metadata identifies all refs as safe controls (`check`/`uncheck` or direct `click`/`tap` on checkbox or radio refs, and `select` on combobox refs). A step whose first token is `snapshot` clears that latch for subsequent steps (pre-spawn intent only; it does not wait for upstream success). These predicates read explicit command capability flags from `command-taxonomy.ts`: navigation/mutation verbs such as `open` / `goto`, `reload`, non-form `click`, and related upstream commands have `invalidatesBatchRefs
|
|
780
|
+
- **Batch stdin ordering:** user `batch` JSON is scanned in order. Any step whose first token satisfies `isRefInvalidatingBatchCommand` sets a latch that blocks later steps whose first token satisfies `isRefGuardedCommand` and that mention `@e…` refs, except for same-snapshot native form-control steps whose current snapshot role metadata identifies all refs as safe controls (`check`/`uncheck` or direct `click`/`tap` on checkbox or radio refs, and `select` on combobox refs). A step whose first token is `snapshot` clears that latch for subsequent steps (pre-spawn intent only; it does not wait for upstream success). These predicates read explicit command capability flags from `command-taxonomy.ts`: navigation/mutation verbs such as `open` / `goto`, `reload`, non-form `click`, and related upstream commands have `invalidatesBatchRefs`, and `record start` steps (any outcome), `record restart` steps with a URL operand, plus WebMCP `invoke` / `result` / `cancel` steps also set the latch because upstream swaps or navigates the active page; same-snapshot `fill` rows and the role-checked native form-control rows stay guarded against missing/stale refs but do not set the latch, allowing ordinary form batches before a final click/submit step. Direct `click`/`tap @e…` is only treated as a safe form-control row when every ref in that step is a latest-snapshot checkbox or radio; other click/tap refs remain invalidating. Ref-guarded commands accept page-scoped refs for interaction (`click`, `fill`, `download`, `scrollintoview` / `scrollinto`, and others centralized in the command taxonomy). Changing either capability requires updating this contract, [`docs/SUPPORT_MATRIX.md`](SUPPORT_MATRIX.md) `RQ-0072`/`RQ-0087` notes, README and command-reference pitfalls, and `test/agent-browser.extension-validation.test.ts`.
|
|
715
781
|
|
|
716
|
-
**Presentation redaction (implementation map):** Successful non-`batch` tool calls and each successful `batchSteps[]` row run upstream `data` through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation/diagnostics.ts`: `cookies` still walk objects/arrays and replace case-insensitive `value` keys with `"[REDACTED]"`; `storage` redacts values when the key or value looks credential-like (token, cookie, auth, secret, JWT, bearer/basic credential, high-entropy token-like string, or nested sensitive JSON) but keeps low-risk primitive QA values such as booleans, numbers, and short strings visible. Redacted storage entries add `valueRedacted` plus `valueRedactionReason` in `details.data`; diagnostic formatters mirror the same decision. Every other command’s payload is recursively scrubbed with `redactStructuredPresentationValue`, which redacts known sensitive key names and applies string-level sensitivity heuristics so network, diff, trace/profiler, stream, dashboard, chat, and other structured results do not echo bearer tokens, proxy credentials, or similar fields verbatim into `details.data`. Echoed `command` arrays in `details` and in batch roll-ups use `redactInvocationArgs` from `extensions/agent-browser/lib/runtime.ts` to mask trailing values for sensitive global flags (including `--body`, `--headers`, `--password`, and `--proxy`), preserve the special positional rules for `cookies set`, `storage local|session set`, and `set credentials`, and scrub other argv tokens for URLs and inline secrets. Failed batch steps additionally run `redactExactValues` on structured step errors so literals taken from that step’s argv (cookie value, storage set value, `--password` / `--password=` tokens) cannot reappear inside formatted error blobs.
|
|
782
|
+
**Presentation redaction (implementation map):** Successful non-`batch` tool calls and each successful `batchSteps[]` row run upstream `data` through `redactPresentationData` in `extensions/agent-browser/lib/results/presentation/diagnostics.ts`: `cookies` still walk objects/arrays and replace case-insensitive `value` keys with `"[REDACTED]"`; `storage` redacts values when the key or value looks credential-like (token, cookie, auth, secret, JWT, bearer/basic credential, high-entropy token-like string, or nested sensitive JSON) but keeps low-risk primitive QA values such as booleans, numbers, and short strings visible. Redacted storage entries add `valueRedacted` plus `valueRedactionReason` in `details.data`; diagnostic formatters mirror the same decision. Every other command’s payload is recursively scrubbed with `redactStructuredPresentationValue`, which redacts known sensitive key names and applies string-level sensitivity heuristics so network, diff, trace/profiler, stream, dashboard, chat, and other structured results do not echo bearer tokens, proxy credentials, or similar fields verbatim into `details.data`. Echoed `command` arrays in `details` and in batch roll-ups use `redactInvocationArgs` from `extensions/agent-browser/lib/runtime.ts` to mask trailing values for sensitive global flags (including `--body`, `--headers`, `--password`, and `--proxy`), preserve the special positional rules for `cookies set`, `storage local|session set`, and `set credentials`, and scrub other argv tokens for URLs and inline secrets. Failed batch steps additionally run `redactExactValues` on structured step errors so literals taken from that step’s argv (cookie value, storage set value, `--password` / `--password=` tokens) cannot reappear inside formatted error blobs. When the full batch is large enough to need its own aggregate spill, that spill reapplies these per-command data and argv redactors before persistence rather than using generic batch redaction.
|
|
717
783
|
|
|
718
|
-
`nextActions` is an optional machine-readable list of exact native `agent_browser` follow-ups. Each entry includes `tool: "agent_browser"`, an `id`, a short `reason`, optional `safety`, and either `params` (`args`, optional `stdin`, optional `sessionMode`, optional `networkSourceLookup`, optional `electron`) or an `artifactPath` for saved-file workflows. Agents should prefer
|
|
784
|
+
`nextActions` is an optional machine-readable list of exact native `agent_browser` follow-ups. Each entry includes `tool: "agent_browser"`, an `id`, a short `reason`, optional `safety`, and either `params` (`args`, optional `stdin`, optional `sessionMode`, optional `networkSourceLookup`, optional `electron`) or an `artifactPath` for saved-file workflows. Failure prose mirrors up to six payloads so Pi models can execute them without access to structured `details`; stdin up to 500 characters is shown exactly after redaction, while longer stdin stays `details.nextActions`-only to bound context. Agents should prefer the visible or structured payload over guessed commands. Browser-bearing follow-ups preserve a known `details.sessionName` with `--session <name>` so retries and diagnostics cannot drift into the implicit session, except actions whose `params.sessionMode` is `"fresh"`, which deliberately stay unprefixed because the planner ignores `sessionMode` alongside an explicit `--session`; when a result also ran under an upstream namespace, follow-up `params.args` preserve its exact value, including explicit `--namespace ""`, so an ambient namespace cannot redirect the same daemon/restore-state identity. Tab/session recovery id strings are centralized in `AGENT_BROWSER_RECOVERY_NEXT_ACTION_IDS`, while rich-input focus/click recovery ids are centralized in `AGENT_BROWSER_RICH_INPUT_RECOVERY_NEXT_ACTION_IDS` plus `getAgentBrowserRichInputRecoveryNextActionId(s)` in `extensions/agent-browser/lib/results/recovery-actions.ts`; docs and tests mirror those registries/helpers rather than inventing recovery ids in prose. Current recommendations include: ordinary `timeout` failures → `inspect-after-timeout` (`snapshot -i`), with `wait --text` using the more specific `inspect-after-text-assertion-failure`, and `wait --url` (including compiled `job.assertUrl`) also appending `fresh-session-after-url-wait-timeout` (`sessionMode: "fresh"` + `open about:blank`, ranked after the inspect action) with guidance that a silently missed upstream click dispatch may have prevented the expected navigation and that the fresh session should replay the flow as one batch; navigation-shaped `upstream-error` failures → `inspect-page-after-navigation-error`; failed script-session cleanup → exact `close-script-session-after-cleanup-failure`; timed-out jobs/batches with a retryable read-only/idempotent first incomplete step → `retry-timeout-step`, while timed-out flows whose first incomplete step may be mutating → `inspect-current-page-after-timeout` (`snapshot -i`) before splitting the remaining work into shorter batches; raw `connect` success → session-scoped `verify-connected-session-url` (`get url`) plus `list-connected-session-tabs`; page-content reads remain blocked until the current target is verified with `get url`, after which the agent can select/confirm a stable `tab t<N>`, verify it with `get url`, and run `snapshot -i`; `snapshot` failures whose upstream error says `No active page` and whose wrapper result has a known session → `list-tabs-after-no-active-page` only, because this path has no wrapper-observed safe tab id to select atomically; browser profile/user-data-dir resolution failures → `inspect-browser-profiles` (`profiles`) and `run-agent-browser-doctor` (`doctor`) before retrying opens; Electron launches → wrapper-tracked `electron.status` / `electron.probe` / `electron.cleanup` actions plus session-scoped tab/snapshot inspection when attached; Electron status/probe mismatch diagnostics → `reattach-electron-launch` plus fresh tab/snapshot inspection; Electron post-command health failures → status/probe/cleanup for the same `launchId`; Electron or contenteditable fill verification mismatches → `inspect-after-fill-verification` and `verify-filled-value`; Electron same-URL ref freshness warnings → `refresh-electron-refs-after-rerender`; packaged-Electron `sourceLookup` no-candidate diagnostics → session snapshot, launch probe, and tab list; Electron cleanup partial failures → status plus retry-cleanup for the same wrapper-owned `launchId`; `open` success → `snapshot -i`; mutating/navigation commands (see `buildAgentBrowserNextActions` in source for the exact command set) → `snapshot -i`; stale refs and selector failures → `snapshot -i` via `refresh-interactive-refs` (prefixed with `--session <name>` when the failed call ran in a named or managed session); selector misses with exact current snapshot role/name matches → direct ref retries via `try-current-visible-ref` or bounded `try-current-visible-ref-N` for non-fill targets; semantic `fill` selector misses with exact current editable refs → `focus-current-editable-ref` / `click-current-editable-ref` or numbered variants that do not include fill text or submit; unknown getter shortcuts such as `title` / `url` → exact read-only retries like `get title` / `get url` with ids `use-get-title` / `use-get-url`; compact `network requests` results with safe request IDs → bounded read-only request detail, `networkSourceLookup`, path filter, or HAR-capture follow-ups; semantic `selector-not-found` failures that compiled from `semanticAction` may append `try-button-name-candidate` or `try-link-name-candidate` after presentation `nextActions` only for the bounded click pair enumerated under `semanticAction`; semantic `stale-ref` failures that compiled from `semanticAction` `find` argv may also include `retry-semantic-action-after-stale-ref` after that snapshot step; successful snapshots or qualifying same-URL non-Electron top-level clicks (see `overlayBlockers` below) with snapshot evidence of likely overlay/banner/dialog close controls may append `inspect-overlay-state` and bounded `try-overlay-blocker-candidate-*` entries; successful top-level `scroll` calls whose pre/post viewport and sampled scroll-container positions do not change may append `inspect-after-noop-scroll` and `verify-noop-scroll-visually`; explicit combobox-targeted actions that focus a combobox without visible options may append `inspect-focused-combobox`, `try-open-combobox-with-arrow`, and `try-open-combobox-with-enter`; `get text <selector>` calls with hidden/multiple CSS matches may append `inspect-visible-text-candidates` with a read-only `eval --stdin` probe (each prefixed with `--session <name>` when `details.sessionName` is set, same `sessionPrefixArgs` rule as other session-scoped follow-ups); confirmations → exact `confirm <id>` and `deny <id>` choices; generic tab drift → `list-tabs-for-recovery` with `tab list` first, then select or confirm the stable target before running `snapshot -i`; about:blank or tab-drift recovery with a wrapper-known target → `list-tabs-for-about-blank-recovery` or `list-tabs-for-tab-drift-recovery`, plus `select-intended-tab-after-drift` and `snapshot-after-tab-recovery` when the wrapper already observed the stable `t<N>` tab id; `wait --text` assertion failures → `inspect-after-text-assertion-failure` with a read-only snapshot; download verification failures or missing successful download artifacts → `wait --download [path]`; saved artifacts → the artifact path to inspect/consume after checking `artifactVerification`/metadata; missing non-download artifacts → `verify-artifact-path` so agents do not trust an absent file. When nothing applies, the field is omitted.
|
|
719
785
|
|
|
720
786
|
**Unknown-command getter hints (failure presentation):** `buildErrorPresentation` in `extensions/agent-browser/lib/results/presentation/errors.ts` only runs this path when upstream error text (after model-facing redaction) matches `unknown command`, `unknown subcommand`, or `unrecognized command` (case-insensitive) **and** the failed invocation’s primary command token is one of `attr`, `count`, `html`, `text`, `title`, `url`, or `value`. Visible text then includes a grouped-`get` hint line plus per-token guidance (`get text <selector>`, `get html …`, `get attr …`, `get count …`, `get value …`, `get title`, `get url`). Machine `nextActions` with ids `use-get-title` / `use-get-url` are emitted only for `title` / `url`, with `params.args` optionally prefixed by `--session <name>` when the failed call targeted a named session. If the error string already contains `Agent-browser hint:` from selector recovery (stale-ref or unsupported selector dialect appendages), the getter block is skipped so two stacked `Agent-browser hint:` headers are not emitted.
|
|
721
787
|
|
|
@@ -723,7 +789,7 @@ For `network requests`, `details.nextActions` is bounded to one selected safe re
|
|
|
723
789
|
|
|
724
790
|
For `batch`, each `batchSteps[]` entry can carry its own `nextActions` for that step’s success or failure. Top-level `details.nextActions` on a failed batch duplicates `batchFailure.failedStep.nextActions` so callers can read one aggregate object. On a fully successful batch, top-level `nextActions` may still list artifact follow-ups derived from the combined step artifacts.
|
|
725
791
|
|
|
726
|
-
`pageChangeSummary` is an optional compact summary for mutation-prone and artifact-producing commands. It includes `changeType` (`"navigation"`, `"mutation"`, `"artifact"`, or `"confirmation"`), `command`, a readable `summary`, optional `title`/`url`, optional `artifactCount` or `savedFilePath`, and `nextActionIds`
|
|
792
|
+
`pageChangeSummary` is an optional compact summary for mutation-prone and artifact-producing commands. It includes `changeType` (`"navigation"`, `"mutation"`, `"artifact"`, or `"confirmation"`), `observed`, `command`, a readable `summary`, optional `title`/`url`, optional `artifactCount` or `savedFilePath`, and `nextActionIds`. `observed: false` means upstream dispatched a mutation-capable action but the wrapper did not observe an application change; standalone results also append a visible `Action dispatched; application change unverified` warning. The wrapper maintains explicit command/subcommand capability checks through `isPageChangeSummaryCommand` in `extensions/agent-browser/lib/command-taxonomy.ts`: those commands still emit a `mutation`-typed summary when upstream JSON lacks navigation metadata, as long as no stronger signal (artifact, saved path, navigation fields, or pending confirmation) applies. That capability is independent from `invalidatesBatchRefs` and `triggersPostMutationSnapshot`, so artifact summaries like `download` / `screenshot` and guarded-but-non-invalidating `fill` are documented directly in the capability table instead of implied by broad set spreading. Commands outside that set omit `pageChangeSummary` unless the parsed payload shows navigation, a confirmation prompt, saved files, or artifacts—including read-only inspection commands, which normally have no summary unless one of those signals appears. For `batch`, the top-level summary favors artifact rollups when any step produced artifacts; otherwise it synthesizes an observed-or-unverified summary from step evidence. Visible batch output promotes dispatch-only mutation evidence before step details and states that fixed waits are not postconditions. Agents should verify URL/text/state or an external receipt for important mutations before continuing.
|
|
727
793
|
|
|
728
794
|
`clickDispatch` may appear after a **top-level non-Electron** direct `click` when the wrapper installed a target-specific DOM-event probe, upstream reported success, and the post-click probe found no trusted DOM event reached the resolved target. Target-specific probes cover `xpath=` targets and role-gated `@e…` refs when the latest wrapper-tracked snapshot has role/name metadata; eligible ref roles are `button`, `checkbox`, `menuitem`, `radio`, `switch`, and `tab`, and duplicate-name refs use the ref's snapshot-order duplicate index rather than requiring a unique accessible name. Raw `find … click` locator calls, including compiled `semanticAction` clicks that still execute as upstream `find`, are not probed because the wrapper has no concrete element before upstream resolves the locator, and document-level probes can falsely fail frame-scoped clicks. It does **not** take a fresh pre-click snapshot because that could recycle upstream refs before the intended click. The wrapper does **not** replay clicks in-page. On a miss it marks the tool failed, appends `Click dispatch diagnostic: …`, and sets `clickDispatch.status` to `"no-native-event-observed"` with `reason: "native-click-produced-no-target-dom-event"`, `nativeEventCount`, and a redacted `target` descriptor (`kind: "xpath"` plus `selector`, or `kind: "accessible"` plus `refId`, `role`, optional `duplicateIndex`, and redacted `name`). `details.nextActions` gains `inspect-click-dispatch-miss` (`snapshot -i`) and `retry-click-after-dispatch-miss` (same upstream click argv, session-prefixed when applicable). If a local static fixture must be exercised despite this diagnostic, a caller may explicitly run a programmatic activation via `eval --stdin` such as `document.querySelector(...).click()`, but that emits an untrusted scripted event and is only a debugging/workaround path; it must not be used as proof that real user-like clicking works or to bypass prompt stop boundaries. This diagnostic is only for standalone top-level direct click calls; `find` locator clicks and `batch`/`job`/`qa` click steps remain upstream-owned behavior.
|
|
729
795
|
|
|
@@ -770,6 +836,7 @@ When `semanticAction` produced compiled `find` argv and the unified result is `f
|
|
|
770
836
|
"pageChangeSummary": {
|
|
771
837
|
"changeType": "navigation",
|
|
772
838
|
"command": "open",
|
|
839
|
+
"observed": true,
|
|
773
840
|
"summary": "Opened Example Domain",
|
|
774
841
|
"title": "Example Domain",
|
|
775
842
|
"url": "https://example.com/",
|
|
@@ -780,15 +847,18 @@ When `semanticAction` produced compiled `find` argv and the unified result is `f
|
|
|
780
847
|
Implementation and precedence:
|
|
781
848
|
|
|
782
849
|
- Shared machine-readable types are centralized in `extensions/agent-browser/lib/results/contracts.ts` (including re-exports such as `AgentBrowserNextAction` from `next-actions.ts`). Classifiers live in `categories.ts` (`classifyAgentBrowserSuccessCategory`, `classifyAgentBrowserFailureCategory`, `buildAgentBrowserResultCategoryDetails`—the last prefers an explicit `failureCategory` when the caller already knows the bucket, otherwise it runs the classifier). Generic follow-up assembly lives in `action-recommendations.ts` (`buildAgentBrowserNextActions`). Tab/session recovery ids live in `recovery-actions.ts` (`AGENT_BROWSER_RECOVERY_NEXT_ACTION_IDS`, `AGENT_BROWSER_RICH_INPUT_RECOVERY_NEXT_ACTION_IDS`, `getAgentBrowserRichInputRecoveryNextActionId`, `getAgentBrowserRichInputRecoveryNextActionIds`, `buildRecoveryNextActions`) and session-aware wrappers live in `recovery-next-actions.ts`. Selector miss and rich-input diagnostic shapes/actions live in `selector-recovery.ts`. Failed upstream `network requests` rows flow through `classifyNetworkRequestFailure` / `summarizeNetworkFailures` in `network.ts` for QA analysis (`analyzeQaPresetResults` in `extensions/agent-browser/index.ts`) and for actionable-vs-benign lines plus request-specific nextActions in `network requests` presentation (`extensions/agent-browser/lib/results/presentation/diagnostics.ts`).
|
|
783
|
-
- Artifact verification: `ArtifactVerificationSummary` / `ArtifactVerificationEntry` types live in `contracts.ts`. `buildArtifactVerificationSummary`, `getArtifactVerificationEntry`, and `getManifestVerificationEntry` in `presentation/artifacts.ts` merge each resolved file artifact with manifest rows whose `storageScope` is not `explicit-path` (those rows duplicate file artifacts) and whose `path` is in the current result’s spill path set. Presentation
|
|
784
|
-
- Inner success categories (`classifyAgentBrowserSuccessCategory` in `categories.ts`, after verification counts are clear): if `inspection` is true → `"inspection"`; else if any
|
|
785
|
-
- Failure: the classifier walks a single ordered chain (first match wins): explicit `options.confirmationRequired` → upstream locator-detail misses (`selector-not-found`, including 0.32.4+ `Names seen:` / `No element found: getByRole(...)` / `Element not found: … Verify the selector, role, or name`) → text-derived `confirmation-required` → `timeout` → `missing-binary` → `parse-failure` → `aborted` → `policy-blocked` → `cleanup-failed` → explicit `options.validationError` → `tab-drift` → `stale-ref` (including “unknown ref” text and a narrow `@eN` plus “element not found” heuristic) → `selector-unsupported` → other `selector-not-found` shapes → `download-not-verified` (download / wait-download style failures) → default `upstream-error`. Locator-detail misses are classified before text-derived confirmation/timeout so an accessible name containing those phrases cannot suppress selector recovery. Wrapper-known missing artifact checks pass an explicit `artifact-missing` category rather than relying on this text classifier.
|
|
850
|
+
- Artifact verification: `ArtifactVerificationSummary` / `ArtifactVerificationEntry` types live in `contracts.ts`. `buildArtifactVerificationSummary`, `getArtifactVerificationEntry`, and `getManifestVerificationEntry` in `presentation/artifacts.ts` merge each resolved file artifact with manifest rows whose `storageScope` is not `explicit-path` (those rows duplicate file artifacts) and whose `path` is in the current result’s spill path set. Presentation fails closed with `failureCategory: "artifact-missing"` when a non-pending artifact (including a previous recording finalized by `record restart`) is absent or when its `mtimeMs` falls outside the command's bounded start/end window with two seconds of filesystem precision tolerance (`status: "stale"`, `state: "unverified"`, and `updatedAtMs` expose that evidence). Batch preflight canonicalizes existing path ancestry, normalizes Unicode and folds case on macOS/Windows, and rejects duplicate explicit artifact destinations, including recording start/restart paths, so filesystem aliases or an earlier step cannot satisfy a later step's verification. Pending video entries from `record start` / `record restart` remain successful as `artifact-pending` until `record stop`.
|
|
851
|
+
- Inner success categories (`classifyAgentBrowserSuccessCategory` in `categories.ts`, after verification counts are clear): if `inspection` is true → `"inspection"`; else if any pending recording artifact exists → `"artifact-pending"`; else if any artifact lacks confirmed on-disk presence (`exists !== true`) and was not upgraded to an `artifact-missing` failure → `"artifact-unverified"`; else if there is a `savedFile` or any `artifacts` → `"artifact-saved"`; else → `"completed"`.
|
|
852
|
+
- Failure: the classifier walks a single ordered chain (first match wins): explicit `options.confirmationRequired` → `tab-gone` (`tab_gone:` signature, before lastUrl can match aborted/policy/about:blank heuristics) → upstream locator-detail misses (`selector-not-found`, including 0.32.4+ `Names seen:` / `No element found: getByRole(...)` / `Element not found: … Verify the selector, role, or name`) → text-derived `confirmation-required` → `timeout` → `missing-binary` → `parse-failure` → `aborted` → `policy-blocked` → `cleanup-failed` → explicit `options.validationError` → `tab-drift` → `stale-ref` (including “unknown ref” text and a narrow `@eN` plus “element not found” heuristic) → `selector-unsupported` → other `selector-not-found` shapes → `download-not-verified` (download / wait-download style failures) → default `upstream-error`. Locator-detail misses are classified before text-derived confirmation/timeout so an accessible name containing those phrases cannot suppress selector recovery. Wrapper-known missing artifact checks pass an explicit `artifact-missing` category rather than relying on this text classifier.
|
|
786
853
|
- The main tool implementation merges these fields into Pi-facing `details` from `extensions/agent-browser/index.ts` and from `extensions/agent-browser/lib/results/presentation.ts` for presentation-time failures.
|
|
787
854
|
|
|
788
855
|
Additional structured fields can appear when relevant:
|
|
789
|
-
- `
|
|
790
|
-
- `
|
|
791
|
-
- `
|
|
856
|
+
- `closeAllApplied: true` when a successful direct or nested `close` / `quit` / `exit --all` reached upstream. The marker makes live state and transcript replay clear every managed/attached/page/ref/route/trace/recording identity in the effective canonical namespace; a later batch row that proves browser reactivation may rebuild only the effective session.
|
|
857
|
+
- `attachedBrowserSession: true` on successful calls that establish or reuse a wrapper-tracked CDP/auto-connect/Electron attachment, and on a failed fresh attachment only when `managedSessionOutcome.activeAfter` proves its daemon remained active for cleanup. The marker restores attachment continuity from the active transcript branch, including that active-after-failure case; live state and transcript replay remove it after a terminal successful close/cleanup even when aggregate verification failed; a close followed by a later step whose lifecycle reports a browser launch preserves it, while a successful non-launching diagnostic leaves the close terminal. Caller config, environment, paths, and file-access settings remain upstream-owned; the marker only adds live-URL verification and lifecycle continuity.
|
|
858
|
+
- `lifecycle: { effectiveLaunch: { browserLaunched } }` when upstream returned that boolean. It separates starting the requested `agent-browser` CLI process from the effective Chrome session context. `readSource` exposes upstream's string `data.source` for direct `read` calls and identifies the raw HTTP fetch path; its lifecycle boolean can be `false` before a browser launch or `true` when the same managed session already has an active browser. Direct reads also append one visible `Read execution` line with the source, CLI-start result, managed browser lifecycle, and managed-session outcome so Pi models do not have to infer model-invisible details.
|
|
859
|
+
- `browserWindow: { mode: "headed", ownership: "wrapper-managed", sessionName, visibility: "unverified" }` only after a successful first/fresh local wrapper-managed headed call (including `batch`) that is not an attachment and whose lifecycle proves a browser launched and whose managed-session outcome is `created` or `replaced`. One visible handoff sentence tells the user to complete the login in that window if they can see it, then continue with `sessionMode: "auto"`; the field never claims OS desktop visibility.
|
|
860
|
+
- `sessionTabTargetUnknown: true` after a spawned `connect`, `state load`, history navigation, or tab-selection/close call changes the active page without a trustworthy observed target. Successful standalone tab selection/close now live-probes URL and a fresh non-blank title before state is committed, even when the new tab shares the prior URL; explicit selection of an existing `about:blank` tab and a post-close blank target are retained, so this marker remains only when the probe cannot verify the target. It is persisted and restored across branch/reload replay, clears stale refs and tab pinning, and blocks page inspection until `get url` or explicit navigation observes a target; `tab list`, tab selection, close, and blocking-dialog `status` / `accept` / `dismiss` remain available. A timeout against an unknown target removes standalone snapshot actions and returns `verify-page-target-after-timeout`, a session-scoped `batch --bail` whose stdin runs `get url` before `snapshot -i`.
|
|
861
|
+
- `compiledSemanticAction` when the call used `semanticAction` and the result includes the unified `details` merge: `{ action, locator, args }` for `find` actions or `{ action: "select", selector?, locator?, values, args }` for `select`, with the same redaction rules as `args` / `effectiveArgs`; omitted for plain `args`/`job` calls and omitted on some early error returns that omit this field (see the `semanticAction` section above)
|
|
792
862
|
- `compiledJob` when the call used `job` or the job-backed `qa` preset: by default `{ args: ["batch", "--bail"], failFast: true, stdin, steps: [{ action, args }] }`; with `failFast: false`, `{ args: ["batch"], failFast: false, stdin, steps: [{ action, args }] }`. Step args are redacted the same way as other invocation details. Semantic `job` click/fill steps appear here as their compiled upstream `find … click|fill …` argv, not as the input object.
|
|
793
863
|
- `compiledQaPreset` when the call used `qa`: the compiled job fields plus the QA `checks` object. `args` is `batch --bail` and `failFast` is `true` for QA presets. `checks.attached` is `true` for current-session QA, `checks.url` is present only for URL-opening QA, and `checks.diagnosticsResetAtStart` is `true` only for URL-opening QA because `qa.attached` preserves existing session diagnostics.
|
|
794
864
|
- `compiledSourceLookup` when the call used `sourceLookup`: `{ args: ["batch"], stdin, steps, query }` with the generated local-evidence plan and original query fields (`selector?`, `reactFiberId?`, `componentName?`, `includeDomHints?`, `maxWorkspaceFiles?`).
|
|
@@ -799,46 +869,49 @@ Additional structured fields can appear when relevant:
|
|
|
799
869
|
- `compiledElectron` when the call used `electron`: redacted action plan for `list`, `launch`, `status`, `cleanup`, or `probe`.
|
|
800
870
|
- `electron` when the call used `electron`: action-specific lifecycle, discovery, probe, and cleanup data; see the `electron` section below.
|
|
801
871
|
- `batchFailure` and `batchSteps` for `batch` rendering, including mixed-success runs
|
|
802
|
-
- `navigationSummary` for navigation-style commands like
|
|
872
|
+
- `navigationSummary` for navigation-style commands like `click`, `back`, `forward`, `reload`, and successful standalone tab selection/close; `urlChanged` records whether the live URL differs from a known pinned pre-command URL, so same-URL clicks and clicks without a comparison baseline remain dispatch-only rather than being mislabeled as observed navigation. Helper probes run `get url` first and run `get title` for any verified non-`about:blank` URL. The title read is skipped when the probed URL already carries a wrapper-observed title for this session, except after a tab selection/close, which always refreshes a non-blank title even when the URL is unchanged: titles are last-observed labels for that URL, while the URL itself is live-probed on every call. Href-less CSS selector clicks use this same post-command helper so `sessionTabTarget` cannot stay on the pre-click page; any click-dispatch check still runs first. A failed non-batch `eval`, `back`, `forward`, `reload`, `connect`, `state load`, or `tab` selection also runs this helper (browser started, not aborted, not watchdog-timed-out), so an observed page stays verified instead of forcing a manual `get url` round trip; a failed or empty probe keeps the prior unverified-page behavior. Because a failed transition can still have mutated or replaced the document, a successful probe also invalidates the prior page-scoped ref snapshot (matching the previous unknown-target behavior, which dropped refs), so the next `@e…` use requires a fresh `snapshot -i`.
|
|
803
873
|
- `pageChangeSummary` for compact mutation/artifact/navigation summaries on commands that can change browser state
|
|
804
|
-
- `clickDispatch` when a top-level non-Electron direct `click` reported upstream success but the target-specific probe found no trusted event reached the resolved target; shape follows `ClickDispatchDiagnostic` in `extensions/agent-browser/lib/orchestration/browser-run/types.ts`
|
|
874
|
+
- `clickDispatch` when a top-level non-Electron direct `click` reported upstream success but the target-specific probe found no trusted event reached the resolved XPath or accessible `@ref` target; shape follows `ClickDispatchDiagnostic` in `extensions/agent-browser/lib/orchestration/browser-run/types.ts`
|
|
805
875
|
- `promptGuard` when the requested-artifact-before-close guard blocks browser close before required prompt artifact paths are verified; implementation lives in `extensions/agent-browser/lib/orchestration/browser-run/prompt-guards.ts`
|
|
806
876
|
- `overlayBlockers` for conservative overlay/banner/dialog blocker candidates when a successful snapshot itself contains strong modal evidence, or after a qualifying top-level `@e…` / `ref=` click stays on the same URL, no `clickDispatch` diagnostic fired, and a fresh snapshot provides evidence (`candidates`, `summary`, and `snapshot` per `OverlayBlockerDiagnostic` in `extensions/agent-browser/index.ts`). CSS selector clicks do not run this overlay probe.
|
|
807
877
|
- `visibleRefFallback` after a raw `find` or compiled `semanticAction` fails with `selector-not-found` and a fresh snapshot finds exact role/name `@ref` matches. Shape follows `VisibleRefFallbackDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, snapshot, summary, target }`, where each candidate has `ref`, `role`, `name`, optional direct ref `args`, and `reason`; visible text appends `Current snapshot ref fallback`. Non-fill candidates with direct args add `try-current-visible-ref` or numbered `try-current-visible-ref-N` actions. Fill candidates omit direct args and target text so recovery details do not repeat potentially sensitive fill text.
|
|
808
|
-
- `refSnapshotInvalidation` after a session `snapshot` fails with `No active page
|
|
809
|
-
- `snapshotFilter` after wrapper-side `snapshot -i --search <text>` or `snapshot -i --filter role=<role>`. Shape: `{ cleanArgs, search?, role?, matchedRefs, totalRefs, visibleLines, totalLines }`.
|
|
878
|
+
- `refSnapshotInvalidation` after a session `snapshot` fails with `No active page`, any upstream-executed `record start` attempt swaps the session to a fresh active page (even one that then fails as `Recording already active`), a `record restart` with a URL operand navigates it, or a failed non-batch transition command (`eval`, `back`, `forward`, `reload`, `connect`, `state load`, `tab` selection) whose live URL re-verification probe observed the page (a failed transition can still have mutated the document, so the verified URL is kept but the prior refs are not). Shape follows `SessionRefSnapshotInvalidation` in `extensions/agent-browser/lib/session-page-state.ts`: `{ reason: "no-active-page" | "page-transition", summary }`; replay preserves the persisted summary. The wrapper deletes prior refs for that session, persists the invalidation for resume, and blocks mutation-prone `@e…` preflight with `failureCategory: "stale-ref"` until a successful fresh `snapshot -i` records refs again.
|
|
879
|
+
- `snapshotFilter` after wrapper-side `snapshot -i --search <text>` or `snapshot -i --filter role=<role>`. Shape: `{ cleanArgs, search?, role?, matchedRefs, totalRefs, visibleLines, totalLines, renderedTextMatches?, renderedTextTotalMatches?, renderedTextTruncated? }`. Search runs one bounded read-only rendered-DOM probe across the full document; each visible match carries bounded `text`, `tagName`, `kind` (`text` or prioritized `validation`), `offscreen`, optional `role`/accessible `name`, and a unique mapped `ref` when the full snapshot supports it. Hidden elements are excluded. The filtered accessibility snapshot remains separate, while `details.refSnapshot` still records the full upstream ref map for later stale-ref checks.
|
|
810
880
|
- `snapshotViewport` after wrapper-side `snapshot --viewport` (with or without `-i`, `--search`, or `--filter`). Shape matches the scroll-position probe: viewport scroll offsets, inner/document dimensions, sampled scrollable-container count, and bounded container offsets. The wrapper strips `--viewport` before upstream spawn and gathers this with a read-only `eval --stdin` call.
|
|
811
881
|
- `snapshotDiff` after wrapper-side `snapshot --diff` (with or without `-i`, `--search`, `--filter`, or `--viewport`). Shape: `{ addedRefs, removedRefs, changedRefs, unchangedRefs, summary }`, comparing ref ids plus role/name metadata from the previous wrapper-tracked snapshot for the session with the newly returned full ref map. It is a quick ref-map delta, not a visual diff.
|
|
812
882
|
- `networkRequestsPageFilter` after wrapper-side `network requests --current-page`, `--current-origin`, or `--current-url`. Shape: `{ cleanArgs, currentUrl, mode, matchedRows, totalRows }`; the visible rows and `details.data.requests` / `items` / `entries` are filtered while the active session page target is read with `get url`.
|
|
813
|
-
- `richInputRecovery` after a raw `find` or compiled `semanticAction` `fill` fails with `selector-not-found` and the same current-ref diagnostic finds exact editable `searchbox` / `textbox` candidates. Shape follows `RichInputRecoveryDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, inputMethodHint, nextActionIds, summary, target }`, where each candidate has `ref`, `role`, `name`, `focusArgs`, `clickArgs`, and `reason`. Visible text appends `Rich input recovery`, and `details.nextActions` gains ids from `getAgentBrowserRichInputRecoveryNextActionIds`: `focus-current-editable-ref` / `click-current-editable-ref` (or numbered variants). These actions are bounded to focus/click/inspect-style recovery: they do not include the fill text, do not press `Enter`, and do not submit. After the right current editable ref is focused,
|
|
883
|
+
- `richInputRecovery` after a raw `find` or compiled `semanticAction` `fill` fails with `selector-not-found` and the same current-ref diagnostic finds exact editable `searchbox` / `textbox` candidates. Shape follows `RichInputRecoveryDiagnostic` in `extensions/agent-browser/lib/results/selector-recovery.ts`: `{ candidates, inputMethodHint, nextActionIds, summary, target }`, where each candidate has `ref`, `role`, `name`, `focusArgs`, `clickArgs`, and `reason`. Visible text appends `Rich input recovery`, and `details.nextActions` gains ids from `getAgentBrowserRichInputRecoveryNextActionIds`: `focus-current-editable-ref` / `click-current-editable-ref` (or numbered variants). These actions are bounded to focus/click/inspect-style recovery: they do not include the fill text, do not press `Enter`, and do not submit. After the right current editable ref is focused, use `keyboard type` for framework-controlled editors that require real key events. Use paste-like `keyboard inserttext` only with separate application-state verification, and submit only when explicitly required by the flow.
|
|
884
|
+
- unsupported `scrollintoview text=<label>` / `scrollinto text=<label>` fails before upstream dispatch, directly or inside an effective raw/stdin batch row, because current upstream can report success without moving the page; help forms pass through unchanged. Visible failure text and `details.nextActions` both return the session-scoped `scroll-semantic-text-target` (`find text <label> hover`) when hover side effects are acceptable, plus `refresh-refs-for-scroll-target` (`snapshot -i`) before `scrollintoview <@ref>`. CSS, `xpath=...`, and current `@e…` targets remain native pass-through.
|
|
814
885
|
- `scrollPage` when the wrapper moves `document.scrollingElement` directly for `scroll <up|down|left|right> [px|percent]` or `scroll to end|top`; it temporarily disables smooth scrolling so immediate before/after offsets are reliable, returns `{ request, result }`, and includes `exitCode: 0` on success. Directional document no-movement falls through to upstream wheel behavior so nested panes still work. Explicit CSS-container calls `scroll <selector> <up|down|left|right> [px|percent]` remain wrapper-handled and report `details.scrollContainer`. All scroll helper shims are skipped when startup-scoped flags are present so the requested browser/profile launches before any helper command.
|
|
815
|
-
- `scrollNoop` after a successful large **top-level** upstream scroll fallback on an existing or fresh managed session when wrapper-side read-only probes before and after the command show no change in `window.scrollX` / `window.scrollY` and no change in the sampled prominent scrollable containers. To avoid pre-launching a session without caller startup state, this probe is skipped for small pixel scrolls, calls that would create a managed session only for the probe, and invocations with startup-scoped flags such as `--profile`, `--state`, `--restore`, `--namespace`, `--session-name`, `--cdp`, providers, init scripts, or similar launch settings. Shape: `{ reason: "no-observed-scroll-position-change", message, before, after, recommendations }`; `before` / `after` include viewport dimensions, document scroll dimensions, and up to ten sampled container descriptors plus scroll offsets. Container descriptors use only sample index, tag name, and ARIA role; DOM ids/classes are intentionally not stored. This diagnostic is conservative evidence that the page-level scroll likely missed a nested pane, not proof that every app-specific region is unchanged. Visible text starts with `Scroll completed with no observed movement`, appends `Scroll diagnostic: no observed scroll movement`, sets `details.data.scrolled` to `false` / `details.data.noMovement` to `true`, and `details.nextActions` gains `inspect-after-noop-scroll` (`snapshot -i`) plus `verify-noop-scroll-visually` (`screenshot`), session-prefixed when applicable.
|
|
886
|
+
- `scrollNoop` after a nominally successful large **top-level** upstream scroll fallback on an existing or fresh managed session when wrapper-side read-only probes before and after the command show no change in `window.scrollX` / `window.scrollY` and no change in the sampled prominent scrollable containers. The wrapper reclassifies this outcome as `failureCategory: "upstream-error"` rather than claiming the page scrolled. To avoid pre-launching a session without caller startup state, this probe is skipped for small pixel scrolls, calls that would create a managed session only for the probe, and invocations with startup-scoped flags such as `--profile`, `--state`, `--restore`, `--namespace`, `--session-name`, `--cdp`, providers, init scripts, or similar launch settings. Shape: `{ reason: "no-observed-scroll-position-change", message, before, after, recommendations }`; `before` / `after` include viewport dimensions, document scroll dimensions, and up to ten sampled container descriptors plus scroll offsets. Container descriptors use only sample index, tag name, and ARIA role; DOM ids/classes are intentionally not stored. This diagnostic is conservative evidence that the page-level scroll likely missed a nested pane, not proof that every app-specific region is unchanged. Visible text starts with `Scroll completed with no observed movement`, appends `Scroll diagnostic: no observed scroll movement`, sets `details.data.scrolled` to `false` / `details.data.noMovement` to `true`, and `details.nextActions` gains `inspect-after-noop-scroll` (`snapshot -i`) plus `verify-noop-scroll-visually` (`screenshot`), session-prefixed when applicable.
|
|
816
887
|
- `comboboxFocus` after a successful explicit combobox-targeted `click` / `fill` / `find … click|fill` (for example `semanticAction` with role `combobox`, including when that semantic action resolves through a current visible `@ref` before execution) when a read-only probe sees the active element is combobox-like, `aria-expanded` is explicitly present (`false` or `true`), and no visible `listbox` / `option` / menu option elements are open. Shape: `{ reason: "focused-combobox-without-visible-options", message, activeElement, visibleListboxCount, visibleOptionCount, recommendations }`; `activeElement` includes bounded role/tag/expanded/hasPopup/name metadata with normal text redaction. Visible text appends `Combobox diagnostic: focused combobox did not expose visible options`, and `details.nextActions` gains `inspect-focused-combobox` (`snapshot -i`), `try-open-combobox-with-arrow` (`press ArrowDown`), and `try-open-combobox-with-enter` (`press Enter`), session-prefixed when applicable. The diagnostic is deliberately gated to explicit combobox-targeted calls to avoid extra probes or false positives on ordinary clicks/textboxes.
|
|
817
888
|
- `recordingDependencyWarning` after a successful `record start` or `record restart` when the wrapper cannot find an executable `ffmpeg` on the Pi process `PATH`. Shape: `{ reason: "ffmpeg-missing-for-recording", dependency: "ffmpeg", command, message, recommendations }`. Visible text appends `Recording dependency warning: ffmpeg not found on PATH`. This is a non-blocking preflight warning: upstream may start recording, but `record stop` needs `ffmpeg` to encode the WebM.
|
|
818
889
|
- `selectorTextVisibility` after a **successful** upstream `get text <selector>` (standalone or inside a successful `batch`) when the wrapper’s follow-up probe finds a hazard: more than one DOM match (upstream reads the first `querySelectorAll` hit, which may be the wrong tab/panel), or the first match is hidden while at least one other match is visible (requires multiple DOM nodes so a visible peer exists; a lone hidden match is not flagged). The probe is a read-only `eval --stdin` script (`buildVisibleTextProbeScript` in `extensions/agent-browser/lib/orchestration/browser-run/diagnostics.ts`) that counts matches, applies a small visibility heuristic (`display`/`visibility`/`opacity` plus non-zero client rects), may include a redacted `firstVisibleTextPreview`, and may include up to eight `visibleCandidates` entries (`index` in `querySelectorAll`, `tagName`, optional `role`, optional redacted `textPreview`). It is **not** run for simple id selectors, page-scoped `@e…` selectors, or when the selector string is withheld because `selectorMayExposeSensitiveLiteral` would risk echoing secrets in probe output. `details.selectorTextVisibility` mirrors the primary diagnostic (first sorted entry); when several selectors in one `batch` qualify, `selectorTextVisibilityAll` lists every diagnostic sorted so hidden-first cases precede generic multi-match ambiguity. Appended visible warning text names the matching `details.nextActions` id and may list visible candidate previews. Appended `details.nextActions` use ids `inspect-visible-text-candidates` and `inspect-visible-text-candidates-2`, … with the probe replayed via `eval --stdin` for each hazardous selector. If the probe still leaves more than one visible candidate, it is only ambiguity evidence; agents should narrow the selector, use a current visible `@ref`, or run a targeted visible-element `eval --stdin` rather than trusting the broad selector.
|
|
819
|
-
- `electronGetTextScopeWarning` after a successful wrapper-tracked attached Electron `get text <selector>` (standalone or successful `batch`) when a broad non-ref CSS selector such as `body`, `html`, `main`, `div`, or `[role=application]` may read the whole app shell. Ordinary browser pages do not qualify without wrapper-owned Electron launch provenance
|
|
890
|
+
- `electronGetTextScopeWarning` after a successful wrapper-tracked attached Electron `get text <selector>` (standalone or successful `batch`) when a broad non-ref CSS selector such as `body`, `html`, `main`, `div`, or `[role=application]` may read the whole app shell. Ordinary browser pages do not qualify without wrapper-owned Electron launch provenance. Shape: `{ selector, summary, electronContext: { launchId?, sessionName?, url? } }`; multiple batched diagnostics use `electronGetTextScopeWarnings`. Visible text appends `Broad Electron get text selector warning`, and next actions use `snapshot-for-electron-text-scope` ids with session-scoped `snapshot -i` payloads.
|
|
820
891
|
- `evalStdinHint` after a successful `eval --stdin` when caller stdin (trimmed) looks function-shaped to the wrapper’s lightweight detector (in `extensions/agent-browser/lib/orchestration/browser-run/diagnostics.ts`: leading `function` / `async function`, parenthesized arrow `(…) =>`, or a concise `name =>` / `async name =>` form) **and** upstream JSON `data` is an object whose `result` field is a plain empty object (`{}`). Arrays such as `[]` do not qualify. It includes `reason` and `suggestion`; visible output appends `Eval stdin hint` with the same guidance. This is a heuristic for the common mistake of returning a function object instead of invoking it or passing a plain expression, not a JavaScript parser or proof that the page returned no useful data. Before this diagnostic path runs, the wrapper also recovers the common malformed native-tool call `args: ["eval", "--stdin", "..."]` with no top-level `stdin` by moving trailing `args` tokens after `--stdin` into the process stdin stream.
|
|
821
892
|
- `evalResultWarning` after a successful `eval --stdin` when the current or prior page URL is `file:` (from navigation summary, session tab target, or persisted session page state), upstream JSON `data.result` is strictly `null`, and stdin is non-empty and not a trivial literal `null`/`undefined`. Fields: `reason`, `suggestion`. Visible output appends `Eval result warning` without failing the tool. Use snapshot -i, ref-based getters, screenshots, or http(s) fixtures when file:// null results are inconclusive.
|
|
822
|
-
- `timeoutPartialProgress` after `runAgentBrowserProcess` reports `timedOut` (wrapper child-process watchdog) when best-effort recovery finds useful context. `summary` is a short sentence counting recovered planned-step state and declared artifact paths, plus whether page context came from live session reads or only from a planned URL (when nothing in the plan declares an artifact path, the fraction may read `0/0` while `currentPage` can still carry session or planned URL context). `steps` lists planned argv from the compiled `job` or `qa` batch plan (`compiledJob` in `extensions/agent-browser/index.ts`, which is only populated for those top-level modes) or, when that object is absent, from the same JSON-array `batch` stdin the tool sends upstream—whether caller-authored or wrapper-generated for `sourceLookup` / `networkSourceLookup` (1-based indices; only JSON-array stdin whose elements are string[] argv arrays is parsed). Generated rows such as `open.loadState` waits may include `generatedFrom`. Each step includes `status` (`completed`, `failed`, `pending`, or `unknown`) and optional `reason`; the first incomplete step becomes `retryStep`, but `retry.args` and top-level `retry-timeout-step` are emitted only for read-only or idempotent commands such as waits, snapshots, screenshots, navigation, and diagnostics. Mutating steps such as clicks, fills, keyboard typing, presses, selects, or checks are still identified as the first incomplete step but omit executable retry args because they may already have run; when the timed-out session is still usable, `details.nextActions` can instead include `inspect-current-page-after-timeout` (`snapshot -i`) so the agent verifies current state before continuing with a shorter split flow. When a retryable step timed out during `sessionMode: "fresh"` and no live URL was recovered, `retry-timeout-step` uses top-level `sessionMode: "fresh"` instead of prefixing the abandoned generated session name. `currentPage` comes from session-scoped `get url` followed by `get title`
|
|
893
|
+
- `timeoutPartialProgress` after `runAgentBrowserProcess` reports `timedOut` (wrapper child-process watchdog) when best-effort recovery finds useful context. `summary` is a short sentence counting recovered planned-step state and declared artifact paths, plus whether page context came from live session reads or only from a planned URL (when nothing in the plan declares an artifact path, the fraction may read `0/0` while `currentPage` can still carry session or planned URL context). `steps` lists planned argv from the compiled `job` or `qa` batch plan (`compiledJob` in `extensions/agent-browser/index.ts`, which is only populated for those top-level modes) or, when that object is absent, from the same JSON-array `batch` stdin the tool sends upstream—whether caller-authored or wrapper-generated for `sourceLookup` / `networkSourceLookup` (1-based indices; only JSON-array stdin whose elements are string[] argv arrays is parsed). Generated rows such as `open.loadState` waits may include `generatedFrom`. Each step includes `status` (`completed`, `failed`, `pending`, or `unknown`) and optional `reason`; the first incomplete step becomes `retryStep`, but `retry.args` and top-level `retry-timeout-step` are emitted only for read-only or idempotent commands such as waits, snapshots, screenshots, navigation, and diagnostics. Mutating steps such as clicks, fills, keyboard typing, presses, selects, or checks are still identified as the first incomplete step but omit executable retry args because they may already have run; when the timed-out session is still usable and its target is already verified, `details.nextActions` can instead include `inspect-current-page-after-timeout` (`snapshot -i`) so the agent verifies current state before continuing with a shorter split flow. When the target is unknown, every standalone snapshot action is removed and replaced by one executable `verify-page-target-after-timeout` action: a session-scoped `batch --bail` with stdin `[["get","url"],["snapshot","-i"]]`, so snapshot runs only after the wrapper's page-target guard is satisfied; visible failure text prints those redacted args and short stdin rather than pointing only to structured details. Dialog `status`, `accept`, and `dismiss` remain allowed while the target is unknown so timeout dialog recovery actions are executable. When a retryable step timed out during `sessionMode: "fresh"` and no live URL was recovered, `retry-timeout-step` uses top-level `sessionMode: "fresh"` instead of prefixing the abandoned generated session name. `currentPage` comes from session-scoped `get url` followed by `get title` when the session answers, otherwise a fallback URL may be inferred from the last `open` / `navigate` / `pushstate` step in the plan; `liveUrlRecovered` is true only when the wrapper recovered a live URL, so planned URLs are not treated as proof that the page actually opened. `openedButPostOpenTimedOut` is set when a live opened page was recovered and a later step appears to have timed out. `artifacts` covers declared output paths on `screenshot`, `pdf`, `download`, and `wait --download` steps (absolute path, existence, `state`, optional `sizeBytes`, `stepIndex`). Visible text repeats the same block under `Timeout partial progress`, applying URL and path-segment redaction; the prose `Planned steps` list shows at most six steps, then an omitted-count line when the plan is longer. This is recovery evidence only; missing entries do not prove the upstream step never ran or that no other side effects occurred.
|
|
823
894
|
- `managedSessionHeadedAutosaveInterval` on active/current-after-failure wrapper-owned headed session rows, containing the canonical effective launch-time `AGENT_BROWSER_AUTOSAVE_INTERVAL_MS` string (invalid or out-of-range explicit values resolve to upstream's `"30000"` default). It is `"0"` for the wrapper default and can hold an explicit interval such as `"1000"`; transcript replay and still-owned off-current helpers reapply the recorded value. It is omitted for sessionless, headless, caller-owned, abandoned, and closed calls. If a resumed Pi process explicitly requests a different value in either direction, non-close calls fail with close-plus-fresh recovery guidance while close still uses the recorded daemon value.
|
|
824
895
|
- `managedSessionHeadedAutosaveDisabled: true` is the narrower compatibility marker that the targeted session uses the wrapper's default interval `0`, rather than an explicit caller interval. It accompanies `managedSessionHeadedAutosaveInterval: "0"` on active rows and remains omitted for explicitly configured autosave.
|
|
825
|
-
- `managedSessionOutcome` after a managed-session plan reaches process execution (`buildManagedSessionOutcome` / `formatManagedSessionOutcomeText` in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`). Populated when `buildExecutionPlan` injects an extension-managed implicit or fresh `--session`, and also when a successful explicit `--session <current-wrapper-managed-session> close` closes the current managed session. It remains omitted for unrelated explicit user-managed sessions and for sessionless inspection/local paths that skip injection. Fields: `status` (`created`, `replaced`, `unchanged`, `closed`, `preserved`, or `abandoned`), `sessionMode`, `attemptedSessionName`, `previousSessionName`, `currentSessionName`, optional `currentSessionNamespace`, optional `replacedSessionName`, optional `replacedSessionNamespace`, optional `replacedSessionClosed` (false means automatic close failed and the previous session remains wrapper-owned/restorable for explicit cleanup), `activeBefore`, `activeAfter`, `succeeded`, and `summary` (machine-oriented; may include generated session names). Use `currentSessionNamespace` with `currentSessionName` when following preserved-session recovery actions; retry-fresh actions stay in the attempted namespace. Model-visible echo: when `sessionMode` is `"fresh"` **and** `succeeded` is false, or when `replacedSessionClosed` is false after a replacement, the wrapper appends action-oriented `Managed session outcome` and `Recovery` lines without repeating generated session ids in visible prose; session names remain in `details.managedSessionOutcome`. Failed fresh launches may also append `details.nextActions` such as `run-agent-browser-doctor`, `verify-current-managed-session`, `snapshot-current-managed-session`, or `retry-fresh-managed-session`. When other trailing diagnostic prose is also emitted in the same result, that block is concatenated **after** semantic-action candidate lines, overlay/selector-visibility tails, eval hints/warnings, and `Timeout partial progress` (see `rawAppendedDiagnosticText` in `extensions/agent-browser/lib/orchestration/browser-run/final-result.ts`). For `"auto"` failures the same struct may appear on `details` without that extra line. When post-upstream analysis (for example **`qa`** preset failure) flips the overall tool result after a successful batch, or a fresh `job`/batch opens the requested page and then a later step fails, the managed-session transition still reflects that the fresh browser became current. The visible recovery says the fresh launch became current and points to `failureCategory` / `qaPreset` / `batchFailure` for the post-launch failure instead of telling the agent that the old session was preserved.
|
|
896
|
+
- `managedSessionOutcome` after a managed-session plan reaches process execution (`buildManagedSessionOutcome` / `formatManagedSessionOutcomeText` in `extensions/agent-browser/lib/orchestration/browser-run/session-state.ts`). Populated when `buildExecutionPlan` injects an extension-managed implicit or fresh `--session`, and also when a successful explicit `--session <current-wrapper-managed-session> close` closes the current managed session. It remains omitted for unrelated explicit user-managed sessions and for sessionless inspection/local paths that skip injection. Successful nested-batch lifecycle rows are evaluated in order: a terminal close reports and replays `status: "closed"` even when aggregate artifact verification makes the tool result fail; a later lifecycle-proven browser launch (including a post-close `record stop`) keeps the session active, an explicitly non-launching diagnostic leaves it closed, and an unknown row stays conservatively active even when the failed batch was the first managed call. Fields: `status` (`created`, `replaced`, `unchanged`, `closed`, `preserved`, or `abandoned`), `sessionMode`, `attemptedSessionName`, `previousSessionName`, `currentSessionName`, optional `currentSessionNamespace`, optional `replacedSessionName`, optional `replacedSessionNamespace`, optional `replacedSessionClosed` (false means automatic close failed and the previous session remains wrapper-owned/restorable for explicit cleanup), `activeBefore`, `activeAfter`, `succeeded`, and `summary` (machine-oriented; may include generated session names). Use `currentSessionNamespace` with `currentSessionName` when following preserved-session recovery actions; retry-fresh actions stay in the attempted namespace. Model-visible echo: when `sessionMode` is `"fresh"` **and** `succeeded` is false, or when `replacedSessionClosed` is false after a replacement, the wrapper appends action-oriented `Managed session outcome` and `Recovery` lines without repeating generated session ids in visible prose; session names remain in `details.managedSessionOutcome`. Failed fresh launches may also append `details.nextActions` such as `run-agent-browser-doctor`, `verify-current-managed-session`, `snapshot-current-managed-session`, or `retry-fresh-managed-session`. When other trailing diagnostic prose is also emitted in the same result, that block is concatenated **after** semantic-action candidate lines, overlay/selector-visibility tails, eval hints/warnings, and `Timeout partial progress` (see `rawAppendedDiagnosticText` in `extensions/agent-browser/lib/orchestration/browser-run/final-result.ts`). For `"auto"` failures the same struct may appear on `details` without that extra line. When post-upstream analysis (for example **`qa`** preset failure) flips the overall tool result after a successful batch, or a fresh `job`/batch opens the requested page and then a later step fails, the managed-session transition still reflects that the fresh browser became current. The visible recovery says the fresh launch became current and points to `failureCategory` / `qaPreset` / `batchFailure` for the post-launch failure instead of telling the agent that the old session was preserved.
|
|
826
897
|
- `imagePath` / `imagePaths` for Pi inline image attachments from the **`screenshot`** command (including batched screenshot steps). **`diff screenshot`** still records the diff output as an `image`-kind entry in `details.artifacts`, but it does **not** populate `imagePath` / `imagePaths` or attach an inline image: only plain `screenshot` is treated as a trusted live-capture path for automatic inlining (`isTrustedScreenshotOutput` in `extensions/agent-browser/lib/results/presentation/artifacts.ts`).
|
|
827
|
-
- `artifacts` for saved files such as screenshots, `state save` outputs, `diff screenshot` diff images, PDFs, downloads, `wait --download` files, traces, CPU profiles, completed WebM recordings, path-bearing HAR captures, and future recording output paths reported by `record start` / `record restart`. Non-file URL payloads such as `data:` / `blob:` / `http(s):` values are not treated as verified local artifacts. For direct artifact commands and batch artifact steps, the wrapper creates parent directories for requested paths before spawning upstream. Each artifact includes the original saved or requested `path`, resolved `absolutePath`, `kind`/`artifactType`, optional `mediaType`, optional `extension`, best-effort disk metadata such as `exists` and `
|
|
828
|
-
- `savedFilePath` / `savedFile` for direct `download`, `pdf`, and `wait --download` saved-file workflows when a host file path is reported or wrapper-verified. Batch results preserve the same fields on the relevant `batchSteps` entry. These fields are metadata only until `artifactVerification` verifies the file. For simple loopback `download <selector> <path>` anchors with a non-ref selector, `details.downloadRecovery.method: "direct-anchor-fetch"` means the wrapper resolved the anchor URL in-session and saved the in-page HTTP(S) response directly to the requested path before using upstream's click/download fallback; non-loopback/profile downloads stay upstream-owned so external provider behavior is preserved.
|
|
829
|
-
- `batchSteps[].artifacts` for per-step artifacts in `batch` output; top-level `artifacts`
|
|
830
|
-
- `artifactVerification` for a normalized verification summary on the unified result and on each successful `batchSteps[]` row (failed batch steps omit artifact rows). Top-level `batch` verification rolls up all step file artifacts; each step’s summary reflects that step’s nested tool presentation (including its spill paths and manifest slice). It reports `verified`, `verifiedCount`, `missingCount`, `pendingCount`, `unverifiedCount`, and `artifacts[]` entries with `path`, optional `absolutePath`, optional `requestedPath`, `kind` (a normal file artifact kind or `"spill"` for manifest-backed rows), optional `mediaType`, optional `exists`, optional `sizeBytes`, optional `status`, optional `retentionState` / `storageScope` on manifest-derived rows, `state` (`verified`, `missing`, `pending`, or `unverified`), and optional `limitation` (human-readable lifecycle or retention context, for example pending `record start` / `record restart`, missing or unverified files, ephemeral spill files, or evicted persisted spills). The summary `verified` boolean is true only when every entry is `verified`. `record start` / `record restart` are `pending` until `record stop`; `state load` may mention a path in command output but is not a saved artifact row.
|
|
898
|
+
- `artifacts` for saved files such as screenshots, `state save` outputs, `diff screenshot` diff images, PDFs, downloads, `wait --download` / `wait -d` files, traces, CPU profiles, completed WebM recordings, path-bearing HAR captures, and future recording output paths reported by `record start` / `record restart`. Non-file URL payloads such as `data:` / `blob:` / `http(s):` values are not treated as verified local artifacts. For direct artifact commands and batch artifact steps, the wrapper creates parent directories for requested paths before spawning upstream. Each artifact includes the original saved or requested `path`, resolved `absolutePath`, `kind`/`artifactType`, optional `mediaType`, optional `extension`, best-effort disk metadata such as `exists`, `sizeBytes`, and `updatedAtMs`, plus `requestedPath`, `status`, `cwd`, `session`, `namespace`, and `tempPath` when applicable. For commands that create/update artifacts, a path that existed but was not updated during this command uses `status: "stale"`; observational `wait --download` may accept a file completed just before the wait began. Pending `record start` / `record restart` artifacts use `status: "pending"`, omit `exists` rather than reporting false, and include `recordingState: "openRecording"` / `willExistOnStop: true`. Within one Pi extension process, the wrapper keeps an unbounded transcript-backed active-recording reservation index separate from the bounded artifact manifest, keyed by canonical namespace plus session; still-live process-owned reservations survive branch switches, while in-memory terminal tombstones are appended to the current branch during shutdown/reload so a close on one branch cannot be resurrected after returning to an older branch. Artifact lifecycle calls, explicit `wait --download <path>` / `wait -d <path>` destinations, and result `outputPath` writes serialize around the global destination check/update, every successful direct, ordered nested-batch, fresh-replacement, script, Electron, or shutdown close retires only its exact identity at that lifecycle point, and destination reuse is rejected through lexical, existing or dangling symlink, hardlink, full Unicode-fold, or macOS/Windows case aliases. Batch preflight rejects `record start` / `record restart` after a close row because upstream can report a recording that did not start; split those operations into separate calls. A definitive `No recording in progress` stop failure, direct or nested in a batch, retires stale reservation state at that ordered step instead of recommending the same stop again; a later successful batch recording row opens its new pending path normally. Batch preflight applies the same distinct-destination rule to the steps upstream will execute: raw argument command strings exclusively when any exist, stdin arrays only otherwise; upstream-ignored stdin rows cannot fail artifact preflight, add pending recordings, or create parent directories. Parent directories are prepared for the effective steps in both modes; raw argument strings are never rewritten, so the screenshot absolute-path normalization and tracked path request apply to stdin rows only.
|
|
899
|
+
- `savedFilePath` / `savedFile` for direct `download`, `pdf`, and `wait --download` / `wait -d` saved-file workflows when a host file path is reported or wrapper-verified. Batch results preserve the same fields on the relevant `batchSteps` entry. These fields are metadata only until `artifactVerification` verifies the file. For simple loopback `download <selector> <path>` anchors with a non-ref selector, `details.downloadRecovery.method: "direct-anchor-fetch"` means the wrapper resolved the anchor URL in-session and saved the in-page HTTP(S) response directly to the requested path before using upstream's click/download fallback; non-loopback/profile downloads stay upstream-owned so external provider behavior is preserved.
|
|
900
|
+
- `batchSteps[].artifacts` for per-step artifacts in `batch` output; top-level `artifacts` and `artifactManifest` coalesce an earlier pending recording into the later saved, missing, or stale terminal result for the same namespace/session identity. `record restart` includes both the previous recording it finalized (or an explicit missing/stale failure) and the new pending recording; missing/stale terminal rows retire the prior pending manifest row. A successful later `close` / `quit` / `exit` represents an earlier unfinalized pending recording as `status: "missing"` / `subcommand: "close-abandoned"`, clears its stop action, and updates aggregate verification/manifest state consistently; a later successful `record stop` replaces that intermediate abandoned row with its saved artifact. Close also resets ref/page/network-route state produced by earlier rows; later lifecycle-proven browser launches, including `record stop`, can rebuild that state without triggering stale pre-close `about:blank` recovery, explicitly non-launching diagnostics cannot, and unknown later rows stay conservatively active. Per-step history remains unchanged. When any later call on the same namespace/session fails while a recording remains pending, `nextActions` combines its normal recovery with exact `stop-pending-recording` args and visible cleanup guidance; the same applies at top level when a later batch step fails.
|
|
901
|
+
- `artifactVerification` for a normalized verification summary on the unified result and on each successful `batchSteps[]` row (failed batch steps omit artifact rows). Top-level `batch` verification rolls up all step file artifacts; each step’s summary reflects that step’s nested tool presentation (including its spill paths and manifest slice). It reports `verified`, `verifiedCount`, `missingCount`, `pendingCount`, `unverifiedCount`, and `artifacts[]` entries with `path`, optional `absolutePath`, optional `requestedPath`, `kind` (a normal file artifact kind or `"spill"` for manifest-backed rows), optional `mediaType`, optional `exists`, optional `sizeBytes`, optional `updatedAtMs`, optional `status`, optional `retentionState` / `storageScope` on manifest-derived rows, `state` (`verified`, `missing`, `pending`, or `unverified`), and optional `limitation` (human-readable lifecycle or retention context, for example pending `record start` / `record restart`, missing, stale, or otherwise unverified files, ephemeral spill files, or evicted persisted spills). The summary `verified` boolean is true only when every entry is `verified`. `record start` / `record restart` are `pending` until `record stop`; `state load` may mention a path in command output but is not a saved artifact row.
|
|
831
902
|
- `fullOutputPath` / `fullOutputPaths` when parse-valid large snapshot output or other oversized tool output is compacted and spilled to a private file; persisted sessions keep that path under a private session-scoped artifact directory with a bounded per-session budget so it survives reload/resume without unbounded growth. Malformed oversized upstream output is discarded after parsing, is omitted from `details.stdout`, and reports `fullOutputUnavailable` instead of creating a parse-failure spill.
|
|
832
|
-
- `artifactManifest` for a bounded, metadata-only inventory of recent session artifacts. Entries include path metadata, artifact `kind`, source `command`/`subcommand` when safe, `storageScope` (`persistent-session`, `process-temp`, or `explicit-path`), and `retentionState` (`live`, `ephemeral`, `missing`, or `evicted`). The default recent window is 100 entries and can be configured with `PI_AGENT_BROWSER_SESSION_ARTIFACT_MANIFEST_MAX_ENTRIES`. The manifest must not store command args, output contents, headers, DOM snapshots, or downloaded file contents.
|
|
903
|
+
- `artifactManifest` for a bounded, metadata-only inventory of recent session artifacts. Entries include path metadata, optional canonical `namespace` plus `session` lifecycle identity, artifact `kind`, source `command`/`subcommand` when safe, `storageScope` (`persistent-session`, `process-temp`, or `explicit-path`), and `retentionState` (`live`, `ephemeral`, `missing`, or `evicted`). The default recent window is 100 entries and can be configured with `PI_AGENT_BROWSER_SESSION_ARTIFACT_MANIFEST_MAX_ENTRIES`. A successful session close retires only that exact namespace/session identity's pending recording rows; the separate active reservation index remains authoritative even if this bounded display inventory evicts them. Only the newest pending recording row per namespace/session identity remains live in the manifest. The manifest must not store command args, output contents, headers, DOM snapshots, or downloaded file contents.
|
|
833
904
|
- `artifactRetentionSummary` with a concise count of live, evicted, ephemeral, and missing artifacts from the current manifest; results append this summary to model-facing text only when retention state affects recovery, such as spill files, ephemeral files, or evictions. Routine explicit saved files keep the summary in details to avoid noisy browsing transcripts.
|
|
834
|
-
- `artifactCleanup` after a successful close command (`close`, `quit`, or `exit`) when `artifactManifest`
|
|
835
|
-
- compact **snapshot** metadata on successful presentation when `details.data.compacted` is true (oversized trees): `previewMode` (`"structured"` vs outline `"outline"`), `structuredPreviewUsed`, `previewRefIds`, `previewSections` (per-section `linesShown` / `omittedLines` / root `role` / `title`), `additionalSectionsOmitted`, counts such as `refCount`, `snapshotLineCount`, and `roleCounts`, optional `highValueControlRefIds` aligned with the visible bounded `Omitted high-value controls` lines, and optional `spillError` when the wrapper could not write the
|
|
905
|
+
- `artifactCleanup` after a successful close command (`close`, `quit`, or `exit`) only when `artifactManifest` contains at least one existing explicit artifact path. Fields: `owner: "host-file-tools"`, `summary` (same retention summary string as `artifactRetentionSummary` for that manifest), `note` explaining that browser close commands do not delete explicit screenshots/downloads/PDFs/traces/HAR/recordings, and `explicitArtifactPaths`: up to ten **distinct existing** paths taken from manifest rows with `storageScope: "explicit-path"` in encounter order (de-duplicated after checking the filesystem); deleted/stale explicit paths are skipped. When the recent window has only spill/ephemeral inventory or explicit paths already deleted, the field and visible cleanup guidance are omitted. The visible close text stays compact and points operators to `details.artifactCleanup.explicitArtifactPaths` instead of listing paths inline. The native browser tool intentionally does not expose a delete operation for arbitrary user-chosen artifact paths; agents should inspect `artifactVerification` / manifest metadata, then remove files with normal host file tools when cleanup is required.
|
|
906
|
+
- compact **snapshot** metadata on successful presentation when `details.data.compacted` is true (oversized trees): `previewMode` (`"structured"` vs outline `"outline"`), `structuredPreviewUsed`, `previewRefIds`, `previewSections` (per-section `linesShown` / `omittedLines` / root `role` / `title`), `additionalSectionsOmitted`, counts such as `refCount`, `snapshotLineCount`, and `roleCounts`, optional `highValueControlRefIds` aligned with the visible bounded `Omitted high-value controls` lines, and optional `spillError` when the wrapper could not write the redacted spill file; the model text still ends with `Full redacted snapshot path:` or an explicit unavailable reason plus `details.fullOutputPath` when a path exists
|
|
836
907
|
- `sessionRecoveryHint` when startup-scoped flags need `sessionMode: "fresh"` while an implicit session is already active: includes `reason`, `recommendedSessionMode` (`"fresh"`), redacted `exampleArgs`, and `exampleParams` where `sessionMode` is `"fresh"` and `args` is the same redacted argv as `exampleArgs` (from `buildExecutionPlan` in `extensions/agent-browser/lib/runtime.ts`, merged through `redactRecoveryHint` in `extensions/agent-browser/index.ts`)
|
|
837
908
|
- `inspection: true` plus `stdout` for successful plain-text inspection commands like `--help` and `--version`
|
|
909
|
+
- `versionValidation` on a browser-backed preflight failure when installed upstream output is not a stable version at or above the supported floor; it includes `expected` and optional parsed `observed`, while top-level `expectedVersion` remains the recommended current baseline, `minimumSupportedVersion` reports the floor, and `observedVersion` reports the installed version. The extension caches a successful `agent-browser --version` probe per cwd/PATH for the Pi process; plain help/version, close recovery, and sessionless local commands remain available without this browser-backed gate.
|
|
910
|
+
- `agentBrowserStarted` on results that reached browser-run processing: `false` proves the requested main subprocess never started (for example a socket-path, policy, or spawn preflight failure); `true` proves only that the CLI started, not that Chrome launched. Use `details.lifecycle.effectiveLaunch.browserLaunched` for the latter. Preparation helpers may already have touched the isolated session, so script leases always take the normal fail-closed cleanup path.
|
|
838
911
|
|
|
839
|
-
When the tool echoes `args` or `effectiveArgs` back into Pi, sensitive values such as `--headers`, proxy credentials, and auth-bearing URL parameters should be redacted first.
|
|
912
|
+
When the tool echoes `args` or `effectiveArgs` back into Pi, sensitive values such as `--headers`, proxy credentials, and auth-bearing URL parameters should be redacted first. URL redaction covers SAMLRequest, SAMLResponse, and RelayState (including common separator/case variants); `state` and `nonce` are redacted only when the same URL token has an auth/login/OAuth/OIDC/SAML/SSO context or another known sensitive query name, so ordinary application state URLs remain useful. Exact internal page-target URLs stay unredacted for browser correctness; model-facing values and persisted snapshot spills receive the redacted copies.
|
|
840
913
|
|
|
841
|
-
For parse-valid oversized snapshots and other oversized tool outputs, details should switch to a compact metadata object and include `fullOutputPath` pointing at a private spill file with the full redacted upstream payload. Malformed oversized output is not safe to redact structurally, so its temporary subprocess spill is deleted and no durable `fullOutputPath` is returned. The model-facing tool text should print the actual spill-file path when one exists instead of only saying to inspect a details key. Oversized batch/job/qa failures include bounded failed-step context inline before the preview so agents can see the failed assertion/error and failure category without opening the spill file. Persisted sessions should keep that spill file under a private session-scoped artifact directory so the path remains usable after reload/restart. The oldest persisted spill files are evicted as needed to stay within `PI_AGENT_BROWSER_SESSION_ARTIFACT_MAX_BYTES` (default 32 MiB), and those evictions are reported as `artifactManifest.entries[].retentionState: "evicted"` instead of silently disappearing from the session inventory. This persisted-spill byte budget is separate from the recent metadata window controlled by `PI_AGENT_BROWSER_SESSION_ARTIFACT_MANIFEST_MAX_ENTRIES`.
|
|
914
|
+
For parse-valid oversized snapshots and other oversized tool outputs, details should switch to a compact metadata object and include `fullOutputPath` pointing at a private spill file with the full redacted upstream payload. When the caller supplied `outputPath`, only matching live wrapper-manifest spills may provide pre-compaction payloads; direct compacted data, compacted result rows, and command-redacted whole-batch data are rehydrated in place, while any unavailable required spill fails without writing compact metadata. Malformed oversized output is not safe to redact structurally, so its temporary subprocess spill is deleted and no durable `fullOutputPath` is returned. The model-facing tool text should print the actual spill-file path when one exists instead of only saying to inspect a details key. Oversized batch/job/qa failures include bounded failed-step context inline before the preview so agents can see the failed assertion/error and failure category without opening the spill file. Persisted sessions should keep that spill file under a private session-scoped artifact directory so the path remains usable after reload/restart. The oldest persisted spill files are evicted as needed to stay within `PI_AGENT_BROWSER_SESSION_ARTIFACT_MAX_BYTES` (default 32 MiB), and those evictions are reported as `artifactManifest.entries[].retentionState: "evicted"` instead of silently disappearing from the session inventory. This persisted-spill byte budget is separate from the recent metadata window controlled by `PI_AGENT_BROWSER_SESSION_ARTIFACT_MANIFEST_MAX_ENTRIES`.
|
|
842
915
|
|
|
843
916
|
## High-value result rendering
|
|
844
917
|
|
|
@@ -848,20 +921,20 @@ The TUI renderer is user-facing only. It may compact or colorize what the human
|
|
|
848
921
|
|
|
849
922
|
Worth doing in v1:
|
|
850
923
|
- screenshots → saved-path summary, visible artifact metadata, `details.artifacts` metadata, and inline image attachment when safe; screenshot paths that upstream would treat ambiguously, such as `.dogfood/run/foo.png`, are normalized to absolute paths before launch and repaired from upstream temp output when possible
|
|
851
|
-
- file artifacts such as PDFs, downloads, `wait --download` files, `state save` state files, diff screenshot output images, traces, CPU profiles, completed WebM recordings, and path-bearing HAR captures → concise saved-path summaries plus metadata in `details.artifacts` and bounded recent metadata in `details.artifactManifest`; `record start` / `record restart` report recording lifecycle state and the future output path without adding a missing manifest entry, and `record restart` can also report the previous wrapper-known recording that was finalized by the restart; upstream needs `ffmpeg` on `PATH` for `record stop` to encode the WebM, and successful `record start` / `record restart` calls may also expose `details.recordingDependencyWarning` when the wrapper cannot find `ffmpeg`; direct saved-file workflows also expose `details.savedFilePath` / `details.savedFile`; large or binary artifacts are not inlined into model context; the recent manifest cap can age out explicit-file metadata but does not remove explicit saved files from disk
|
|
924
|
+
- file artifacts such as PDFs, downloads, `wait --download` / `wait -d` files, `state save` state files, diff screenshot output images, traces, CPU profiles, completed WebM recordings, and path-bearing HAR captures → concise saved-path summaries plus metadata in `details.artifacts` and bounded recent metadata in `details.artifactManifest`; `record start` / `record restart` report recording lifecycle state and the future output path without adding a missing manifest entry, and `record restart` can also report the previous wrapper-known recording that was finalized by the restart; upstream needs `ffmpeg` on `PATH` for `record stop` to encode the WebM, and successful `record start` / `record restart` calls may also expose `details.recordingDependencyWarning` when the wrapper cannot find `ffmpeg`; direct saved-file workflows also expose `details.savedFilePath` / `details.savedFile`; large or binary artifacts are not inlined into model context; the recent manifest cap can age out explicit-file metadata but does not remove explicit saved files from disk
|
|
852
925
|
- `diff screenshot` → same file-artifact pattern as above for the **diff** image path only (summary text uses “Saved diff image” only when the diff output exists; missing output says “Diff image reported; file not verified” and fails as `artifact-missing`); baseline paths and other fields stay in the structured payload but are not echoed as separate saved artifacts in the visible artifact block, and there is no Pi inline image attachment for the diff output
|
|
853
926
|
- `state load` → completion text may mention the loaded path, but the wrapper does **not** treat that path as a new saved artifact (`artifacts` / `artifactManifest` stay unset) the way `state save` does
|
|
854
927
|
- auth, cookies, storage, clipboard, dialog, frame, state, network, debug, diff, stream, dashboard, chat, and other structured results → concise summaries that avoid expanding secret-bearing payloads; `state show` exposes metadata only in visible text and redacts every cookie/localStorage/sessionStorage `value` in structured details; credential-like keys, values, URLs, body snippets, bearer/basic credentials, clipboard write text, cookie values, and likely secret storage values are redacted before model-facing output and `details.data`, while benign primitive storage values may remain visible for local QA
|
|
855
928
|
- TUI display → custom `agent_browser` call/result rendering with colorized command/output text and a built-in-style collapsed view for long visible output; top-level native modes render as `agent_browser qa → batch --bail`, `agent_browser job → batch --bail` by default (`agent_browser job → batch` when `failFast:false`), or `agent_browser semanticAction → find …` so reviewers can see both the native input mode and compiled upstream command; failed results keep `resultCategory` / `failureCategory` visible before truncated output; `ctrl+o` expansion reveals the full rendered tool result without changing the model-facing content
|
|
856
|
-
- snapshots → origin + ref count + main-content-first compact preview, with the
|
|
929
|
+
- snapshots → origin + ref count + main-content-first compact preview, with the redacted snapshot spill path printed directly in content and kept in `details.fullOutputPath` plus `details.artifactManifest` when the inline result would otherwise be too large
|
|
857
930
|
- oversized generic outputs such as large `eval --stdin` payloads → compact preview plus the actual spill file path instead of dumping the whole payload into model context
|
|
858
|
-
- `read [url]` → upstream `data.content` first, with source/content-type/status/final-URL metadata retained in `details.data`;
|
|
931
|
+
- `read [url]` → upstream `data.content` first, with source/content-type/status/final-URL metadata retained in `details.data`; direct reads also expose a visible `Read execution` line plus `details.readSource`, `agentBrowserStarted`, and upstream `lifecycle.effectiveLaunch.browserLaunched`, whose value reflects the effective managed session (including an already-active browser) rather than changing the raw HTTP read source. Explicit fetched URLs are diagnostic-only and do not replace the active browser tab target, while `read --timeout <ms>` extends the wrapper subprocess budget across upstream's per-request `.md` and ancestor-`llms.txt` fallback sequence when needed
|
|
859
932
|
- extraction-style commands like `eval --stdin` and `get title` → scalar-first text with lightweight origin context when available
|
|
860
933
|
- navigation actions like `click`, `back`, `forward`, and `reload` → lightweight post-action title/url summary when available
|
|
861
934
|
- tab lists → compact summary/table
|
|
862
935
|
- stream status → enabled/connected/port summary plus WebSocket URL and frame format when a port is known; `stream enable` errors that only say streaming is already enabled are normalized to a successful idempotent no-op with `details.data.alreadyEnabled: true` and status/disable nextActions; if the caller explicitly passed `--json`, visible text is valid JSON instead of a prose summary
|
|
863
|
-
- diagnostic/status families (`session`, `session list`, `profiles`, `doctor`, `auth list`/`show`, `cookies`, `storage`, `dialog`, `frame`, `state`, `network requests`, `console`, `errors`, and dashboard start/stop/status outputs) → compact readable summaries with counts and stable fields; `doctor` renders status/check/fix rows even when upstream puts those fields at the top level of its JSON envelope; `session list` keeps
|
|
864
|
-
- trace/profiler owner conflicts → when the wrapper has observed one owner active for a session, block conflicting starts/stops with "wrapper believes ..." wording because upstream or external CLI use can desynchronize wrapper-local state
|
|
936
|
+
- diagnostic/status families (`session`, `session list`, `profiles`, `doctor`, `auth list`/`show`, `cookies`, `storage`, `dialog`, `frame`, `state`, `network requests`, `console`, `errors`, and dashboard start/stop/status outputs) → compact readable summaries with counts and stable fields; `doctor` renders status/check/fix rows even when upstream puts those fields at the top level of its JSON envelope; `session list` keeps every upstream name/label/active marker/title/URL readable, including wrapper-prefixed rows, and `tab list` keeps its corresponding fields readable instead of opaque generated ids only; `network requests` and `console` previews label their scope as the upstream session aggregate unless upstream or a URL-opening QA preset explicitly cleared/filtered the buffers first; network request lists include an actionable-vs-benign failed-request summary and mark low-impact browser icon failures separately; active route mocks can add failed/pending/CORS route diagnostics; `data:image` artifact request rows are hidden from compact previews while preserved in raw details; request-detail URLs from `network request` and fetched URLs from explicit `read <url>` remain diagnostic-only rather than session page targets; large log/request/error outputs use previews plus `fullOutputPath` spill files; sensitive nested auth/header/token fields are not expanded in the model-facing text
|
|
937
|
+
- trace/profiler owner conflicts → when the wrapper has observed one owner active for a session, block conflicting starts/stops with "wrapper believes ..." wording because upstream or external CLI use can desynchronize wrapper-local state; every successful direct or nested close clears that wrapper owner at its ordered step, namespace-scoped `close --all` clears every matching session owner, and a later successful trace/profiler row may establish a new owner after browser reactivation
|
|
865
938
|
|
|
866
939
|
## Missing binary behavior
|
|
867
940
|
|
|
@@ -881,14 +954,14 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
881
954
|
- set one idle timeout on extension-managed sessions as a backstop for abnormal exits or cleanup failures, and pass that same `AGENT_BROWSER_IDLE_TIMEOUT_MS` to top-level commands plus every wrapper helper subprocess so upstream does not restart the background browser, reset the active tab, or discard current refs between a snapshot and action
|
|
882
955
|
- clean up process-private temp spill artifacts on shutdown, while keeping persisted-session snapshot spill files in a private session-scoped artifact directory so `details.fullOutputPath` survives reload/restart and the oldest spill files are evicted if the per-session artifact budget is exceeded
|
|
883
956
|
- reconstruct the current branch-visible extension-managed session, every transcript-proven still-active wrapper-owned identity, latest page-scoped refs, newest-revision aggregate `artifactManifest`, and wrapper-tracked Electron launch records from the active transcript branch on `session_start` and Pi `session_tree` so later default and explicit off-current calls keep following owned managed browsers and can continue reporting artifact retention state; successful explicit wrapper-owned close rows and `electron.cleanup` managed-session steps are restore-visible close events
|
|
884
|
-
- keep runtime cleanup ownership separate from branch-visible state: `session_tree` restore and wrapper-owned browser commands are serialized with managed-session work; caller-owned explicit-session commands use separate process-local queues keyed by effective canonical namespace/session (explicit namespace argv wins over inherited `AGENT_BROWSER_NAMESPACE`, including an explicit empty default), so the live URL probe, preparation helpers, semantic snapshot, main command, and state commit for one identity cannot interleave while different identities remain concurrent. Namespace and session identity components are additionally Unicode-normalized and case-folded on macOS and Windows to match their case-insensitive daemon paths. Only the outer tool execution acquires that key; nested helpers run under it without re-entry. Policy, route, and artifact deltas survive unrelated managed-state commits, while a separate branch-restore generation guard prevents stale completions from overwriting a newer branch. Concurrent artifact-producing results carry a monotonic aggregate manifest revision so transcript replay selects the complete bounded manifest rather than whichever call happened to occupy the last row. Extension-managed sessions and wrapper-launched Electron records owned by the current process remain eligible for quit/cleanup, and fresh-session allocation stays monotonic across branch restores, including auto rows and close rows that reference wrapper-generated fresh names
|
|
957
|
+
- keep runtime cleanup ownership separate from branch-visible state: `session_tree` restore and wrapper-owned browser commands are serialized with managed-session work; caller-owned explicit-session commands use separate process-local queues keyed by effective canonical namespace/session (explicit namespace argv wins over inherited `AGENT_BROWSER_NAMESPACE`, including an explicit empty default), so the live URL probe, preparation helpers, semantic snapshot, main command, and state commit for one identity cannot interleave while different identities remain concurrent. Namespace-scoped `close --all` is the exception: it drains and exclusively barriers managed plus matching caller-owned work before clearing global namespace state. Namespace and session identity components are additionally Unicode-normalized and case-folded on macOS and Windows to match their case-insensitive daemon paths. Only the outer tool execution acquires that key; nested helpers run under it without re-entry. Policy, route, and artifact deltas survive unrelated managed-state commits, while a separate branch-restore generation guard prevents stale completions from overwriting a newer branch. Concurrent artifact-producing results carry a monotonic aggregate manifest revision so transcript replay selects the complete bounded manifest rather than whichever call happened to occupy the last row. Extension-managed sessions and wrapper-launched Electron records owned by the current process remain eligible for quit/cleanup, and fresh-session allocation stays monotonic across branch restores, including auto rows and close rows that reference wrapper-generated fresh names
|
|
885
958
|
- when a close command or `electron.cleanup` successfully closes the current wrapper-managed session, clear live page/ref state, reserve the next generated fresh-session ordinal, and rotate the next default auto call to a fresh wrapper-generated session name rather than reusing the closed name
|
|
886
959
|
- when `/reload` shuts down an extension instance, close off-branch owned managed sessions and off-branch owned Electron launches before clearing process-local ownership; retain attached-browser context for those still-owned off-branch resources so cleanup cannot resend local-launch defaults. Preserve only the current branch-visible active managed session and active Electron launch plus its isolated `userDataDir` for reload continuity, and also persistently protect `userDataDir` paths when partial cleanup intentionally skips or fails profile removal so later temp cleanup, process exit, and stale temp-root pruning after restart do not violate Electron cleanup's safety decision; rebuild active branch state from the active branch on the next `session_start`
|
|
887
960
|
- when an unnamed `sessionMode: "fresh"` launch succeeds, make it the new extension-managed session so later default calls keep using it
|
|
888
961
|
- when an unnamed `sessionMode: "fresh"` launch fails or times out, preserve the previous managed session when one was active or report the attempted fresh session as abandoned when no managed session was active (`details.managedSessionOutcome`; visible `Managed session outcome: …` when the final tool call used `sessionMode: "fresh"` and failed, or when automatic close of its replaced session failed—see `#details`)
|
|
889
962
|
- if that unnamed fresh launch replaced an already-active managed session, best-effort close the old managed session after the switch succeeds; `details.managedSessionOutcome.replacedSessionClosed` records the cleanup result, and `false` keeps the older identity wrapper-owned across transcript resume for explicit follow-up or cleanup
|
|
890
|
-
- treat explicit caller-provided
|
|
891
|
-
- before a content-bearing read or interaction against a caller-owned explicit session or
|
|
963
|
+
- treat every explicit caller-provided `--session` as user-managed, including `piab-*` names. Wrapper-owned implicit sessions set a Pi-transcript- and Git-checkout-generation-scoped `AGENT_BROWSER_RESTORE` key automatically unless disabled with `PI_AGENT_BROWSER_MANAGED_SESSION_RESTORE=0`; explicit caller sessions do not receive that injection unless they exactly target the current wrapper-owned identity. Caller state/restore paths, profiles, upstream config, file access, launch arguments, environment variables, local file pages, `outputPath`, and close arguments pass through unchanged. `session list` and `state list` keep all upstream rows and restore identifiers visible. Automatic restore still validates and pins its own checkout/storage/namespace identity and coordinates same-daemon reuse so the wrapper cannot mix restore pools or corrupt managed lifecycle state. Ambiguous tab, attachment, history, script, or state-load transitions remain page-target correctness boundaries: content calls live-check `get url` or require explicit navigation before acting. Native Windows command-first adaptation moves only valid leading globals, rewrites valued `--restore <name>` as `--restore=<name>`, consumes only exact lowercase boolean literals, and leaves invalid or command-scoped leading input unchanged.
|
|
964
|
+
- before a content-bearing read or interaction against a caller-owned explicit session or established attachment, run a session-scoped `get url` probe so stale transcript state cannot target the wrong page. A failed or non-URL probe blocks the requested content command. The process-local namespace/session queue keeps that probe atomic with semantic snapshot resolution and the main command inside one extension instance. Nested `batch` steps remain unsupported; raw batch command strings mirror upstream's ASCII-space tokenizer, including quoting and backslash handling.
|
|
892
965
|
- pass explicit `--profile` straight through to upstream `agent-browser`; no profile-cloning or isolation layer is added in v1
|
|
893
966
|
<!-- agent-browser-playbook:start wrapper-tab-recovery -->
|
|
894
967
|
<!-- Generated from extensions/agent-browser/lib/playbook.ts. Run `npm run docs -- playbook write` to update. -->
|
|
@@ -896,13 +969,14 @@ If `agent-browser` is not on `PATH`, fail with a message that:
|
|
|
896
969
|
- After the wrapper observes tab-drift risk for a session (for example open correction, overlapping stale opens, or resumed session state), later active-tab commands best-effort pin that tab inside the same upstream invocation. Routine same-session commands are not preflighted with tab list just because a target tab or ref snapshot is known.
|
|
897
970
|
- For sessions with observed tab-drift risk, after a successful command on a known target tab, agent_browser also best-effort restores that intended tab if a restored/background tab steals focus after the command completes. Routine same-session commands skip this post-command tab-list probe.
|
|
898
971
|
- If a known session target unexpectedly reports about:blank, agent_browser best-effort re-selects the prior intended target when it still exists; if recovery fails, it records the observed about:blank target and reports exact recovery guidance instead of treating the prior page as active.
|
|
972
|
+
- If upstream reports tab_gone, the pinned bound tab is gone; use details.nextActions (tab list / tab new) instead of assuming another tab is yours.
|
|
899
973
|
<!-- agent-browser-playbook:end wrapper-tab-recovery -->
|
|
900
|
-
- on local Unix launches, set a short private socket directory for wrapper-spawned `agent-browser` processes so extension-generated session names do not fail the upstream Unix socket-path length limit in longer cwd/session-name combinations; require an absolute non-symlink directory owned by the current uid with mode `0700`, otherwise fail before spawn
|
|
901
|
-
- keep wrapper-spawned commands bounded by clamping `AGENT_BROWSER_DEFAULT_TIMEOUT` to the upstream documented default of 25 seconds while the default wrapper child-process watchdog is 35 seconds (`PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides it, and top-level `timeoutMs` overrides it per call for browser CLI subprocesses). Explicit `wait <ms
|
|
974
|
+
- on local Unix launches, set a short private socket directory for wrapper-spawned `agent-browser` processes so extension-generated session names do not fail the upstream Unix socket-path length limit in longer cwd/session-name combinations; require an absolute non-symlink directory owned by the current uid with mode `0700`, otherwise fail before spawn. Android/Termux uses a short directory under the owner-only `/data/data/<package>` app sandbox, compacts generated managed identities to one 80-bit digest so ordinary namespace plus fresh-session paths remain within the limit, stores policy-lock coordination under `os.tmpdir()`, and probes process identity with Termux's `ps` beside Node instead of unavailable `/bin/ps`
|
|
975
|
+
- keep wrapper-spawned commands bounded by clamping `AGENT_BROWSER_DEFAULT_TIMEOUT` to the upstream documented default of 25 seconds while the default wrapper child-process watchdog is 35 seconds (`PI_AGENT_BROWSER_PROCESS_TIMEOUT_MS` overrides it, and top-level `timeoutMs` overrides it per call for browser CLI subprocesses). Explicit `wait <ms>`, `wait --timeout <ms>`, and WebMCP `invoke` / `result --timeout <ms>` calls can exceed that default; when top-level `timeoutMs` is omitted, the wrapper derives a per-call subprocess watchdog from the requested command duration plus a small grace window. Dialog commands use `PI_AGENT_BROWSER_DIALOG_PROCESS_TIMEOUT_MS` (default 5000 ms), and click/tap/find refs or tokens plus `eval --stdin` snippets whose text looks like alert/confirm/prompt/dialog triggers use `PI_AGENT_BROWSER_DIALOG_TRIGGER_PROCESS_TIMEOUT_MS` (default 8000 ms). Timed-out compiled `job` / `qa` or caller `batch` calls may add `details.timeoutPartialProgress` and visible `Timeout partial progress` evidence with per-step status, retry payloads, current page title/URL, and declared artifact path checks; timed-out dialog-like commands may add dialog status/dismiss/fresh-session recovery next actions
|
|
902
976
|
- interactive or long-running upstream families such as `chat` without a prompt, `dashboard start`, `stream enable`, `trace start`, `profiler start`, `record start`, `inspect`, `install`, `upgrade`, `doctor --fix`, and `confirm-interactive` are passed through thinly but remain bounded by the same wrapper timeout/session planning rules; prefer explicit arguments, single-shot `chat <message>`, non-interactive flags like `doctor --offline --quick` or `doctor --json`, and cleanup pairs such as `dashboard stop`, `stream disable`, `trace stop`, `profiler stop`, and `record stop`
|
|
903
977
|
- treat successful plain-text inspection commands like `--help` and `--version` as stateless: do not inject the implicit managed session and do not let those calls claim the managed-session slot
|
|
904
|
-
- if startup-scoped flags like `--profile`, `--executable-path`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--enable`, `-p` / `--provider`, or iOS `--device`
|
|
905
|
-
- for direct headless local Chrome launches to `chat.com` / `chatgpt.com` / `chat.openai.com` or `dash.cloudflare.com`, allow a narrow compatibility fallback that injects a normal Chrome `--user-agent` only when the caller did not explicitly provide one and did not choose raw Chrome arguments, headed, CDP, auto-connect, provider-backed, custom-UA, or non-Chrome behavior through argv or matching upstream environment. Wrapper-managed sessions retain that wrapper-owned user agent across follow-up calls and branch reload/resume so upstream does not
|
|
978
|
+
- if startup-scoped flags like `--profile`, `--args`, `--user-agent`, `--executable-path`, `--ca-cert`, `--no-ca-cert`, `--webgpu`, `--restore`, `--restore-save`, restore check flags, `--namespace`, `--session-name`, `--cdp`, `--state`, `--auto-connect`, `--init-script`, `--idle-timeout`, `--enable`, `-p` / `--provider`, or iOS `--device` target the current active managed session, return a validation error with a structured recovery hint that recommends `sessionMode: "fresh"`; when the call explicitly names the current managed session, remove that `--session` from the recovery payload so rotation is actionable. If daemon inspection proves an explicitly targeted older wrapper-owned session active, reject its startup-scoped flags with close-first or fresh-rotation guidance before spawn
|
|
979
|
+
- for direct headless local Chrome launches to `chat.com` / `chatgpt.com` / `chat.openai.com` or `dash.cloudflare.com`, allow a narrow compatibility fallback that injects a normal Chrome `--user-agent` only when the caller did not explicitly provide one and did not choose raw Chrome arguments, headed, CDP, auto-connect, provider-backed, custom-UA, or non-Chrome behavior through argv or matching upstream environment. Wrapper-managed sessions retain that wrapper-owned user agent as per-session state across follow-up calls, failed replacement closes, and branch reload/resume. Active daemons omit both launch forms so upstream does not replace a launch-configured browser; a session proven inactive receives the retained compatibility launch values, including a fixed, comma-safe Chrome launch argument because the per-page override does not propagate to new tabs or SSO popups.
|
|
906
980
|
|
|
907
981
|
## Non-goals
|
|
908
982
|
|